Tata Consultancy Services (TCS)
Must Have Technical/Functional Skill
Design, develop, and maintain cloud native data engineering solutions using Azure Databricks.
Build and manage PySpark notebooks to process large scale structured and semi structured datasets.
Design, create, and maintain Delta Lake tables, ensuring data reliability, ACID transactions, and schema enforcement.
Develop scalable data workflows and pipelines using Databricks notebooks and orchestration patterns.
Optimize performance of Spark jobs, including tuning partitions, memory usage, caching strategies, and query execution.
Work extensively with PySpark and Spark SQL, choosing the appropriate approach based on use case and performance needs.
Support cloud data migration initiatives, migrating data pipelines from on prem or legacy platforms to Azure Databricks.
Integrate Databricks with upstream and downstream systems (e.g., data sources, storage layers, reporting tools).
Ensure data pipelines are robust, reusable, and maintainable, following enterprise data engineering best practices.
Implement error handling, logging, monitoring, and recovery strategies for production grade data pipelines.
Collaborate with data architects, analysts, and downstream consumers to understand data requirements.
Perform debugging and root cause analysis for data quality, performance, or pipeline failures.
Support testing, validation, and reconciliation of data during development, migration, and production phases.
Follow security, governance, and compliance standards applicable to cloud data platforms.
Actively participate in Agile/Scrum delivery, owning data engineering stories from development through deployment.
Maintain documentation for notebooks, workflows, data models, and migration approaches.
Roles & Responsibilities
Develop and maintain data engineering solutions using Azure Databricks and PySpark.
Create, enhance, and optimize Databricks notebooks for data ingestion, transformation, and aggregation.
Design and manage Delta Lake tables and pipelines supporting analytics and reporting use cases.
Support cloud data migrations, including data validation and performance benchmarking.
Optimize Spark jobs for performance, scalability, and cost efficiency.
Collaborate with platform, DevOps, and data governance teams to ensure environment stability.
Perform data pipeline testing and validation, ensuring correctness and completeness.
Troubleshoot and resolve issues related to Spark jobs, Delta tables, and workflow execution.
Participate in code reviews and enforce data engineering best practices.
Support production deployments and post deployment stabilization.
Provide inputs to data architecture and platform improvement initiatives.
Mentor junior data engineers when required.
Salary Range $120,000-$140,000 Per year
TCS Employee Benefits Summary:
Discretionary Annual Incentive.
Comprehensive Medical Coverage: Medical & Health, Dental & Vision, Disability Planning & Insurance, Pet Insurance Plans.
Family Support: Maternal & Parental Leaves.
Insurance Options: Auto & Home Insurance, Identity Theft Protection.
Convenience & Professional Growth: Commuter Benefits & Certification & amp; Training Reimbursement.
Time Off: Vacation, Time Off, Sick Leave & Holidays.
Legal & Financial Assistance: Legal Assistance, 401K Plan, Performance Bonus, College Fund, Student Loan Refinancing.
#LI-SP1
Location
Louisville, KY
Job Function
TECHNOLOGY
Role
Engineer
Job Id
408315
Desired Skills
Azure
Salary Range
$120,000-$140,000 a year
Desired Candidate Profile
Qualifications : BACHELOR OF COMPUTER SCIENCE