About The Opportunity
Data Engineer III
In this role, you will focus on building and supporting modern data pipelines, executing complex data migrations, and ensuring data accuracy, consistency, and accessibility across different platforms. You will work with cloud-based data technologies, leveraging advanced SQL, data transformation frameworks, and analytical techniques to support business operations and data-driven decision-making.
You will collaborate closely with engineering teams, analysts, and business stakeholders to develop efficient data workflows, troubleshoot data-quality issues, and deliver actionable insights. You will also explore opportunities to leverage AI-enabled tools to improve data engineering processes, automation, and analytical efficiency.
3–5 years of experience
hybrid in Hyderabad
Responsibilities
-
Design, develop, test, and maintain scalable batch and streaming data pipelines using modern cloud and data-platform technologies.
-
Plan and execute data migration and large-scale data conversion initiatives, including data profiling, source-to-target mapping, transformations, validation, reconciliation, and cutover support.
-
Write, optimize, and troubleshoot complex SQL queries for data transformation, analysis, validation, and performance improvement.
-
Develop and maintain ETL/ELT pipelines to integrate data from multiple sources, including relational databases, files, APIs, and event streams.
-
Implement data-quality checks to ensure completeness, validity, consistency, and accuracy across data pipelines and migrations.
-
Design and maintain data models that support efficient data processing, analytics, and business reporting.
-
Analyze complex datasets to identify patterns, trends, anomalies, and potential data-quality issues, translating findings into actionable insights.
-
Monitor pipeline performance, freshness, completeness, and reliability, proactively investigating failures and resolving production issues.
-
Work with modern data platforms and technologies, including cloud object storage, data warehouses, transformation frameworks, and orchestration tools.
-
Collaborate with cross-functional teams to understand data requirements, align on technical solutions, and support data migration and integration initiatives.
-
Evaluate and apply AI-enabled tools and techniques to improve data analysis, automate validation processes, and optimize data engineering workflows while ensuring accuracy and data security.
-
Contribute to code reviews, technical documentation, and continuous improvements in data reliability, performance, maintainability, and cost efficiency.
-
Support production data workloads, troubleshooting technical issues and implementing sustainable solutions.
Required Qualifications
-
3–5 years of professional experience in Data Engineering, Data Migration, Analytics Engineering, or related technical roles.
-
Strong proficiency in SQL, including complex queries, data transformations, query optimization, data validation, and reconciliation.
-
Demonstrated hands-on experience with data migration or large-scale data conversion, covering data profiling, source-to-target mapping, transformation, validation, reconciliation, and cutover activities.
-
Solid experience designing, developing, and maintaining ETL/ELT pipelines and integrating data from multiple sources.
-
Strong analytical and problem-solving skills, with the ability to investigate complex datasets, identify trends and anomalies, and communicate meaningful insights.
-
Experience with data modeling, data transformation, and database management concepts.
-
Hands-on experience implementing data-quality checks, monitoring data consistency, and resolving data integrity issues.
-
Familiarity with modern cloud data platforms and data engineering technologies, such as AWS, S3, Redshift, Snowflake, dbt, Airflow, Spark, or equivalent tools.
-
Understanding of batch and streaming data-processing concepts.
-
Experience monitoring data pipelines, troubleshooting production failures, and improving data reliability and performance.
-
Awareness of AI capabilities and their practical applications in data engineering and analytics, including the ability to evaluate AI-generated outputs and follow responsible data-handling practices.
-
Experience working with multiple data sources, including databases, files, APIs, and event-based systems.
-
Strong communication and collaboration skills, with the ability to work effectively with technical and business stakeholders.
-
Ability to work independently, manage technical challenges, and deliver reliable data solutions.
Nice To Have Skills
-
Experience working with Data Hub architectures or enterprise data integration platforms.
-
Familiarity with streaming technologies such as Apache Kafka.
-
Experience with lakehouse architectures and related data technologies.
-
Experience developing automation solutions for data migrations, validation, and reconciliation.
-
Familiarity with CI/CD pipelines and infrastructure-as-code practices.
-
Experience optimizing cloud data workloads for performance and cost efficiency.
-
Experience using AI-assisted tools to accelerate SQL development, data analysis, testing, or data-quality validation.
-
Bachelor's degree in Computer Science, Engineering, Information Systems, or a related field, or equivalent practical experience.
Avenue Code reinforces its commitment to privacy and to all the principles guaranteed by the most accurate global data protection laws, such as GDPR, LGPD, CCPA and CPRA. The Candidate data shared with Avenue Code will be kept confidential and will not be transmitted to disinterested third parties, nor will it be used for purposes other than the application for open positions. As a Consultancy company, Avenue Code may share your information with its clients and other Companies from the CompassUol Group to which Avenue Code’s consultants are allocated to perform its services.
