340 Apache Airflow Jobs - Page 8

You have over 8 years of experience and are located in Balewadi, Pune. You possess a strong understanding of Data Architecture and have led data-driven projects. Your expertise includes knowledge of Data Modelling paradigms like Kimball, Inmon, Data Marts, Data Vault, Medallion, etc. Experience with Cloud Based data strategies, particularly AWS, is preferred. Designing data pipelines for ETL with expert knowledge on ingestion, transformation, and data quality is a must, along with hands-on experience in SQL. In-depth understanding of PostGreSQL development, query optimization, and designing indexes is a key requirement. Proficiency in Postgres PL/SQL for complex warehouse workflows is necessary. You should be able to manipulate intermediate to complex SQL and use advanced SQL concepts like RANK, DENSE_RANK, and apply advanced statistical concepts through SQL. Working experience with PostGres SQL extensions like PostGIS is desired. Expertise in writing ETL pipelines combining Python + SQL is required, as well as understanding of data manipulation libraries in Python like Pandas, Polars, DuckDB. Experience in designing Data visualization with tools such as Tableau and PowerBI is desirable. Your responsibilities include participation in designing and developing features in the existing Data Warehouse, providing leadership in establishing connections between Engineering, product, and analytics/data scientists team. Designing, implementing, and updating existing/new batch ETL pipelines, defining and implementing data architecture, and working with various data orchestration tools like Apache Airflow, Dagster, Prefect, and others. Collaboration with engineers and data analysts to build reliable datasets that can be trusted and used by the company is essential. You should be comfortable in a fast-paced start-up environment, passionate about your job, and enjoy a dynamic international working environment. Background or experience in the telecom industry is a plus, though not mandatory. You should have a penchant for automating tasks and enjoy monitoring processes.,

As a Python Machine Learning Engineer at our organization, you will be responsible for designing, developing, and deploying scalable recommender systems using various machine learning algorithms. Your primary focus will be on leveraging machine learning techniques to personalize user experiences, enhance engagement, and drive business outcomes. You will collaborate closely with cross-functional teams to understand business requirements and translate them into actionable machine learning solutions. Conducting thorough exploratory data analysis to identify relevant features and patterns in large-scale datasets will be a crucial part of your role. Additionally, you will implement and optimize machine learning models for performance, scalability, and efficiency while continuously monitoring and evaluating model performance using relevant metrics. To qualify for this role, you should have a Bachelor's or Master's degree in Computer Science, Engineering, Statistics, or a related field, along with 2 to 3 years of hands-on experience in developing machine learning models, specifically focusing on recommender systems. Proficiency in Python programming and popular machine learning libraries/frameworks such as TensorFlow, PyTorch, or Scikit-learn is required. A solid understanding of fundamental machine learning concepts like supervised and unsupervised learning, feature engineering, and model evaluation is essential. Experience working with large-scale datasets, strong analytical and problem-solving skills, and excellent communication abilities are also necessary for this role. You should be able to work with SQL and NoSQL databases to store and retrieve training data, as well as write efficient ETL pipelines to feed real-time and batch ML models using Apache Airflow. Preferred qualifications include experience with cloud computing platforms such as AWS, familiarity with recommendation system evaluation techniques, knowledge of natural language processing techniques, and contributions to open-source machine learning projects or participation in relevant competitions like Kaggle. Experience in MLOps & Deployment (Docker, Airflow) and Cloud Platforms (AWS, GCP, Azure, SageMaker) would be beneficial. This is a full-time position located in Bangalore, Karnataka, and requires in-person work. If you are a talented Python Machine Learning Engineer with a passion for developing and maintaining recommender systems using cutting-edge machine learning algorithms, we encourage you to apply for this exciting opportunity.,

The ideal candidate for this role should have proficiency in Python along with practical experience in working with Databricks (PySpark). It is mandatory for the candidate to possess these skills in order to effectively carry out the responsibilities associated with the position. In addition to the mandatory requirements, it would be advantageous for the candidate to have hands-on experience with Apache Airflow. Furthermore, a working knowledge of databases such as PostgreSQL and MongoDB would be beneficial for the role. Basic experience with cloud technologies like Azure, AWS, and Google Cloud Platform is also considered to be a positive attribute. Overall, the candidate should be well-versed in Python and Databricks (PySpark) to excel in this position. Familiarity with Apache Airflow, PostgreSQL, MongoDB, and cloud technologies would be considered as added advantages for the role.,

As a technically proficient Product Owner with expertise in data and analytics platforms, you will play a crucial role in leading the development of scalable and insight-driven data products. Your responsibilities will involve collaborating closely with data engineers, architects, analysts, and business stakeholders to convert raw data into impactful tools and solutions that drive business intelligence, advanced analytics, and operational efficiency. You will be responsible for defining and managing the product roadmap for data-centric platforms, services, and analytics tools. This will include translating business and analytical goals into detailed technical product requirements, user stories, and epics. By owning and prioritizing the product backlog, you will maximize business value and technical scalability while ensuring seamless delivery of high-performance features through collaboration with engineering, analytics, and design teams. In the realm of analytics and data product development, you will lead the creation of dashboards, reporting tools, self-service analytics, and predictive models. Additionally, you will guide the design and implementation of scalable data pipelines, data lakes, and warehouse architectures using tools such as Snowflake, Redshift, Power BI, Tableau, and Looker. Defining key performance indicators (KPIs) and grounding all features in measurable outcomes will also be key aspects of your role. Acting as a liaison between product, engineering, data science, and business teams, you will partner with engineering and data teams on ETL workflows, data modeling, APIs, and system integration. You will drive delivery using Agile methodologies, ensuring feature launches are supported with documentation, training resources, and adoption strategies. Governance, compliance, and scalability will be crucial areas where you will focus, ensuring product compliance with data governance, GDPR, and security best practices. You will promote scalable architecture and engineering best practices through reusable data models and pipelines, advocating for observability, monitoring, and data quality practices. Preferred Technical Environment: - Languages & Tools: SQL (mandatory), Python or R (preferred), Git, JIRA - BI & Analytics Tools: Power BI, Tableau, Looker - Data Infrastructure: Snowflake, Redshift, BigQuery, dbt, Fivetran, Airbyte - Cloud Platforms: AWS, Azure, or GCP - Agile Tooling: JIRA, Confluence, Miro - Version Control & CI/CD: GitHub, GitLab, Jenkins Qualifications: - Minimum 7 years of experience as a Product Owner or Technical Product Manager for analytics or data products - Proven ability to work with cloud-native data platforms and modern data engineering stacks - Strong understanding of data pipelines, data modeling, ETL orchestration, and warehouse design - Hands-on experience with SQL and at least one modern BI platform - Experience in driving measurable business outcomes through data product initiatives This role offers you the opportunity to make a significant impact by leveraging your technical expertise to drive the development of impactful data products that empower businesses to make informed decisions based on data-driven insights.,

You are an experienced Data Engineer with expertise in PySpark, Snowflake, and AWS. Your role involves designing, developing, and optimizing data pipelines and workflows in a cloud-based environment, utilizing AWS services, PySpark, and Snowflake for data processing and analytics. Your key responsibilities include designing and implementing scalable ETL pipelines using PySpark on AWS, developing and optimizing data workflows for Snowflake integration, managing and configuring AWS services like S3, Lambda, Glue, EMR, and Redshift, collaborating with data analysts and business teams to understand requirements and deliver solutions, ensuring data security and compliance with best practices in AWS and Snowflake environments, monitoring and troubleshooting data pipelines and workflows for performance and reliability, and writing efficient, reusable, and maintainable code for data processing and transformation. You should possess strong experience with AWS services (S3, Lambda, Glue, MSK, etc.), proficiency in PySpark for large-scale data processing, hands-on experience with Snowflake for data warehousing and analytics, solid understanding of SQL and database optimization techniques, knowledge of data lake and data warehouse architectures, familiarity with CI/CD pipelines and version control systems (e.g., Git), strong problem-solving and debugging skills, experience with Terraform or CloudFormation for infrastructure as code, knowledge of Python for scripting and automation, familiarity with Apache Airflow for workflow orchestration, understanding of data governance and security best practices, and certification in AWS or Snowflake is a plus. You should have a Bachelor's degree in Computer Science, Engineering, or a related field with 6 to 10 years of experience, 5+ years of experience in AWS cloud engineering, and 2+ years of experience with PySpark and Snowflake. This position falls under the Technology Job Family Group and specifically in the Digital Software Engineering Job Family. It is a full-time role. Please refer to the above requirements for the most relevant skills needed for this position. For additional skills, you can review the details provided above or reach out to the recruiter for more information. If you require a reasonable accommodation due to a disability to use our search tools or apply for a career opportunity, please review Accessibility at Citi. You can also view Citi's EEO Policy Statement and the Know Your Rights poster.,

You will be part of a team responsible for developing a next-generation Data Analytics Engine that converts raw market and historical data into actionable insights for the electronics supply chain industry. This platform processes high-volume data from suppliers, parts, and trends to provide real-time insights and ML-driven applications. We are seeking an experienced Lead or Staff Data Engineer to assist in shaping and expanding our core data infrastructure. The ideal candidate should have a strong background in designing and implementing scalable ETL pipelines and real-time data systems in AWS and open-source environments such as Airflow, Spark, and Kafka. This role involves taking technical ownership, providing leadership, improving our architecture, enforcing best practices, and mentoring junior engineers. Your responsibilities will include designing, implementing, and optimizing scalable ETL pipelines using AWS-native tools, migrating existing pipelines to open-source orchestration tools, leading data lake and data warehouse architecture design, managing CI/CD workflows, implementing data validation and quality checks, contributing to Infrastructure as Code, and offering technical mentorship and guidance on architectural decisions. To qualify for this role, you should have at least 8 years of experience as a Data Engineer or similar role with production ownership, expertise in AWS tools, deep knowledge of open-source data stack, strong Python programming skills, expert-level SQL proficiency, experience with CI/CD tools, familiarity with Infrastructure as Code, and the ability to mentor engineers and drive architectural decisions. Preferred qualifications include a background in ML/AI pipelines, experience with serverless technologies and containerized deployments, and familiarity with data observability tools and alerting systems. A Bachelor's or Master's degree in Computer Science, Data Engineering, Information Systems, or a related field is preferred. In return, you will have the opportunity to work on impactful supply chain intelligence problems, receive mentorship from experienced engineers and AI product leads, work in a flexible and startup-friendly environment, and enjoy competitive compensation with opportunities for career growth.,

As a Lead Data Engineer with 7-12 years of experience, you will be an integral part of our team, contributing significantly to the design, development, and maintenance of our data infrastructure. Your primary responsibilities will revolve around creating and managing robust data architectures, ETL processes, data warehouses, and utilizing big data and cloud technologies to support our business intelligence and analytics needs. You will lead the design and implementation of data architectures that facilitate data warehousing, integration, and analytics platforms. Developing and optimizing ETL pipelines will be a key aspect of your role, ensuring efficient processing of large datasets and implementing data transformation and cleansing processes to maintain data quality. Your expertise will be crucial in building and maintaining scalable data warehouse solutions using technologies such as Snowflake, Databricks, or Redshift. Additionally, you will leverage AWS Glue and PySpark for large-scale data processing, manage data pipelines with Apache Airflow, and utilize cloud platforms like AWS, Azure, and GCP for data storage, processing, and analytics. Establishing data governance and security best practices, ensuring data integrity, accuracy, and availability, and implementing monitoring and alerting systems are vital components of your responsibilities. Collaborating closely with stakeholders, mentoring junior engineers, and leading data-related projects will also be part of your role. Furthermore, your technical skills should include proficiency in ETL tools like Informatica Power Center, Python, PySpark, SQL, RDBMS platforms, and data warehousing concepts. Soft skills such as excellent communication, leadership, problem-solving, and the ability to manage multiple projects effectively will be essential for success in this role. Preferred qualifications include experience with machine learning workflows, certification in relevant data engineering technologies, and familiarity with Agile methodologies and DevOps practices. Location: Hyderabad Employment Type: Full-time,

As a Data Engineer at our company, you will be responsible for building and maintaining scalable data pipelines and ETL processes using Python and related technologies. Your primary focus will be on developing efficient data pipelines to handle large volumes of data and optimize processing times. Additionally, you will collaborate closely with our team of data scientists and engineers at Matrix Space. To qualify for this role, you should have 2-5 years of experience in data engineering or a related field, with a strong proficiency in Python programming. You must be well-versed in libraries such as Pandas, NumPy, and SQL Alchemy, and have hands-on experience with data engineering tools like Apache Airflow, Luigi, or similar frameworks. A working knowledge of SQL and experience with relational databases such as PostgreSQL or MySQL is also required. In addition to technical skills, we are looking for candidates with strong problem-solving abilities who can work both independently and as part of a team. Effective communication skills are essential, as you will be required to explain technical concepts to non-technical stakeholders. The ability to complete tasks efficiently and effectively is a key trait we value in potential candidates. If you are an immediate joiner and can start within a week, we encourage you to apply for this opportunity. Join our team and be a part of our exciting projects in data engineering.,

You are a talented and driven Machine Learning Engineer with 2-5 years of experience, looking to join a dynamic team in Chennai. Your expertise lies in machine learning principles and hands-on experience in building, deploying, and managing ML models in production environments. In this role, you will focus on MLOps practices and orchestration to ensure robust, scalable, and automated ML pipelines. Your responsibilities will include designing, developing, and implementing end-to-end MLOps pipelines for deploying, monitoring, and managing machine learning models in production. You will use orchestration tools such as Apache Airflow, Kubeflow, AWS Step Functions, or Azure Data Factory to automate ML workflows. Implementing CI/CD practices for ML code, models, and infrastructure will be crucial for ensuring rapid and reliable releases. You will also establish monitoring and alerting systems for deployed ML models, optimize performance, troubleshoot and debug issues across the ML lifecycle, and create and maintain technical documentation. To qualify for this role, you should have a Bachelor's or Master's degree in Computer Science, Data Science, Engineering, or a related quantitative field, along with 2-5 years of professional experience as a Machine Learning Engineer or MLOps Engineer. Your skills should include proficiency in Python and its ML ecosystem, hands-on experience with major cloud platforms and their ML/MLOps services, knowledge of orchestration tools, containerization technologies, CI/CD pipelines, and database systems. Strong problem-solving, analytical, and communication skills are essential for collaborating effectively with Data Scientists, Data Engineers, and Software Developers in an Agile environment.,

At Rhythm, our values serve as the cornerstone of our organization. We are deeply committed to customer success, fostering innovation, and nurturing our employees. These values shape our decisions, actions, and interactions, ensuring that we consistently create a positive impact on the world around us. Rhythm Innovations is currently looking for a skilled and enthusiastic Machine Learning (ML) Developer to conceptualize, create, and implement machine learning models that enhance our supply chain risk management and other cutting-edge solutions. As an ML Developer, you will collaborate closely with our AI Architect and diverse teams to construct intelligent systems that tackle intricate business challenges and further our goal of providing unparalleled customer satisfaction. Key Responsibilities Model Development: Devise, execute, and train machine learning models utilizing cutting-edge algorithms and frameworks like TensorFlow, PyTorch, and scikit-learn. Data Preparation: Process, refine, and convert extensive datasets for the training and assessment of ML models. Feature Engineering: Identify and engineer pertinent features to enhance model performance and precision. Algorithm Optimization: Explore and implement advanced algorithms to cater to specific use cases such as classification, regression, clustering, and anomaly detection. Integration: Coordinate with software developers to integrate ML models into operational systems and guarantee smooth functionality. Performance Evaluation: Assess model performance using suitable metrics and consistently refine for accuracy, efficacy, and scalability. MLOps: Aid in establishing and overseeing CI/CD pipelines for model deployment and monitoring in production environments. Research and Development: Stay abreast of the latest breakthroughs in Gen AI AI/ML technologies and propose inventive solutions. Collaboration: Engage closely with data engineers, product teams, and stakeholders to grasp requirements and deliver customized ML solutions. Requirements Educational Background: Bachelor's in Engineering in Computer Science, Data Science, Artificial Intelligence, or a related field. Experience: 3 to 6 years of practical experience in developing and deploying machine learning models. Technical Skills Proficiency in Python and ML libraries/frameworks (e.g., scikit-learn, TensorFlow, PyTorch). Experience with data manipulation tools like Pandas, NumPy, and visualization libraries such as Matplotlib or Seaborn. Familiarity with big data frameworks (Hadoop, Spark) is advantageous. Knowledge of SQL/NoSQL databases and data pipeline tools (e.g., Apache Airflow). Hands-on experience with cloud platforms (AWS, Azure, Google Cloud) and their Gen AI AI/ML services. Thorough understanding of supervised and unsupervised learning, deep learning, and reinforcement learning. Exposure to MLOps practices and model deployment pipelines. Soft Skills Strong problem-solving and analytical abilities. Effective communication and teamwork skills. Capability to thrive in a dynamic, collaborative environment.,

DifferentByte Technologies Pvt Ltd, a GENAI consultancy firm located in Cochin, Kerala, is in immediate need of a talented Python Developer/Backend Developer to join their team. The primary focus of this position is to contribute to the development of an AI-based high-frequency trading platform. This role presents a unique opportunity for individuals looking to establish a rewarding career in a technology-oriented trading organization. As a Python Developer at DifferentByte Technologies Pvt Ltd, your responsibilities will include writing effective and scalable code, enhancing back-end components to optimize responsiveness and performance, integrating user-facing features into applications, as well as testing and debugging programs. You will also be tasked with improving the functionality of existing systems, implementing security measures, evaluating and prioritizing feature requests, and collaborating with internal teams to understand user needs and offer technical solutions. The ideal candidate should possess a minimum of 2 years of demonstrable experience as a Python Developer within a product-based company. Proficiency in Python frameworks such as Fast API and Flask, along with knowledge of PostgreSQL and familiarity with Supabase, are essential requirements for this role. Additionally, expertise in queuing systems like Apache Kafka and Airflow, as well as strong problem-solving skills, are highly valued. Candidates with experience in artificial intelligence, machine learning, deep learning, and technologies like TensorFlow and PyTorch will be preferred. A degree in B.Tech/B.E. in Computer Science, Engineering, or a related field is a prerequisite. This position is open to candidates currently residing in Kerala or those willing to relocate to Cochin. Apart from the technical qualifications, DifferentByte Technologies Pvt Ltd offers a collaborative and informal work environment with a casual dress code and a flat structure. Joining the highly driven and Agile team at DifferentByte presents an excellent opportunity for career advancement, competitive salary, and a comprehensive benefits package. If you are looking to be a part of an innovative team that values code quality, testing, and making a significant impact on the future of AI-based trading platforms, apply now by contacting hr@differentbyte.in.,

We empower our people to stay resilient and relevant in a constantly changing world. We are looking for individuals who are always seeking creative ways to grow and learn, individuals who aspire to make a real impact, both now and in the future. If this resonates with you, then you would be a valuable addition to our dynamic international team. We are currently seeking a Senior Software Engineer - Data Engineer (AI Solutions). In this role, you will have the opportunity to: - Design, build, and maintain data pipelines to cater to the requirements of various stakeholders, including software developers, data scientists, analysts, and business teams. - Ensure that the data pipelines are modular, resilient, and optimized for performance and low maintenance. - Collaborate with AI/ML teams to support training, inference, and monitoring needs through structured data delivery. - Implement ETL/ELT workflows for structured, semi-structured, and unstructured data using cloud-native tools. - Work with large-scale data lakes, streaming platforms, and batch processing systems to ingest and transform data. - Establish robust data validation, logging, and monitoring strategies to uphold data quality and lineage. - Optimize data infrastructure for scalability, cost-efficiency, and observability in cloud-based environments. - Ensure adherence to governance policies and data access controls across projects. To excel in this role, you should possess the following qualifications and skills: - A Bachelor's degree in Computer Science, Information Systems, or a related field. - Minimum of 4 years of experience in designing and deploying scalable data pipelines in cloud environments. - Proficiency in Python, SQL, and data manipulation tools and frameworks such as Apache Airflow, Spark, dbt, and Pandas. - Practical experience with data lakes, data warehouses (e.g., Redshift, Snowflake, BigQuery), and streaming platforms (e.g., Kafka, Kinesis). - Strong understanding of data modeling, schema design, and data transformation patterns. - Experience with AWS (Glue, S3, Redshift, Sagemaker) or Azure (Data Factory, Azure ML Studio, Azure Storage). - Familiarity with CI/CD for data pipelines and infrastructure-as-code (e.g., Terraform, CloudFormation). - Exposure to building data solutions that support AI/ML pipelines, including feature stores and real-time data ingestion. - Understanding of observability, data versioning, and pipeline testing tools. - Previous engagement with diverse stakeholders, data requirement gathering, and support for iterative development cycles. - Background or familiarity with the Power, Energy, or Electrification sector is advantageous. - Knowledge of security best practices and data compliance policies for enterprise-grade systems. This position is based in Bangalore, offering you the opportunity to collaborate with teams that impact entire cities, countries, and shape the future. Siemens is a global organization comprising over 312,000 individuals across more than 200 countries. We are committed to equality and encourage applications from diverse backgrounds that mirror the communities we serve. Employment decisions at Siemens are made based on qualifications, merit, and business requirements. Join us with your curiosity and creativity to help shape a better tomorrow. Learn more about Siemens careers at: www.siemens.com/careers Discover the Digital world of Siemens here: www.siemens.com/careers/digitalminds,

Golden Eagle IT Technologies Pvt. Ltd. is looking for a skilled Data Engineer with 2 to 4 years of experience to join the team in Indore. The ideal candidate should have a solid background in data engineering, big data technologies, and cloud platforms. As a Data Engineer, you will be responsible for designing, building, and maintaining efficient, scalable, and reliable data pipelines. You will be expected to develop and maintain ETL pipelines using tools like Apache Airflow, Spark, and Hadoop. Additionally, you will design and implement data solutions on AWS, leveraging services such as DynamoDB, Athena, Glue Data Catalog, and SageMaker. Working with messaging systems like Kafka for managing data streaming and real-time data processing will also be part of your responsibilities. Proficiency in Python and Scala for data processing, transformation, and automation is essential. Ensuring data quality and integrity across multiple sources and formats will be a key aspect of your role. Collaboration with data scientists, analysts, and other stakeholders to understand data needs and deliver solutions is crucial. Optimizing and tuning data systems for performance and scalability, as well as implementing best practices for data security and compliance, are also expected. Preferred skills include experience with infrastructure as code tools like Pulumi, familiarity with GraphQL for API development, and exposure to machine learning and data science workflows, particularly using SageMaker. Qualifications for this position include a Bachelor's degree in Computer Science, Information Technology, or a related field, along with 2-4 years of experience in data engineering or a similar role. Proficiency in AWS cloud services and big data technologies, strong programming skills in Python and Scala, knowledge of data warehousing concepts and tools, as well as excellent problem-solving and communication skills are required.,

We are looking for a Data Engineer to join our data team. You will be responsible for managing our master data set, developing reports, and troubleshooting data issues. To excel in this role, attention to detail, experience as a data analyst, and a deep understanding of popular data analysis tools and databases are essential. Your responsibilities include: - Building, maintaining, and managing data pipelines for efficient data flow between systems. - Collaborating with stakeholders to design and manage customized data pipelines. - Testing various ETL (Extract, Transform, Load) tools for data ingestion and processing. - Assisting in scaling the data infrastructure to meet the organization's growing data demands. - Monitoring data pipeline performance and troubleshooting data issues. - Documenting pipeline architectures and workflows for future reference and scaling. - Evaluating data formats, sources, and transformation techniques. - Working closely with data scientists to ensure data availability and reliability for analytics. We require the following skill sets/experience: - Proficiency in Python, PySpark, and Big Data concepts such as Data Lakes and Data Warehouses. - Strong background in SQL. - Familiarity with cloud computing platforms like AWS, Azure, or Google Cloud. - Basic knowledge of containerization technologies like Docker. - Exposure to data orchestration tools like Apache Airflow or Luigi. Pedigree: - Bachelor's degree in Computer Science, Electrical Engineering, or IT.,

Job Description 15 to 18 years. with at-least 3 to 4 years as expertise in ETL, data engineering and Cloud Technologies, with a proven ability to orchestrate cutting-edge technology to connect various applications within the cloud environment in a large development project. Primary Technical Skills: ETL, Apache Spark, AWS EMR, EKS, Serverless, Data Engineering, Distributed Computing, Data Lineage, Apache Airflow, Java 17+, Springboot/ Quarkus, Hibernate ORM, REST, Postgres or any RDBMS, Microservices, Cloud-native development, Secondary Technical Skills Devops : Docker, Kubernetes, CI / CD stack Jenkins or Gitlab CI, Maven, GIT, SonarQube, Nexus, AWS, expertise in at least one Data Engineering tools (e.g. Informatica, Data stage) Apache Airflow, Redis, No-SQL (any Document DB), Kafka / Rabbit MQ, OAUTH2, ARGO, Swagger, OAS Experience / Application of skills Experience in ETL implementation using Cloud technologies, Distributed computing and Big data processing. Orchestrate the integration of Cloud-native principles, Kubernetes, Micro profile specs, Spark framework. Hands-on Java Lead. Strong in OOPs concepts, Java design patterns, Reactive programming, writing High level solutions & Clean architecture. Very strong advocate of coding best practices (SOLID, DRY, Clean Code, Exception handling, TDD, Unit testing, Integration testing). Have implemented common framework for an application/platform (like Exception Library, Security Authentication/Authorization, Auditing, Idempotency, Connectors etc) Experience in implementing HLD, Microservices architecture, design patterns like Resiliency, Service Orchestration, DB per service, CQRS etc. Preferred personal qualities Proactive, Self-starter , Willing to learn new technology. Develop rapid prototypes/PoC/MVP in data integration within cloud environments. Working with team members, mentoring, and guiding them in their career track Excellent problem-solving skills and ability to work in a fast-paced environment. Stay updated with the latest advancements in cloud and data technologies, as well as best practices. Strong leadership and communication skills. Role Engineering Lead Data Engineering Architect Shift : General shift Location : Chennai , Hyderabad and Pune. Those who are interested can drop their resumes at Krishna.Kumaravel@ltimindtree.com

As a Data Engineer specializing in ETL, you should possess a minimum of 7 to 8 years of relevant experience in the field. This position is open across Pan India, and immediate joiners are highly preferred. You will be expected to demonstrate expertise in a range of mandatory skills, including ETL Developer, Synapse, Pyspark, ADF, SSIS, Databricks, SQL, Apache Airflow, and proficiency in Azure & AWS. It is important to note that proficiency in all the mentioned skills is a prerequisite for this role. The selection process for this position involves a total of three rounds - L1 with the External Panel, L2 with the Internal Panel, and L3 with the Client Round. Your responsibilities will include working as an ETL Developer for at least 7+ years, demonstrating proficiency in Pyspark for 5+ years, SSIS for 3 to 4+ years, Databricks for 4 to 4+ years, SQL for 6+ years, Apache Airflow for 4+ years, and experience in Azure and AWS for 3 to 4 years. Additionally, familiarity with Synapse for 3 to 4 years is required to excel in this role.,

We are searching for a Senior Data Engineer with significant experience in developing ETL processes utilizing PySpark Notebooks and Microsoft Fabric, as well as supporting existing legacy SQL Server environments. The perfect candidate will have a solid foundation in Spark-based development, showcase advanced SQL skills, and feel at ease working autonomously, collaboratively within a team, or guiding other developers when necessary, all while possessing excellent communication abilities. The ideal candidate will also demonstrate expertise with Azure Data Services, such as Azure Data Factory, Azure Synapse, or similar tools, familiarity with creating DAG's, implementing activities, and running Apache Airflow, and knowledge of DevOps practices, CI/CD pipelines, and Azure DevOps. Key Responsibilities: - Design, develop, and manage ETL Notebook orchestration pipelines utilizing PySpark and Microsoft Fabric. - Collaborate with data scientists, analysts, and stakeholders to grasp data requirements and provide effective data solutions. - Migrate and integrate data from legacy SQL Server environments into modern data platforms. - Optimize data pipelines and workflows for scalability, efficiency, and reliability. - Provide technical leadership and mentorship to junior developers and team members. - Troubleshoot and resolve complex data engineering issues related to performance, data quality, and system scalability. - Develop, maintain, and uphold data engineering best practices, coding standards, and documentation. - Conduct code reviews and offer constructive feedback to enhance team productivity and code quality. - Support data-driven decision-making processes by ensuring data integrity, availability, and consistency across different platforms. Qualifications: - Bachelors or Masters degree in Computer Science, Data Science, Engineering, or a related field. - 10+ years of experience in data engineering, focusing on ETL development using PySpark or other Spark-based tools. - Proficiency in SQL with extensive experience in complex queries, performance tuning, and data modeling. - Experience with Microsoft Fabric or similar cloud-based data integration platforms is advantageous. - Strong understanding of data warehousing concepts, ETL frameworks, and big data processing. - Familiarity with other data processing technologies (e.g., Hadoop, Hive, Kafka) is a plus. - Experience dealing with both structured and unstructured data sources. - Excellent problem-solving skills and the ability to troubleshoot complex data engineering issues. - Experience with Azure Data Services, including Azure Data Factory, Azure Synapse, or similar tools. - Experience of creating DAG's, implementing activities, and running Apache Airflow. - Familiarity with DevOps practices, CI/CD pipelines, and Azure DevOps. In conclusion, Aspire Systems is a global technology services firm that acts as a trusted technology partner for over 275 clients worldwide. Aspire collaborates with leading enterprises in Banking, Insurance, Retail, and ISVs to help them leverage technology for business transformation in the current digital era. The company's dedication to Attention. Always. reflects its commitment to providing care and attention to both its customers and employees. With over 4900 employees globally and a CMMI Level 3 certification, Aspire Systems operates in North America, LATAM, Europe, Middle East, and Asia Pacific. Aspire Systems has been consistently recognized as one of the Top 100 Best Companies to Work For by the Great Place to Work Institute for the 12th consecutive time. For more information about Aspire Systems, please visit https://www.aspiresys.com/.,

We are looking for a skilled and motivated Data Engineer with at least 4 years of experience in GCP, Teradata, and Data Warehousing. The ideal candidate should have hands-on expertise in developing robust data engineering solutions on Google Cloud Platform (GCP) and working experience with Teradata. You must be proficient in designing and automating scalable data pipelines and possess excellent leadership, communication, and collaboration skills. Your responsibilities will include analyzing source systems, profiling data, and resolving data quality issues. You will be required to gather and comprehend business requirements for data transformation, design, develop, test, and deploy ETL/data pipelines using GCP services and Airflow. Additionally, writing complex SQL queries for data extraction, formatting, and analysis, creating and maintaining Source to Target Mapping, and designing documentation will be part of your role. You will also need to build metadata-driven frameworks for scalable data pipelines, perform unit testing, and document results, utilize DevOps tools for version control and deployment, provide production support, enhancements, and bug fixes, troubleshoot issues, and support ad-hoc business requests. Collaboration with stakeholders to resolve EDW incidents, manage expectations, apply ITIL concepts for incident and problem management, perform data cleaning, transformation, and validation, and stay updated on GCP advancements and industry best practices are also key responsibilities. Requirements: - Minimum 4 years of experience in ETL and Data Warehousing - Hands-on experience with GCP services such as BigQuery, Dataflow, Cloud Storage, etc. - Experience in Apache Airflow for workflow orchestration - Experience in automating ETL solutions - Experience in executing at least 2 GCP Cloud Data Warehousing projects - Exposure to Agile/SAFe methodologies in at least 2 projects - Mid-level proficiency in PySpark and Teradata - Strong SQL skills and experience working with semi-structured data formats like JSON, Parquet, XML - Experience with DevOps tools like GitHub, Jenkins, or similar - Deep understanding of Data Warehousing concepts, data profiling, quality, and mapping Preferred Qualifications: - B.Tech/B.E. in Computer Science or a related field - Google Cloud Professional Data Engineer Certification - Strong leadership and communication skills ,

You have the opportunity to apply for the position of Senior ETL and Feature Engineer at PrivaSapien, based in Bangalore. PrivaSapien is at the forefront of Privacy Enhancing & Responsible AI Technologies, where you will play a crucial role in setting up the big data ecosystem for the world's first privacy red teaming and blue teaming platform. As an individual contributor, you will work on cutting-edge privacy platform requirements with clients globally, spanning across various industry verticals. Joining as one of the early employees, you will receive a significant ESOP option and collaborate with brilliant minds from prestigious institutions such as IISc and IIMs. Your responsibilities will include developing and maintaining ETL pipelines for processing large-scale datasets, creating a Python connector for ETL applications, and demonstrating proficiency in AWS Glue. You will be involved in ETL pipeline development for AI/ML workloads, orchestrating scaling, and resource management. Additionally, you will work on managing unstructured data tasks, optimizing query performance in SQL databases, and integrating multiple databases into the ETL pipeline within a multi-cloud environment. To be eligible for this role, you should have a minimum of 4 years of hands-on experience in setting up ETL and feature engineering pipelines on cloud or big data ecosystems. Proficiency in Apache Spark, pyspark, Apache Airflow, and AWS Glue is essential, along with expertise in at least one ETL tool. Strong programming skills in Python, familiarity with data manipulation libraries, and experience in handling various data types are required. Furthermore, you should possess knowledge in SQL databases, networking, security, and cloud platforms. The interview process will consist of a technical round with the Director, an assessment, an assessment review round with the Senior Backend person, and an HR round. To apply for this opportunity, you need to register or login on the portal, fill out the application form, clear the video screening, and click on "Apply" to be shortlisted. Your profile will then be shared with the client for the interview round upon selection. At Uplers, our aim is to simplify and expedite the hiring process, assisting talents in finding and applying for relevant contractual onsite opportunities. We provide support for any challenges faced during the engagement and assign a dedicated Talent Success Coach to guide you throughout the process. If you are prepared for a new challenge, a conducive work environment, and an opportunity to elevate your career, seize this chance today. We look forward to welcoming you aboard!,

As a Data Engineer at our company, your primary responsibility will be the development and maintenance of scalable and efficient data pipelines and ETL processes using Python and related technologies. You will play a crucial role in optimizing the performance of these pipelines and queries to handle large volumes of data and improve processing times. Collaboration is key in this role, as you will closely work with our team of data scientists and engineers at Matrix Space. To excel in this position, you should have 2-5 years of experience in data engineering or a related field with a strong focus on Python. Proficiency in Python programming is a must, including knowledge of libraries such as Pandas, NumPy, and SQL Alchemy. Additionally, hands-on experience with data engineering tools and frameworks like Apache Airflow, Luigi, or similar is highly desirable. A solid grasp of SQL and experience with relational databases such as PostgreSQL and MySQL will be beneficial. In addition to technical skills, we value certain soft skills in our team members. Problem-solving abilities, the capacity to work both independently and collaboratively, and effective communication skills are essential. You should be able to articulate technical concepts to non-technical stakeholders and demonstrate a proven track record of completing tasks efficiently. If you are an immediate joiner and can commence within a week, we encourage you to apply for this position. Join our team and be part of an exciting journey in data engineering where your skills and expertise will be valued and put to good use.,

You are a Sr. Data Engineer with a strong background in building ELT pipelines and expertise in modern data engineering practices. You are experienced with Databricks and DBT, proficient in SQL and Python, and have a solid understanding of data warehousing methodologies such as Kimball or Data Vault. You are comfortable working with DevOps tools, particularly within AWS, Databricks, and GitLab. Your role involves collaborating with cross-functional teams to design, develop, and maintain scalable data infrastructure and pipelines using Databricks and DBT. Your responsibilities include designing, building, and maintaining scalable ELT pipelines for processing and transforming large datasets efficiently in Databricks. You will implement Kimball data warehousing methodologies or other multi-dimensional modeling approaches using DBT. Leveraging AWS, Databricks, and GitLab, you will implement CI/CD practices for data engineering workflows. Additionally, you will optimize SQL queries and database performance, monitor and fine-tune data pipelines and queries, and ensure compliance with data security, privacy, and governance standards. Key qualifications for this role include 6+ years of data engineering experience, hands-on experience with Databricks and DBT, proficiency in SQL and Python, experience with Kimball data warehousing or Data Vault methodologies, familiarity with DevOps tools and practices, strong problem-solving skills, and the ability to work in a fast-paced, agile environment. Preferred qualifications include experience with Apache Spark for large-scale data processing, familiarity with CI/CD pipelines for data engineering workflows, understanding of orchestration tools like Apache Airflow, and certifications in AWS, Databricks, or DBT. In return, you will receive benefits such as medical insurance for employees, spouse, and children, accidental life insurance, provident fund, paid vacation time, paid holidays, employee referral bonuses, reimbursement for high-speed internet at home, one-month free stay for employees moving from other cities, tax-free benefits, and other bonuses as determined by management.,

As a Data Engineer at Synoptek, you will be responsible for designing, developing, and maintaining robust and scalable data pipelines on the Google Cloud Platform (GCP). You will leverage your hands-on experience with GCP services such as BigQuery, Jitterbit, Cloud Dataflow, Cloud Pub/Sub, and Cloud Storage to build efficient data processing solutions. Collaborating with cross-functional teams, you will translate their data needs into technical requirements, ensuring data quality, integrity, and security throughout the data lifecycle. Your role will involve developing and optimizing ETL/ELT processes to extract, transform, and load data from various sources into data warehouses and data lakes. Additionally, you will build and maintain data models and schemas to support business intelligence and analytics, while troubleshooting data quality issues and performance bottlenecks. To excel in this position, you should have a Bachelor's degree in Computer Science, Engineering, or a related field, along with 3 to 4 years of experience as a Data Engineer focusing on GCP. Proficiency in Python, SQL, and BigQuery is essential, as well as hands-on experience with data ingestion, transformation, and loading tools like Jitterbit and Apache Beam. A strong understanding of data warehousing and data lake concepts, coupled with experience in data modeling and schema design, will be beneficial. The ideal candidate will exhibit excellent problem-solving and analytical skills, working both independently and collaboratively with internal and external teams. Familiarity with acquiring and managing data from various sources, as well as the ability to identify trends in complex datasets and propose business solutions, are key attributes for success in this role. At Synoptek, we value employees who embody our core DNA behaviors, including clarity, integrity, innovation, accountability, and a results-focused mindset. We encourage continuous learning, adaptation, and growth in a fast-paced environment, promoting a culture of teamwork, flexibility, respect, and collaboration. If you have a passion for data engineering, a drive for excellence, and a commitment to delivering impactful results, we invite you to join our dynamic team at Synoptek. Work hard, play hard, and let's achieve superior outcomes together.,

As a Data Engineer, you will be responsible for designing and building efficient data pipelines using Azure Databricks (PySpark). You will implement business logic for data transformation and enrichment at scale, as well as manage and optimize Delta Lake storage solutions. Additionally, you will develop REST APIs using FastAPI to expose processed data and deploy them on Azure Functions for scalable and serverless data access. Your role will also involve developing and managing Airflow DAGs to orchestrate ETL processes, ingesting and processing data from various internal and external sources on a scheduled basis. You will handle data storage and access using PostgreSQL and MongoDB, writing optimized SQL queries to support downstream applications and analytics. Collaboration is key in this role, as you will work cross-functionally with teams to deliver reliable, high-performance data solutions. It is essential to follow best practices in code quality, version control, and documentation to ensure the success of projects. To excel in this position, you should have at least 5 years of hands-on experience as a Data Engineer and strong expertise in Azure Cloud services. Proficiency in Azure Databricks, PySpark, Delta Lake, Python, and FastAPI for API development is required. Experience with Azure Functions for serverless API deployments, managing ETL pipelines using Apache Airflow, and hands-on experience with PostgreSQL and MongoDB are also essential. Strong SQL skills and experience in handling large datasets will be beneficial for this role.,

As a Senior Developer specializing in SnapLogic and Apache Airflow, you will be responsible for designing, developing, and maintaining enterprise-level data integration solutions. Your expertise in ETL development, workflow orchestration, and cloud technologies will be crucial for automating data workflows, optimizing performance, and ensuring the reliability and scalability of data systems. Your key responsibilities will include designing, developing, and managing ETL pipelines using SnapLogic to ensure efficient data transformation and integration across various systems and applications. You will leverage Apache Airflow for workflow automation, job scheduling, and task dependencies to ensure optimized execution and monitoring. Collaboration with cross-functional teams such as Data Engineering, DevOps, and Data Science will be essential to understand data requirements and deliver effective solutions. In this role, you will be involved in designing and implementing data pipeline architectures to support large-scale data processing in cloud environments like AWS, Azure, and GCP. Developing reusable SnapLogic pipelines, integrating with third-party applications and data sources, optimizing pipeline performance, and providing guidance to junior developers will be part of your responsibilities. Additionally, troubleshooting pipeline failures, implementing automated testing, continuous integration (CI), and continuous delivery (CD) practices for data pipelines will be crucial for maintaining high data quality and minimal downtime. The required skills and experience for this role include at least 6 years of hands-on experience in data engineering with a focus on SnapLogic and Apache Airflow. Proficiency in SnapLogic Designer, SnapLogic cloud environment, and Apache Airflow for building data integrations and ETL pipelines is essential. You should have a strong understanding of ETL concepts, data integration, cloud platforms like AWS, Azure, or Google Cloud, data storage systems such as S3, Azure Blob, and Google Cloud Storage, as well as experience with SQL, relational databases, NoSQL databases, REST APIs, and CI/CD pipelines. Your problem-solving skills, ability to work in an Agile development environment, and strong communication and collaboration skills will be valuable assets in this role. By staying current with new SnapLogic features, Airflow upgrades, and industry best practices, you will contribute to the continuous improvement of data integration solutions. Join our team at Virtusa, where teamwork, quality of life, and professional development are values we embody. Be part of a global team that cares about your growth and provides exciting projects, opportunities, and exposure to state-of-the-art technologies throughout your career with us. At Virtusa, great minds come together to nurture new ideas and foster excellence in a dynamic environment.,

The Digital Success Engineering team at our organization is comprised of a diverse group of distributed software engineers, architects, and engineering managers who collaborate to develop Unified Experiences for our Trailblazers. As the pioneers of our products, we leverage Salesforce technology to create an intuitive and expert self-service platform that drives Trailblazer Success. As a Marketing Cloud Engineer (SMTS) in our Customer Engagement Engineering team, you will play a crucial role in providing technical solutions and support to our customers and business users. We are looking for someone with a strong background in the Salesforce Marketing Cloud platform, exceptional technical skills, and the ability to effectively communicate with stakeholders. The ideal candidate should excel in troubleshooting and coordinating operational issues within our Customer Engagement Ecosystem, working closely with various cross-functional teams to develop solutions and workarounds. Your responsibilities will include collaborating with business users, engineering teams, operations teams, and vendor teams to enhance existing and new business processes and systems functionality. You will be responsible for analyzing problems, providing root cause analysis, resolving issues, and planning long-term support strategies. Key Responsibilities: - Adapting to the Digital Success Engineering team structures and culture - Conducting technical requirements gathering, review, and architecture sessions - Translating business requirements into data architecture, data pipelines, and Salesforce Marketing Cloud software design - Providing guidance and thought leadership in resolving complex problems - Designing and implementing robust, scalable solutions for Salesforce Marketing Cloud projects - Triaging, troubleshooting, and ensuring timely issue resolution - Developing and maintaining technical expertise in assigned areas of product functionality - Engaging in continuous feedback with engineering and data platform teams - Communicating effectively with development, leadership, and business users - Collaborating with cross-functional teams to drive successful program execution - Meeting deadlines, prioritizing requests, and managing tasks efficiently - Serving as the primary contact for application support in Pacific time zone Professional Experience/Skills Required: - Bachelor's degree in Computer Science, Software Engineering, or equivalent - 8+ years of hands-on experience in Salesforce Marketing Cloud and related Salesforce Core products - Strong technical proficiency in various technologies including AMPScript, HTML, CSS, JavaScript, SQL, Python, etc. - Experience in systems integrations, APIs, marketing compliance, and security protocols - Ability to work under pressure, adapt quickly, and drive data-driven decisions - Excellent verbal and written communication skills - Proficiency in technical communication and project management - Strong analytical and problem-solving abilities Join our team and be part of a dynamic environment where your technical expertise and problem-solving skills will contribute to the success of our customers and business users.,