147 Apache Airflow Jobs - Page 3

As a Data Engineer, you will be responsible for designing, developing, and maintaining scalable data pipelines using Spark (PySpark or Spark with Scala). Your role will involve building data ingestion and transformation frameworks for structured and unstructured data sources. Collaborating with data analysts, data scientists, and business stakeholders to understand requirements and deliver reliable data solutions will be a key aspect of your responsibilities. Additionally, you will work with large volumes of data to ensure quality, integrity, and consistency, optimizing data workflows for performance, scalability, and cost efficiency on cloud platforms such as AWS, Azure, or GCP. Implementing data quality checks and automation for ETL/ELT pipelines, monitoring and troubleshooting data issues in production, and documenting technical processes, system designs, and operational procedures are also part of your duties. To excel in this role, you should have at least 3 years of experience as a Data Engineer or in a similar role. Hands-on experience with PySpark or Spark using Scala is essential, along with a strong knowledge of SQL for data querying and transformation. You should also have experience working with any cloud platform (AWS, Azure, or GCP), a solid understanding of data warehousing concepts and big data architecture, and familiarity with version control systems like Git. While not mandatory, it would be beneficial to have experience with data orchestration tools like Apache Airflow, Databricks Workflows, or similar, knowledge of Delta Lake, HDFS, or Kafka, familiarity with containerization tools such as Docker or Kubernetes, exposure to CI/CD practices and DevOps principles, and an understanding of data governance, security, and compliance standards. If you are ready to join immediately and possess the required skills and experience, please share your details via email at nitin.patil@ust.com. Act fast for immediate attention!,

Key Skills: Data engineering, Apache Airflow, GCP, BigQuery, GCS, SQL, ETL/ELT, Docker, Kubernetes, data governance, Agile, CI/CD, DevOps, pipeline orchestration, technical leadership. Roles & Responsibilities: Evaluate and provide scalable technical solutions to address complex and interdependent data processes. Ensure data quality and accuracy by implementing data quality checks, data contracts, and governance processes. Collaborate with software development teams and business analysts to understand data requirements and deliver fit-for-purpose data solutions. Lead the team in delivering end-to-end data engineering solutions. Design, develop, and maintain complex applications to support data processing workflows. Develop and manage data pipelines and workflows using Apache Airflow on GCP. Integrate data from various sources into Google BigQuery and Google Cloud Storage (GCS). Write and optimize advanced SQL queries for ETL/ELT processes. Maintain data consistency and troubleshoot issues in data workflows. Create and maintain detailed technical documentation for pipelines and workflows. Mentor junior data engineers and provide technical leadership and support. Lead project planning, execution, and successful delivery of data engineering initiatives. Stay updated with emerging trends and technologies in data engineering and cloud computing. Experience Requirement: 6-8 yeras of experience in leading the design, development, and deployment of complex data pipelines. Strong working knowledge of Apache Airflow on GCP for orchestration. Hands-on experience integrating data into Google BigQuery and GCS from various sources. Proficient in writing and optimizing complex SQL queries for large-scale data processing. Practical knowledge of containerization technologies like Docker and Kubernetes. Experience in implementing data governance and adhering to data security best practices. Familiarity with Agile methodology and working in cross-functional teams. Experience with CI/CD pipelines and DevOps practices for data engineering workflows. Education: B.Tech M.Tech (Dual), B.Tech, M. Tech.

Job Summary Synechron is seeking a skilled PySpark Data Engineer to design, develop, and optimize data processing solutions leveraging modern big data technologies. In this role, you will lead efforts to build scalable data pipelines, support data integration initiatives, and work closely with cross-functional teams to enable data-driven decision-making. Your expertise will contribute to enhancing business insights and operational efficiency, positioning Synechron as a pioneer in adopting emerging data technologies. Software Requirements Required Software Skills: PySpark (Apache Spark with Python) experience in developing data pipelines Apache Spark ecosystem knowledge Python programming (versions 3.7 or higher) SQL and relational database management systems (e.g., PostgreSQL, MySQL) Cloud platforms (preferably AWS or Azure) Version control: GIT Data workflow orchestration tools like Apache Airflow Data management tools: SQL Developer or equivalent Preferred Software Skills: Experience with Hadoop ecosystem components Knowledge of containerization (Docker, Kubernetes) Familiarity with data lake and data warehouse solutions (e.g., AWS S3, Redshift, Snowflake) Monitoring and logging tools (e.g., Prometheus, Grafana) Overall Responsibilities Lead the design and implementation of large-scale data processing solutions using PySpark and related technologies Collaborate with data scientists, analysts, and business teams to understand data requirements and deliver scalable pipelines Mentor junior team members on best practices in data engineering and emerging technologies Evaluate new tools and methodologies to optimize data workflows and improve data quality Ensure data solutions are robust, scalable, and aligned with organizational data governance policies Stay informed on industry trends and technological advancements in big data and analytics Support production environment stability and performance tuning of data pipelines Drive innovative approaches to extract value from large and complex datasets Technical Skills (By Category) Programming Languages: Required: Python (PySpark experience minimum 2 years) Preferred: Scala (for Spark), SQL, Bash scripting Databases/Data Management: Relational databases (PostgreSQL, MySQL) Distributed storage solutions (HDFS, cloud object storage like S3 or Azure Blob Storage) Data warehousing platforms (Snowflake, Redshift preferred) Cloud Technologies: Required: Experience deploying and managing data solutions on AWS or Azure Preferred: Knowledge of cloud-native services like EMR, Data Factory, or Azure Data Lake Frameworks and Libraries: Apache Spark (PySpark) Airflow or similar orchestration tools Data processing frameworks (Kafka, Spark Streaming preferred) Development Tools and Methodologies: Version control with GIT Agile management tools: Jira, Confluence Continuous integration/deployment pipelines (Jenkins, GitLab CI) Security Protocols: Understanding of data security, access controls, and GDPR compliance in cloud environments Experience Requirements Minimum of 5+ years in data engineering, with hands-on PySpark experience Proven track record of developing, deploying, and maintaining scalable data pipelines Experience working with data lakes, data warehouses, and cloud data services Demonstrated leadership in projects involving big data technologies Experience mentoring junior team members and collaborating across teams Prior experience in financial, healthcare, or retail sectors is beneficial but not mandatory Day-to-Day Activities Develop, optimize, and deploy big data pipelines using PySpark and related tools Collaborate with data analysts, data scientists, and business teams to define data requirements Conduct code reviews, troubleshoot pipeline issues, and optimize performance Mentor junior team members on best practices and emerging technologies Design solutions for data ingestion, transformation, and storage Evaluate new tools and frameworks for continuous improvement Maintain documentation, monitor system health, and ensure security compliance Participate in sprint planning, daily stand-ups, and project retrospectives to align priorities Qualifications Bachelors or Masters degree in Computer Science, Information Technology, or related discipline Relevant industry certifications (e.g., AWS Data Analytics, GCP Professional Data Engineer) preferred Proven experience working with PySpark and big data ecosystems Strong understanding of software development lifecycle and data governance standards Commitment to continuous learning and professional development in data engineering technologies Professional Competencies Analytical mindset and problem-solving acumen for complex data challenges Effective leadership and team management skills Excellent communication skills tailored to technical and non-technical audiences Adaptability in fast-evolving technological landscapes Strong organizational skills to prioritize tasks and manage multiple projects Innovation-driven with a passion for leveraging emerging data technologies

Atomicwork is dedicated to revolutionizing the digital workplace experience by integrating people, processes, and platforms through AI automation. Our team is focused on developing a cutting-edge service management platform that empowers growing businesses to streamline operations and achieve business success. We are currently looking for a talented and driven Data Pipeline Engineer to join our team. As a Data Pipeline Engineer, your main responsibility will be to design, construct, and maintain scalable data pipelines that support our enterprise search capabilities. Your efforts will ensure that data from diverse sources is efficiently ingested, processed, and indexed, facilitating seamless and secure search experiences across the organization. We prioritize practical skills and a proactive approach over formal qualifications. While proficiency in programming languages like Python, Java, or Scala is essential, experience with data pipeline frameworks such as Apache Airflow and tools like Apache NiFi is highly valued. Familiarity with search platforms like Elasticsearch or OpenSearch, as well as knowledge of data ingestion, transformation, and indexing processes, are also crucial for this role. Additionally, a strong understanding of enterprise search concepts, data security best practices, and cloud platforms like AWS, GCP, or Azure is required. Experience with Model Context Protocol (MCP) would be advantageous. Your responsibilities as a Data Pipeline Engineer will include designing, developing, and maintaining data pipelines for enterprise search applications, implementing data ingestion processes from various sources, developing data transformation and enrichment processes, integrating with search platforms, ensuring data quality and integrity, monitoring pipeline performance, collaborating with cross-functional teams, implementing security measures, documenting pipeline architecture, processes, and best practices, and staying updated with industry trends in data engineering and enterprise search. At Atomicwork, you have the opportunity to contribute to the company's growth and development, from conception to execution. Our cultural values emphasize agency, taste, ownership, mastery, impatience, and customer obsession, fostering a positive and innovative workplace environment. We offer competitive compensation and benefits, including a fantastic team, convenient offices across five cities, paid time off, comprehensive health insurance, flexible allowances, and annual outings. If you are excited about the opportunity to work with us, click on the apply button to begin your application. Answer a few questions about yourself and your work, and await further communication from us regarding the next steps. If you have any additional queries or information to share, please feel free to reach out to us at careers@atomicwork.com.,

You will play a vital role as an AI/ML Engineer in a pioneering PropTech startup based in Dubai. The project involves developing a cutting-edge digital real estate platform that integrates AI/ML, data analytics, Web3/blockchain, and conversational AI for long-term sales and short-term stays. Your primary responsibility will be to operationalize machine learning models ensuring they are scalable and reliable for our innovative features. Your tasks will include designing and maintaining scalable infrastructure for training and deploying ML models, creating low-latency APIs for production use, managing data pipelines, and overseeing the MLOps lifecycle. Collaboration with data scientists, backend developers, and product managers will be essential to ensure efficient delivery of AI-driven features. Key Responsibilities: - Design, build, and maintain scalable infrastructure for training and deploying machine learning models. - Operationalize ML models such as the "TruValue UAE" AVM and property recommendation engine by creating robust APIs. - Develop and manage data pipelines to provide clean and reliable data for training and real-time inference. - Implement and manage the MLOps lifecycle including CI/CD for models, monitoring for model drift, and automated retraining. - Optimize the performance of ML models for speed and cost-efficiency in a cloud environment. - Collaborate with backend engineers to integrate ML services with the core platform architecture. - Work with data scientists to enhance model efficacy and feasibility. - Build the technical backend for the AI-powered chatbot, integrating it with NLP services and platform data. Required Skills and Experience: - 3-5+ years of experience in Software Engineering, Machine Learning Engineering, or related roles. - Bachelor's or Master's degree in Computer Science, Engineering, or a related technical field. - Strong proficiency in Python and software engineering fundamentals. - Experience deploying ML models in a production environment on major cloud platforms. - Hands-on experience with ML frameworks like TensorFlow, PyTorch, and Scikit-learn. - Experience with data pipelines using tools such as Apache Airflow, Kubeflow Pipelines, or cloud-native solutions. - Collaboration with cross-functional teams to integrate AI solutions into products. - Experience with cloud platforms and containerization (AWS, Azure, GCP, Docker, Kubernetes). Preferred Qualifications: - Experience in PropTech or FinTech sectors. - Direct experience with MLOps tools and platforms (MLflow, Kubeflow, AWS SageMaker, Google AI Platform). - Familiarity with big data technologies (Spark, BigQuery, Redshift). - Experience in building real-time machine learning inference systems. - Strong understanding of microservices architecture. - Experience working collaboratively with data scientists.,

As a Software Engineer, you will be responsible for designing, developing, and maintaining scalable backend services and workflow orchestration components using Python and GoLang. You will collaborate with the Airflow and Temporal team to build and optimize data pipelines and asynchronous job execution frameworks. Your role will involve implementing and managing complex workflow logic using Apache Airflow and Temporal to ensure high code quality through unit testing, integration testing, and code reviews. Additionally, you will work closely with cross-functional teams, including Data Engineering, DevOps, and Platform Engineering. Your contributions will include participating in architectural discussions and decision-making processes to ensure scalable and maintainable systems. It is essential to write clear documentation and actively participate in knowledge-sharing sessions. To excel in this role, you should possess at least 5-7 years of professional software engineering experience. Strong hands-on programming skills in Python and GoLang are required, along with a solid understanding of concurrent and distributed systems. Previous experience with Apache Airflow and/or Temporal.io is highly beneficial. You should also have expertise in designing and developing robust APIs and backend services while being familiar with containerization tools such as Docker and CI/CD practices. A good understanding of the software development lifecycle (SDLC) and Agile methodologies is necessary. Excellent problem-solving, communication, and collaboration skills are key to success in this position. It would be advantageous to have experience with cloud platforms like AWS, GCP, or Azure. Exposure to microservices architecture and event-driven systems, as well as familiarity with monitoring and observability tools, would be considered a plus.,

About the Job : We are seeking a highly skilled and experienced Senior Data Engineer to join our dynamic team. In this pivotal role, you will be instrumental in driving our data engineering initiatives, with a strong emphasis on leveraging Dataiku's capabilities to enhance data processing and analytics. You will be responsible for designing, developing, and optimizing robust data pipelines, ensuring seamless integration of diverse data sources, and maintaining high data quality and accessibility to support our business intelligence and advanced analytics projects. This role requires a unique blend of expertise in traditional data engineering principles, advanced data modeling, and a forward-thinking approach to integrating cutting-AI technologies, particularly LLM Mesh for Generative AI applications. If you are passionate about building scalable data solutions and are eager to explore the cutting edge of AI, we encourage you to apply. Key Responsibilities : - Dataiku Leadership : Drive data engineering initiatives with a strong emphasis on leveraging Dataiku capabilities for data preparation, analysis, visualization, and the deployment of data solutions. - Data Pipeline Development : Design, develop, and optimize robust and scalable data pipelines to support various business intelligence and advanced analytics projects. This includes developing and maintaining ETL/ELT processes to automate data extraction, transformation, and loading from diverse sources. - Data Modeling & Architecture : Apply expertise in data modeling techniques to design efficient and scalable database structures, ensuring data integrity and optimal performance. - ETL/ELT Expertise : Implement and manage ETL processes and tools to ensure efficient and reliable data flow, maintaining high data quality and accessibility. - Gen AI Integration : Explore and implement solutions leveraging LLM Mesh for Generative AI applications, contributing to the development of innovative AI-powered features. - Programming & Scripting : Utilize programming languages such as Python and SQL for data manipulation, analysis, automation, and the development of custom data solutions. - Cloud Platform Deployment : Deploy and manage scalable data solutions on cloud platforms such as AWS or Azure, leveraging their respective services for optimal performance and cost-efficiency. - Data Quality & Governance : Ensure seamless integration of data sources, maintaining high data quality, consistency, and accessibility across all data assets. Implement data governance best practices. - Collaboration & Mentorship : Collaborate closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver impactful solutions. Potentially mentor junior team members. - Performance Optimization : Continuously monitor and optimize the performance of data pipelines and data systems. Required Skills & Experience : - Proficiency in Dataiku : Demonstrable expertise in Dataiku for data preparation, analysis, visualization, and building end-to-end data pipelines and applications. - Expertise in Data Modeling : Strong understanding and practical experience in various data modeling techniques (e.g., dimensional modeling, Kimball, Inmon) to design efficient and scalable database structures. - ETL/ELT Processes & Tools : Extensive experience with ETL/ELT processes and a proven track record of using various ETL tools (e.g., Dataiku's built-in capabilities, Apache Airflow, Talend, SSIS, etc.). - Familiarity with LLM Mesh : Familiarity with LLM Mesh or similar frameworks for Gen AI applications, understanding its concepts and potential for integration. - Programming Languages : Strong proficiency in Python for data manipulation, scripting, and developing data solutions. Solid command of SQL for complex querying, data analysis, and database interactions. - Cloud Platforms : Knowledge and hands-on experience with at least one major cloud platform (AWS or Azure) for deploying and managing scalable data solutions (e.g., S3, EC2, Azure Data Lake, Azure Synapse, etc.). - Gen AI Concepts : Basic understanding of Generative AI concepts and their potential applications in data engineering. - Problem-Solving : Excellent analytical and problem-solving skills with a keen eye for detail. - Communication : Strong communication and interpersonal skills to collaborate effectively with cross-functional teams. Bonus Points (Nice to Have) : - Experience with other big data technologies (e.g., Spark, Hadoop, Snowflake). - Familiarity with data governance and data security best practices. - Experience with MLOps principles and tools. - Contributions to open-source projects related to data engineering or AI. Education : Bachelor's or Master's degree in Computer Science, Data Science, Engineering, or a related quantitative field.

About the Job : We are seeking a highly skilled and experienced Senior Data Engineer to join our dynamic team. In this pivotal role, you will be instrumental in driving our data engineering initiatives, with a strong emphasis on leveraging Dataiku's capabilities to enhance data processing and analytics. You will be responsible for designing, developing, and optimizing robust data pipelines, ensuring seamless integration of diverse data sources, and maintaining high data quality and accessibility to support our business intelligence and advanced analytics projects. This role requires a unique blend of expertise in traditional data engineering principles, advanced data modeling, and a forward-thinking approach to integrating cutting-AI technologies, particularly LLM Mesh for Generative AI applications. If you are passionate about building scalable data solutions and are eager to explore the cutting edge of AI, we encourage you to apply. Key Responsibilities : - Dataiku Leadership : Drive data engineering initiatives with a strong emphasis on leveraging Dataiku capabilities for data preparation, analysis, visualization, and the deployment of data solutions. - Data Pipeline Development : Design, develop, and optimize robust and scalable data pipelines to support various business intelligence and advanced analytics projects. This includes developing and maintaining ETL/ELT processes to automate data extraction, transformation, and loading from diverse sources. - Data Modeling & Architecture : Apply expertise in data modeling techniques to design efficient and scalable database structures, ensuring data integrity and optimal performance. - ETL/ELT Expertise : Implement and manage ETL processes and tools to ensure efficient and reliable data flow, maintaining high data quality and accessibility. - Gen AI Integration : Explore and implement solutions leveraging LLM Mesh for Generative AI applications, contributing to the development of innovative AI-powered features. - Programming & Scripting : Utilize programming languages such as Python and SQL for data manipulation, analysis, automation, and the development of custom data solutions. - Cloud Platform Deployment : Deploy and manage scalable data solutions on cloud platforms such as AWS or Azure, leveraging their respective services for optimal performance and cost-efficiency. - Data Quality & Governance : Ensure seamless integration of data sources, maintaining high data quality, consistency, and accessibility across all data assets. Implement data governance best practices. - Collaboration & Mentorship : Collaborate closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver impactful solutions. Potentially mentor junior team members. - Performance Optimization : Continuously monitor and optimize the performance of data pipelines and data systems. Required Skills & Experience : - Proficiency in Dataiku : Demonstrable expertise in Dataiku for data preparation, analysis, visualization, and building end-to-end data pipelines and applications. - Expertise in Data Modeling : Strong understanding and practical experience in various data modeling techniques (e.g., dimensional modeling, Kimball, Inmon) to design efficient and scalable database structures. - ETL/ELT Processes & Tools : Extensive experience with ETL/ELT processes and a proven track record of using various ETL tools (e.g., Dataiku's built-in capabilities, Apache Airflow, Talend, SSIS, etc.). - Familiarity with LLM Mesh : Familiarity with LLM Mesh or similar frameworks for Gen AI applications, understanding its concepts and potential for integration. - Programming Languages : Strong proficiency in Python for data manipulation, scripting, and developing data solutions. Solid command of SQL for complex querying, data analysis, and database interactions. - Cloud Platforms : Knowledge and hands-on experience with at least one major cloud platform (AWS or Azure) for deploying and managing scalable data solutions (e.g., S3, EC2, Azure Data Lake, Azure Synapse, etc.). - Gen AI Concepts : Basic understanding of Generative AI concepts and their potential applications in data engineering. - Problem-Solving : Excellent analytical and problem-solving skills with a keen eye for detail. - Communication : Strong communication and interpersonal skills to collaborate effectively with cross-functional teams. Bonus Points (Nice to Have) : - Experience with other big data technologies (e.g., Spark, Hadoop, Snowflake). - Familiarity with data governance and data security best practices. - Experience with MLOps principles and tools. - Contributions to open-source projects related to data engineering or AI. Education : Bachelor's or Master's degree in Computer Science, Data Science, Engineering, or a related quantitative field.

Location- Pune, Mumbai, Nagpur, Goa, Noida, Gurgaon, Ahmedabad, Jaipur, Indore, Kolkata, Kochi, Hyderabad, Bangalore, Chennai,) Experience: 5-7 years Notice: 0-15 days Open position: 6 JD: Proven experience with DataStage for ETL development. Strong understanding of data warehousing concepts and best practices. Hands-on experience with Apache Airflow for workflow management. Proficiency in SQL and Python for data manipulation and scripting. Solid knowledge of Unix/Linux shell scripting. Experience with Apache Spark and Databricks for big data processing. Expertise in Snowflake for cloud data warehousing. Familiarity with version control systems (e.g., Git) and CI/CD pipelines. Excellent problem-solving and communication skills.

We are seeking a highly skilled and experienced Snowflake Architect to take charge of designing, developing, and deploying enterprise-grade cloud data solutions. The ideal candidate should possess a solid background in data architecture, cloud data platforms, and Snowflake implementation, along with practical experience in end-to-end data pipeline and data warehouse design. In this role, you will be responsible for leading the architecture, design, and implementation of scalable Snowflake-based data warehousing solutions. You will also be tasked with defining data modeling standards, best practices, and governance frameworks. Collaborating with stakeholders to comprehend data requirements and translating them into robust architectural solutions will be a key part of your responsibilities. Furthermore, you will be required to design and optimize ETL/ELT pipelines utilizing tools like Snowpipe, Azure Data Factory, Informatica, or DBT. Implementing data security, privacy, and role-based access controls within Snowflake is also essential. Providing guidance to development teams on performance tuning, query optimization, and cost management within Snowflake will be part of your duties. Additionally, ensuring high availability, fault tolerance, and compliance across data platforms will be crucial. Mentoring developers and junior architects on Snowflake capabilities is an important aspect of this role. Qualifications and Experience: - 8+ years of overall experience in data engineering, BI, or data architecture, with a minimum of 3+ years of hands-on Snowflake experience. - Expertise in Snowflake architecture, data sharing, virtual warehouses, clustering, and performance optimization. - Strong proficiency in SQL, Python, and cloud data services (e.g., AWS, Azure, or GCP). - Hands-on experience with ETL/ELT tools like ADF, Informatica, Talend, DBT, or Matillion. - Good understanding of data lakes, data mesh, and modern data stack principles. - Experience with CI/CD for data pipelines, DevOps, and data quality frameworks. - Solid knowledge of data governance, metadata management, and cataloging. Desired Skills: - Snowflake certification (e.g., SnowPro Core/Advanced Architect). - Familiarity with Apache Airflow, Kafka, or event-driven data ingestion. - Knowledge of data visualization tools such as Power BI, Tableau, or Looker. - Experience in healthcare, BFSI, or retail domain projects. Please note that this job description is sourced from hirist.tech.,

We are looking for a highly skilled and experienced Senior Data Engineer to take charge of developing complex compliance and supervision models. Your expertise in cloud-based infrastructure, ETL pipeline development, and financial domains will be crucial in creating robust, scalable, and efficient solutions. As a Senior Data Engineer, your key responsibilities will include leading the development of advanced models using AWS services such as EMR, Glue, and Glue Notebooks. You will design, build, and optimize scalable cloud infrastructure solutions, drawing on a minimum of 5 years of experience in cloud infrastructure. Creating, managing, and optimizing ETL pipelines using PySpark for large-scale data processing will also be a core part of your role. In addition, you will be responsible for building and maintaining CI/CD pipelines for deploying and maintaining cloud-based applications, performing detailed data analysis to deliver actionable insights, and collaborating closely with cross-functional teams to ensure alignment with business goals. Operating effectively in agile or hybrid agile environments and enhancing existing frameworks to support evolving business needs will be key aspects of your role. To qualify for this position, you must have a minimum of 5 years of experience with Python programming, 5+ years of experience in cloud infrastructure (particularly AWS), 3+ years of experience with PySpark (including usage with EMR or Glue Notebooks), and 3+ years of experience with Apache Airflow for workflow orchestration. A strong understanding of capital markets, financial systems, or prior experience in the financial domain is essential, along with proficiency in cloud-native technologies and frameworks. Furthermore, familiarity with CI/CD practices and tools like Jenkins, GitLab CI/CD, or AWS CodePipeline, experience with notebooks for interactive development, excellent problem-solving skills, and strong communication and interpersonal skills are required for this role. The ability to thrive in a fast-paced, dynamic environment is also crucial. In return, you will receive standard company benefits. Join us at DATAECONOMY and be part of a fast-growing data & analytics company at the forefront of innovation in the industry.,

At Medtronic, you can embark on a life-long career dedicated to exploration and innovation, all while contributing to the cause of advancing healthcare access and equity for all. Your role will be pivotal in leading with purpose to break down barriers to innovation in a more connected and compassionate world. As a PySpark Data Engineer at Medtronic's new Minimed India Hub, you will play a crucial part in designing, developing, and maintaining data pipelines using PySpark. Collaborating closely with data scientists, analysts, and other stakeholders, your responsibilities will revolve around ensuring the efficient processing and analysis of large datasets, managing complex transformations, and aggregations. This opportunity allows you to make a significant impact within Medtronic's Diabetes business. With the announcement of the intention to separate the Diabetes division to drive future growth and innovation, you will have the chance to operate with increased speed and agility. This move is expected to unlock potential and drive innovation to enhance the impact on patient care. Key Responsibilities: - Design, develop, and maintain scalable and efficient ETL pipelines using PySpark. - Collaborate with data scientists and analysts to understand data requirements and deliver high-quality datasets. - Implement data quality checks, ensure data integrity, and troubleshoot data pipeline issues. - Stay updated with the latest trends and technologies in big data and distributed computing. Required Knowledge and Experience: - Bachelor's degree in computer science, Engineering, or related field. - 4-5 years of experience in data engineering with a focus on PySpark. - Proficiency in Python and Spark, strong coding and debugging skills. - Strong knowledge of SQL and experience with relational databases. - Hands-on experience with cloud platforms such as AWS, Azure, or Google Cloud Platform. - Experience with data warehousing solutions like Redshift, Snowflake, Databricks, or Google BigQuery. - Familiarity with data lake architectures, big data technologies, and data storage solutions. - Excellent problem-solving skills and ability to troubleshoot complex issues. - Strong communication and collaboration skills. Preferred Skills: - Experience with Databricks and orchestration tools like Apache Airflow or AWS Step Functions. - Knowledge of machine learning workflows and data security best practices. - Familiarity with streaming data platforms, real-time data processing, and CI/CD pipelines. Medtronic offers a competitive Salary and flexible Benefits Package. The company values its employees and provides resources and compensation plans to support their growth at every career stage. This position is eligible for the Medtronic Incentive Plan (MIP). About Medtronic: Medtronic is a global healthcare technology leader committed to addressing the most challenging health problems facing humanity. With a mission to alleviate pain, restore health, and extend life, the company unites a team of over 95,000 passionate individuals who work tirelessly to generate real solutions for real people through engineering and innovation.,

About the Job : We are seeking a highly skilled and experienced Senior Data Engineer to join our dynamic team. In this pivotal role, you will be instrumental in driving our data engineering initiatives, with a strong emphasis on leveraging Dataiku's capabilities to enhance data processing and analytics. You will be responsible for designing, developing, and optimizing robust data pipelines, ensuring seamless integration of diverse data sources, and maintaining high data quality and accessibility to support our business intelligence and advanced analytics projects. This role requires a unique blend of expertise in traditional data engineering principles, advanced data modeling, and a forward-thinking approach to integrating cutting-AI technologies, particularly LLM Mesh for Generative AI applications. If you are passionate about building scalable data solutions and are eager to explore the cutting edge of AI, we encourage you to apply. Key Responsibilities : - Dataiku Leadership : Drive data engineering initiatives with a strong emphasis on leveraging Dataiku capabilities for data preparation, analysis, visualization, and the deployment of data solutions. - Data Pipeline Development : Design, develop, and optimize robust and scalable data pipelines to support various business intelligence and advanced analytics projects. This includes developing and maintaining ETL/ELT processes to automate data extraction, transformation, and loading from diverse sources. - Data Modeling & Architecture : Apply expertise in data modeling techniques to design efficient and scalable database structures, ensuring data integrity and optimal performance. - ETL/ELT Expertise : Implement and manage ETL processes and tools to ensure efficient and reliable data flow, maintaining high data quality and accessibility. - Gen AI Integration : Explore and implement solutions leveraging LLM Mesh for Generative AI applications, contributing to the development of innovative AI-powered features. - Programming & Scripting : Utilize programming languages such as Python and SQL for data manipulation, analysis, automation, and the development of custom data solutions. - Cloud Platform Deployment : Deploy and manage scalable data solutions on cloud platforms such as AWS or Azure, leveraging their respective services for optimal performance and cost-efficiency. - Data Quality & Governance : Ensure seamless integration of data sources, maintaining high data quality, consistency, and accessibility across all data assets. Implement data governance best practices. - Collaboration & Mentorship : Collaborate closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver impactful solutions. Potentially mentor junior team members. - Performance Optimization : Continuously monitor and optimize the performance of data pipelines and data systems. Required Skills & Experience : - Proficiency in Dataiku : Demonstrable expertise in Dataiku for data preparation, analysis, visualization, and building end-to-end data pipelines and applications. - Expertise in Data Modeling : Strong understanding and practical experience in various data modeling techniques (e.g., dimensional modeling, Kimball, Inmon) to design efficient and scalable database structures. - ETL/ELT Processes & Tools : Extensive experience with ETL/ELT processes and a proven track record of using various ETL tools (e.g., Dataiku's built-in capabilities, Apache Airflow, Talend, SSIS, etc.). - Familiarity with LLM Mesh : Familiarity with LLM Mesh or similar frameworks for Gen AI applications, understanding its concepts and potential for integration. - Programming Languages : Strong proficiency in Python for data manipulation, scripting, and developing data solutions. Solid command of SQL for complex querying, data analysis, and database interactions. - Cloud Platforms : Knowledge and hands-on experience with at least one major cloud platform (AWS or Azure) for deploying and managing scalable data solutions (e.g., S3, EC2, Azure Data Lake, Azure Synapse, etc.). - Gen AI Concepts : Basic understanding of Generative AI concepts and their potential applications in data engineering. - Problem-Solving : Excellent analytical and problem-solving skills with a keen eye for detail. - Communication : Strong communication and interpersonal skills to collaborate effectively with cross-functional teams. Bonus Points (Nice to Have) : - Experience with other big data technologies (e.g., Spark, Hadoop, Snowflake). - Familiarity with data governance and data security best practices. - Experience with MLOps principles and tools. - Contributions to open-source projects related to data engineering or AI. Education : Bachelor's or Master's degree in Computer Science, Data Science, Engineering, or a related quantitative field.

As a Data Engineer at Synram Software Services Pvt. Ltd., a subsidiary of FG International GmbH, you will be an integral part of our team dedicated to providing innovative IT solutions in ERP systems, E-commerce platforms, Mobile Applications, and Digital Marketing. We are committed to delivering customized solutions that drive success across various industries. In this role, you will be responsible for designing, building, and maintaining scalable data pipelines and infrastructure. Working closely with data analysts, data scientists, and software engineers, you will facilitate data-driven decision-making throughout the organization. Your key responsibilities will include developing, testing, and maintaining data architectures, designing and implementing ETL processes, optimizing data systems, collaborating with cross-functional teams to understand data requirements, ensuring data quality, integrity, and security, automating repetitive data tasks, monitoring and troubleshooting production data pipelines, and documenting systems, processes, and best practices. To excel in this role, you should possess a Bachelor's/Master's degree in Computer Science, Information Technology, or a related field, along with at least 2 years of experience as a Data Engineer or in a similar role. Proficiency in SQL, Python, or Scala is essential, as well as experience with data pipeline tools like Apache Airflow and familiarity with big data tools such as Hadoop and Spark. Hands-on experience with cloud platforms like AWS, GCP, or Azure is preferred, along with knowledge of data warehouse solutions like Snowflake, Redshift, or BigQuery. Preferred qualifications include knowledge of CI/CD for data applications, experience with containerization tools like Docker and Kubernetes, and exposure to data governance and compliance standards. If you are ready to be part of a data-driven transformation journey, apply now to join our team at Synram Software Pvt Ltd. For inquiries, contact us at career@synram.co or +91-9111381555. Benefits of this full-time, permanent role include a flexible schedule, internet reimbursement, leave encashment, day shift with fixed hours and weekend availability, joining bonus, and performance bonus. The ability to commute/relocate to Gwalior, Madhya Pradesh, is preferred. Don't miss the opportunity to contribute your expertise to our dynamic team. The application deadline is 20/07/2025, and the expected start date is 12/07/2025. We look forward to welcoming you aboard for a rewarding and challenging career in data engineering.,

The successful candidate for the Full Stack Developer position at U.S. Pharmacopeial Convention (USP) will have a demonstrated understanding of the organization's mission and a commitment to excellence through inclusive and equitable behaviors and practices. They should possess the ability to quickly build credibility with stakeholders. As a Full Stack Developer, you will be part of the Digital & Innovation group at USP, responsible for building innovative digital products using cutting-edge cloud technologies. Your role will be crucial in creating an amazing digital experience for customers. Your responsibilities will include building scalable applications and platforms using the latest cloud technologies, ensuring systems are regularly reviewed and upgraded based on governance principles and security policies. You will participate in code reviews, architecture discussions, and agile development processes to maintain high-quality, maintainable, and scalable code. Additionally, you will provide technical guidance and mentorship to junior developers and team members, as well as document and communicate technical designs, processes, and solutions to both technical and non-technical stakeholders. To qualify for this role, you should have a Bachelor's or Master's degree in Computer Science, Engineering, or a related field, along with 6-10 years of experience in software development with a focus on cloud computing. Strong knowledge of cloud platforms such as AWS, Azure, and Google Cloud, as well as services like compute, storage, networking, and security, is essential. Experience in leading and mentoring junior software developers, extensive knowledge of Java spring boot applications, and proficiency in programming languages like Python or Node.js are also required. Moreover, you should have experience with AWS/Azure services, containerization technologies like Docker and Kubernetes, front-end technologies like React.js/Node.js, and microservices. Familiarity with cloud architecture patterns, best practices, security principles, data pipelines, and ETL tools is a plus. Experience in leading initiatives related to continuous improvement or new technology implementations, strong analytical and problem-solving skills, and the ability to manage multiple projects and priorities in a fast-paced environment are also desirable attributes. Additional preferences include experience with scientific chemistry nomenclature, pharmaceutical datasets, knowledge graphs, and the ability to explain complex technical issues to a non-technical audience. Strong communication skills, the ability to work independently, make tough decisions, and prioritize tasks are essential for this role. As a Full Stack Developer at USP, you will have supervisory responsibilities and will be eligible for a comprehensive benefits package that includes company-paid time off, healthcare options, and retirement savings. USP is an independent scientific organization dedicated to developing quality standards for medicines, dietary supplements, and food ingredients in collaboration with global health and science authorities. The organization values inclusivity, mentorship, and professional growth, emphasizing Diversity, Equity, Inclusion, and Belonging in its mission to ensure quality in health and healthcare worldwide.,

Join us as a Data Engineer at Barclays, where you will spearhead the evolution of our infrastructure and deployment pipelines, driving innovation and operational excellence. You will harness cutting-edge technology to build and manage robust, scalable and secure infrastructure, ensuring seamless delivery of our digital solutions. To be successful as a Data Engineer, you should have experience with hands-on experience in Pyspark and a strong knowledge of Dataframes, RDD, and SparkSQL. You should also have hands-on experience in developing, testing, and maintaining applications on AWS Cloud. A strong hold on AWS Data Analytics Technology Stack (Glue, S3, Lambda, Lake formation, Athena) is essential. Additionally, you should be able to design and implement scalable and efficient data transformation/storage solutions using Snowflake. Experience in data ingestion to Snowflake for different storage formats such as Parquet, Iceberg, JSON, CSV, etc., is required. Familiarity with using DBT (Data Build Tool) with Snowflake for ELT pipeline development is necessary. Advanced SQL and PL SQL programming skills are a must. Experience in building reusable components using Snowflake and AWS Tools/Technology is highly valued. Exposure to data governance or lineage tools such as Immuta and Alation is an added advantage. Knowledge of Orchestration tools such as Apache Airflow or Snowflake Tasks is beneficial, and familiarity with Abinitio ETL tool is a plus. Some other highly valued skills may include the ability to engage with stakeholders, elicit requirements/user stories, and translate requirements into ETL components. A good understanding of infrastructure setup and the ability to provide solutions either individually or working with teams is essential. Knowledge of Data Marts and Data Warehousing concepts, along with good analytical and interpersonal skills, is required. Implementing Cloud-based Enterprise data warehouse with multiple data platforms along with Snowflake and NoSQL environment to build data movement strategy is also important. You may be assessed on key critical skills relevant for success in the role, such as risk and controls, change and transformation, business acumen, strategic thinking, digital and technology, as well as job-specific technical skills. The role is based out of Chennai. Purpose of the role: To build and maintain the systems that collect, store, process, and analyze data, such as data pipelines, data warehouses, and data lakes to ensure that all data is accurate, accessible, and secure. Accountabilities: - Build and maintenance of data architectures pipelines that enable the transfer and processing of durable, complete, and consistent data. - Design and implementation of data warehouses and data lakes that manage the appropriate data volumes and velocity and adhere to the required security measures. - Development of processing and analysis algorithms fit for the intended data complexity and volumes. - Collaboration with data scientists to build and deploy machine learning models. Analyst Expectations: - Meet the needs of stakeholders/customers through specialist advice and support. - Perform prescribed activities in a timely manner and to a high standard which will impact both the role itself and surrounding roles. - Likely to have responsibility for specific processes within a team. - Lead and supervise a team, guiding and supporting professional development, allocating work requirements, and coordinating team resources. - Demonstrate a clear set of leadership behaviours to create an environment for colleagues to thrive and deliver to a consistently excellent standard. - Manage own workload, take responsibility for the implementation of systems and processes within own work area and participate in projects broader than the direct team. - Execute work requirements as identified in processes and procedures, collaborating with and impacting on the work of closely related teams. - Provide specialist advice and support pertaining to own work area. - Take ownership for managing risk and strengthening controls in relation to the work you own or contribute to. - Deliver work and areas of responsibility in line with relevant rules, regulations, and codes of conduct. - Maintain and continually build an understanding of how all teams in the area contribute to the objectives of the broader sub-function, delivering impact on the work of collaborating teams. - Continually develop awareness of the underlying principles and concepts on which the work within the area of responsibility is based, building upon administrative/operational expertise. - Make judgements based on practice and previous experience. - Assess the validity and applicability of previous or similar experiences and evaluate options under circumstances that are not covered by procedures. - Communicate sensitive or difficult information to customers in areas related specifically to customer advice or day-to-day administrative requirements. - Build relationships with stakeholders/customers to identify and address their needs. All colleagues will be expected to demonstrate the Barclays Values of Respect, Integrity, Service, Excellence, and Stewardship our moral compass, helping us do what we believe is right. They will also be expected to demonstrate the Barclays Mindset to Empower, Challenge, and Drive the operating manual for how we behave.,

Imagine what you could do here. At Apple, new ideas have a way of becoming phenomenal products, services, and customer experiences very quickly. Every single day, people do amazing things at Apple. Do you want to impact the future of Manufacturing here at Apple through cutting edge ML techniques This position involves a wide variety of skills, innovation, and is a rare opportunity to be working on ground breaking, new applications of machine-learning, research and implementation. Ultimately, your work would have a huge impact on billions of users across the globe. You can help inspire change, by using your skills to influence globally recognized products" supply chain. The goal of Apple's Manufacturing & Operations team is to take a vision of a product and turn it into a reality. Through the use of statistics, the scientific process, and machine learning, the team recommends and implements solutions to the most challenging problems. Were looking for experienced machine learning professionals to help us revolutionize how we manufacture Apples amazing products. Put your experience to work in this highly visible role. The Operations Advanced Analytics team is looking for creative and motivated hands-on individual contributors who thrive in a dynamic environment and enjoy working with multi-functional teams. As a member of our team, you will work on applied machine-learning algorithms to seek problems that focus on topics such as classification, regression, clustering, optimizations, and other related algorithms to impact and optimize Apple's supply chain and manufacturing processes. As a part of this role, you would work with the team to build end-to-end machine learning systems and modules, and deploy the models to our factories. You'll be collaborating with Software Engineers, Machine Learning Engineers, Operations, and Hardware Engineering teams across the company. Minimum Qualifications: - 3+ years experience in machine learning algorithms, software engineering, and data mining models with an emphasis on large language models (LLM) or large multimodal models (LMM). - Masters in Machine Learning, Artificial intelligence, Computer Science, Statistics, Operations Research, Physics, Mechanical Engineering, Electrical Engineering, or related field. Preferred Qualifications: - Proven experience in LLM and LMM development, fine-tuning, and application building. Experience with agents and agentic workflows is a major plus. - Experience with modern LLM serving and inference frameworks, including vLLM for efficient model inference and serving. - Hands-on experience with LangChain and LlamaIndex, enabling RAG applications and LLM orchestration. - Strong software development skills with proficiency in Python. Experienced user of ML and data science libraries such as PyTorch, TensorFlow, Hugging Face Transformers, and scikit-learn. - Familiarity with distributed computing, cloud infrastructure, and orchestration tools, such as Kubernetes, Apache Airflow (DAG), Docker, Conductor, Ray for LLM training and inference at scale is a plus. - Deep understanding of transformer-based architectures (e.g., BERT, GPT, LLaMA) and their optimization for low-latency inference. - Ability to meaningfully present results of analyses in a clear and impactful manner, breaking down complex ML/LLM concepts for non-technical audiences. - Experience applying ML techniques in manufacturing, testing, or hardware optimization is a major plus. - Proven experience in leading and mentoring teams is a plus. Submit CV,

We are seeking a highly skilled and experienced Snowflake Architect to take charge of designing, developing, and deploying enterprise-grade cloud data solutions. As the ideal candidate, you should possess a robust background in data architecture, cloud data platforms, and Snowflake implementation. Hands-on experience in end-to-end data pipeline and data warehouse design is essential for this role. Your responsibilities will include leading the architecture, design, and implementation of scalable Snowflake-based data warehousing solutions. You will be tasked with defining data modeling standards, best practices, and governance frameworks. Designing and optimizing ETL/ELT pipelines using tools such as Snowpipe, Azure Data Factory, Informatica, or DBT will be a key aspect of your role. Collaboration with stakeholders to understand data requirements and translating them into robust architectural solutions will also be expected. Additionally, you will be responsible for implementing data security, privacy, and role-based access controls within Snowflake. Guiding development teams on performance tuning, query optimization, and cost management in Snowflake is crucial. Ensuring high availability, fault tolerance, and compliance across data platforms will also fall under your purview. Mentoring developers and junior architects on Snowflake capabilities is another important aspect of this role. In terms of Skills & Experience, we are looking for candidates with at least 8+ years of overall experience in data engineering, BI, or data architecture, and a minimum of 3+ years of hands-on Snowflake experience. Expertise in Snowflake architecture, data sharing, virtual warehouses, clustering, and performance optimization is highly desirable. Strong proficiency in SQL, Python, and cloud data services (e.g., AWS, Azure, or GCP) is required. Hands-on experience with ETL/ELT tools like ADF, Informatica, Talend, DBT, or Matillion is also necessary. A good understanding of data lakes, data mesh, and modern data stack principles is preferred. Experience with CI/CD for data pipelines, DevOps, and data quality frameworks is a plus. Solid knowledge of data governance, metadata management, and cataloging is beneficial. Preferred qualifications include holding a Snowflake certification (e.g., SnowPro Core/Advanced Architect), familiarity with Apache Airflow, Kafka, or event-driven data ingestion, knowledge of data visualization tools such as Power BI, Tableau, or Looker, and experience in healthcare, BFSI, or retail domain projects. If you meet these requirements and are ready to take on a challenging and rewarding role as a Snowflake Architect, we encourage you to apply.,

You will be responsible for designing, developing, and maintaining scalable backend services and workflow orchestration components using Python and GoLang. Collaborating with the Airflow and Temporal team to build and optimize data pipelines and asynchronous job execution frameworks will be a key part of your role. You will also be tasked with implementing and managing complex workflow logic using Apache Airflow and Temporal. Ensuring high code quality through unit testing, integration testing, and code reviews is essential. Working closely with cross-functional teams, including Data Engineering, DevOps, and Platform Engineering, will be necessary. Additionally, contributing to architectural discussions and decision-making processes to ensure scalable and maintainable systems is expected. Writing clear documentation and participating in knowledge-sharing sessions are important aspects of this role. To excel in this position, you should possess 5-7 years of professional software engineering experience. Strong hands-on programming skills with Python and GoLang are required. A solid understanding of concurrent and distributed systems is essential, as well as hands-on experience with Apache Airflow and/or Temporal.io. Experience in designing and developing robust APIs and backend services is necessary. Familiarity with containerization tools (e.g., Docker) and CI/CD practices is a plus. A good understanding of the software development lifecycle (SDLC) and Agile methodologies is preferred. Excellent problem-solving, communication, and collaboration skills are crucial for success in this role. It would be advantageous to have experience with cloud platforms (e.g., AWS, GCP, or Azure), exposure to microservices architecture and event-driven systems, and familiarity with monitoring and observability tools.,

At Medtronic, you can embark on a rewarding career dedicated to exploration and innovation, all while contributing to the advancement of healthcare access and equity for all. As a Digital Engineer at our new Minimed India Hub, you will play a crucial role in leveraging technology to enhance healthcare solutions on a global scale. Specifically, as a PySpark Data Engineer, you will be tasked with designing, developing, and maintaining data pipelines using PySpark. Your collaboration with data scientists, analysts, and stakeholders will be essential in ensuring the efficient processing and analysis of large datasets, as well as handling complex transformations and aggregations. This role offers an exciting opportunity to work within Medtronic's Diabetes business. As the Diabetes division prepares for separation to foster future growth and innovation, you will have the chance to operate with increased speed and agility. By working as a separate entity, there will be a focus on driving meaningful innovation and enhancing the impact on patient care. Your responsibilities will include designing, developing, and maintaining scalable and efficient ETL pipelines using PySpark, working with structured and unstructured data from various sources, optimizing PySpark applications for performance and scalability, collaborating with data scientists and analysts to understand data requirements, implementing data quality checks, monitoring and troubleshooting data pipeline issues, documenting technical specifications, and staying updated on the latest trends and technologies in big data and distributed computing. To excel in this role, you should possess a Bachelor's degree in computer science, engineering, or a related field, along with 4-5 years of experience in data engineering focusing on PySpark. Proficiency in Python and Spark, strong coding and debugging skills, knowledge of SQL and relational databases, hands-on experience with cloud platforms, familiarity with data warehousing solutions, experience with big data technologies, problem-solving abilities, and effective communication and collaboration skills are essential. Preferred skills include experience with Databricks, orchestration tools like Apache Airflow, knowledge of machine learning workflows, understanding of data security and governance best practices, familiarity with streaming data platforms, and knowledge of CI/CD pipelines and version control systems. Medtronic offers a competitive salary and flexible benefits package, along with a commitment to recognizing and supporting employees at every stage of their career and life. As part of the Medtronic team, you will contribute to the mission of alleviating pain, restoring health, and extending life by tackling the most challenging health problems facing humanity. Join us in engineering solutions that make a real difference in people's lives.,

Atomicwork is dedicated to revolutionizing the digital workplace experience by merging people, processes, and platforms through AI automation. The team is currently focused on developing a cutting-edge service management platform that empowers businesses to streamline operations and achieve success. We are in search of a talented and driven Data Pipeline Engineer to become a part of our dynamic team. As a Data Pipeline Engineer, you will play a pivotal role in designing, constructing, and managing scalable data pipelines that support our enterprise search capabilities. Your main responsibility will involve ensuring that data from diverse sources is effectively ingested, processed, and indexed to facilitate seamless and secure search experiences throughout the organization. Qualifications: - Proficiency in programming languages like Python, Java, or Scala. - Strong expertise in data pipeline frameworks and tools such as Apache Airflow and Apache NiFi. - Experience with search platforms like Elasticsearch or OpenSearch. - Familiarity with data ingestion, transformation, and indexing processes. - Understanding of enterprise search concepts including crawling, indexing, and query processing. - Knowledge of data security and access control best practices. - Experience with cloud platforms like AWS, GCP, or Azure and related Backend Engineer - Search/Integrations services. - Knowledge of Model Context Protocol (MCP) is advantageous. - Strong problem-solving and analytical skills. - Excellent communication and collaboration abilities. Responsibilities: - Design, develop, and maintain data pipelines for enterprise search applications. - Implement data ingestion processes from various sources like databases, file systems, and APIs. - Develop data transformation and enrichment processes to prepare data for indexing. - Integrate with search platforms to efficiently index and update data. - Ensure data quality, consistency, and integrity throughout the pipeline. - Monitor pipeline performance and troubleshoot issues as they arise. - Collaborate with cross-functional teams including data scientists, engineers, and product managers. - Implement security measures to safeguard sensitive data during processing and storage. - Document pipeline architecture, processes, and best practices. - Stay abreast of industry trends and advancements in data engineering and enterprise search. At Atomicwork, you have the opportunity to contribute to the company's growth from conceptualization to production. Our cultural values emphasize self-direction, attention to detail, ownership, continuous improvement, impatience for progress, and customer obsession. We offer competitive compensation and benefits including a fantastic team environment, well-located offices in five cities, unlimited sick leaves, comprehensive health insurance with 75% premium coverage, flexible allowances, and annual outings for team bonding. To apply for this role, click on the apply button, answer a few questions about yourself and your work, and await further communication from us regarding the next steps. For any additional inquiries, feel free to reach out to careers@atomicwork.com.,

About the Job : We are seeking a highly skilled and experienced Senior Data Engineer to join our dynamic team. In this pivotal role, you will be instrumental in driving our data engineering initiatives, with a strong emphasis on leveraging Dataiku's capabilities to enhance data processing and analytics. You will be responsible for designing, developing, and optimizing robust data pipelines, ensuring seamless integration of diverse data sources, and maintaining high data quality and accessibility to support our business intelligence and advanced analytics projects. This role requires a unique blend of expertise in traditional data engineering principles, advanced data modeling, and a forward-thinking approach to integrating cutting-AI technologies, particularly LLM Mesh for Generative AI applications. If you are passionate about building scalable data solutions and are eager to explore the cutting edge of AI, we encourage you to apply. Key Responsibilities : - Dataiku Leadership : Drive data engineering initiatives with a strong emphasis on leveraging Dataiku capabilities for data preparation, analysis, visualization, and the deployment of data solutions. - Data Pipeline Development : Design, develop, and optimize robust and scalable data pipelines to support various business intelligence and advanced analytics projects. This includes developing and maintaining ETL/ELT processes to automate data extraction, transformation, and loading from diverse sources. - Data Modeling & Architecture : Apply expertise in data modeling techniques to design efficient and scalable database structures, ensuring data integrity and optimal performance. - ETL/ELT Expertise : Implement and manage ETL processes and tools to ensure efficient and reliable data flow, maintaining high data quality and accessibility. - Gen AI Integration : Explore and implement solutions leveraging LLM Mesh for Generative AI applications, contributing to the development of innovative AI-powered features. - Programming & Scripting : Utilize programming languages such as Python and SQL for data manipulation, analysis, automation, and the development of custom data solutions. - Cloud Platform Deployment : Deploy and manage scalable data solutions on cloud platforms such as AWS or Azure, leveraging their respective services for optimal performance and cost-efficiency. - Data Quality & Governance : Ensure seamless integration of data sources, maintaining high data quality, consistency, and accessibility across all data assets. Implement data governance best practices. - Collaboration & Mentorship : Collaborate closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver impactful solutions. Potentially mentor junior team members. - Performance Optimization : Continuously monitor and optimize the performance of data pipelines and data systems. Required Skills & Experience : - Proficiency in Dataiku : Demonstrable expertise in Dataiku for data preparation, analysis, visualization, and building end-to-end data pipelines and applications. - Expertise in Data Modeling : Strong understanding and practical experience in various data modeling techniques (e.g., dimensional modeling, Kimball, Inmon) to design efficient and scalable database structures. - ETL/ELT Processes & Tools : Extensive experience with ETL/ELT processes and a proven track record of using various ETL tools (e.g., Dataiku's built-in capabilities, Apache Airflow, Talend, SSIS, etc.). - Familiarity with LLM Mesh : Familiarity with LLM Mesh or similar frameworks for Gen AI applications, understanding its concepts and potential for integration. - Programming Languages : Strong proficiency in Python for data manipulation, scripting, and developing data solutions. Solid command of SQL for complex querying, data analysis, and database interactions. - Cloud Platforms : Knowledge and hands-on experience with at least one major cloud platform (AWS or Azure) for deploying and managing scalable data solutions (e.g., S3, EC2, Azure Data Lake, Azure Synapse, etc.). - Gen AI Concepts : Basic understanding of Generative AI concepts and their potential applications in data engineering. - Problem-Solving : Excellent analytical and problem-solving skills with a keen eye for detail. - Communication : Strong communication and interpersonal skills to collaborate effectively with cross-functional teams. Bonus Points (Nice to Have) : - Experience with other big data technologies (e.g., Spark, Hadoop, Snowflake). - Familiarity with data governance and data security best practices. - Experience with MLOps principles and tools. - Contributions to open-source projects related to data engineering or AI. Education : Bachelor's or Master's degree in Computer Science, Data Science, Engineering, or a related quantitative field.

You should have a strong understanding of the tech stack including GCP Services such as BigQuery, Cloud Dataflow, Pub/Sub, Dataproc, and Cloud Storage. Experience with Data Processing tools like Apache Beam (batch/stream), Apache Kafka, and Cloud Dataprep is crucial. Proficiency in programming languages like Python, Java/Scala, and SQL is required. Your expertise should extend to Orchestration tools like Apache Airflow (Cloud Composer) and Terraform, and Security aspects including IAM, Cloud Identity, and Cloud Security Command Center. Knowledge of Containerization using Docker and Kubernetes (GKE) is essential. Familiarity with Machine Learning platforms such as Google AI Platform, TensorFlow, and AutoML is expected. Candidates with certifications like Google Cloud Data Engineer and Cloud Architect are preferred. You should have a proven track record of designing scalable AI/ML systems in production, focusing on high-performance and cost-effective solutions. Strong experience with cloud platforms (Google Cloud, AWS, Azure) and cloud-native AI/ML services like Vertex AI and SageMaker is important. Your role will involve implementing MLOps practices, including model deployment, monitoring, retraining, and version control. Leadership skills are key to guide teams, mentor engineers, and collaborate effectively with cross-functional teams to achieve business objectives. A deep understanding of frameworks like TensorFlow, PyTorch, and Scikit-learn for designing, training, and deploying models is necessary. Experience with data engineering principles, scalable pipelines, and distributed systems (e.g., Apache Kafka, Spark, Kubernetes) is also required. Nice to have requirements include strong leadership and mentorship capabilities to guide teams towards best practices and high-quality deliverables. Excellent problem-solving skills focusing on designing efficient, high-performance systems are valued. Effective project management abilities are necessary to handle multiple initiatives and ensure timely delivery. Collaboration and teamwork are emphasized to foster a positive and productive work environment.,

As a Data Engineer specializing in Data Warehousing and Business Intelligence, you will play a critical role in architecting scalable data warehouses and optimizing ETL pipelines to support analytics and reporting needs. Your expertise in SQL query optimization, database management, and data governance will ensure data accuracy, consistency, and completeness across structured and semi-structured datasets. You will collaborate with cross-functional teams to propose and implement data solutions, leveraging your strong SQL skills and hands-on experience with MySQL, PostgreSQL, and Spark. Your proficiency in tools like Apache Airflow for workflow orchestration and BI platforms such as Power BI, Tableau, and Apache Superset will enable you to create insightful dashboards and reports that drive informed decision-making. A key aspect of your role will involve implementing data governance best practices, defining data standards, access controls, and policies to maintain a well-governed data ecosystem. Your ability to troubleshoot data challenges independently and identify opportunities for system improvements will be essential in ensuring the efficiency and effectiveness of data operations. If you have 5-7 years of experience in data engineering and BI, along with a strong understanding of data modeling techniques, this position at Zenda offers you the opportunity to make a significant impact by designing and developing innovative data solutions. Experience with dbt for data transformations would be a bonus, showcasing your expertise in enhancing data transformation processes.,

As a skilled PySpark Data Engineer, you will be responsible for designing, implementing, and maintaining PySpark-based applications to handle complex data processing tasks, ensure data quality, and integrate with diverse data sources. Your role will involve developing, testing, and optimizing PySpark applications to process, transform, and analyze large-scale datasets from various sources such as relational databases, NoSQL databases, batch files, and real-time data streams. You will collaborate with data analysts, data scientists, and data architects to understand data processing requirements and deliver high-quality data solutions. Your key responsibilities will include designing efficient data transformation and aggregation processes, developing error handling mechanisms for data integrity, optimizing PySpark jobs for performance, and working with distributed datasets in Spark. Additionally, you will design and implement ETL processes to ingest and integrate data from multiple sources, ensuring consistency, accuracy, and performance. You should have a Bachelor's degree in Computer Science or a related field, along with 5+ years of hands-on experience in big data development. Proficiency in PySpark, Apache Spark, and ETL development tools is essential for this role. To succeed in this position, you should have a strong understanding of data processing principles, techniques, and best practices in a big data environment. You must possess excellent analytical and problem-solving skills, with the ability to translate business requirements into technical solutions. Strong communication and collaboration skills are also crucial for effectively working with data analysts, data architects, and other team members. If you are looking to drive the development of robust data processing and transformation solutions within a fast-paced, data-driven environment, this role is ideal for you.,