49 Apache Beam Jobs

You will have the opportunity to work at Capgemini, a company that empowers you to shape your career according to your preferences. You will be part of a collaborative community of colleagues worldwide, where you can reimagine what is achievable and contribute to unlocking the value of technology for leading organizations to build a more sustainable and inclusive world. Your Role: - You should have a very good understanding of current work, tools, and technologies being used. - Comprehensive knowledge and clarity on Bigquery, ETL, GCS, Airflow/Composer, SQL, Python are required. - Experience with Fact and Dimension tables, SCD is necessary. - Minimum 3 years of experience in GCP Data Engineering is mandatory. - Proficiency in Java/ Python/ Spark on GCP, with programming experience in Python, Java, or PySpark, SQL. - Hands-on experience with GCS (Cloud Storage), Composer (Airflow), and BigQuery. - Ability to work with handling big data efficiently. Your Profile: - Strong data engineering experience using Java or Python programming languages or Spark on Google Cloud. - Experience in pipeline development using Dataflow or Dataproc (Apache Beam etc). - Familiarity with other GCP services or databases like Datastore, Bigtable, Spanner, Cloud Run, Cloud Functions, etc. - Possess proven analytical skills and a problem-solving attitude. - Excellent communication skills. What you'll love about working here: - You can shape your career with a range of career paths and internal opportunities within the Capgemini group. - Access to comprehensive wellness benefits including health checks, telemedicine, insurance with top-ups, elder care, partner coverage, or new parent support via flexible work. - Opportunity to learn on one of the industry's largest digital learning platforms with access to 250,000+ courses and numerous certifications. About Capgemini: Capgemini is a global business and technology transformation partner, helping organizations accelerate their dual transition to a digital and sustainable world while creating tangible impact for enterprises and society. With a diverse team of over 340,000 members in more than 50 countries, Capgemini leverages its over 55-year heritage to unlock the value of technology for clients across the entire breadth of their business needs. The company delivers end-to-end services and solutions, combining strengths from strategy and design to engineering, fueled by market-leading capabilities in AI, generative AI, cloud, and data, along with deep industry expertise and a strong partner ecosystem.,

Dear Candidate, Greetings of the day!! My name is Arumugam Veera, and I'm reaching out to you regarding an exciting opportunity with TechMango. You can connect with me on below platform. LinkedIn https://www.linkedin.com/in/arumugamv/ Techmango Technology Services is a full-scale software development services company founded in 2014 with a strong focus on emerging technologies. It holds a primary objective of delivering strategic solutions towards the goal of its business partners in terms of technology. We are a full-scale leading Software and Mobile App Development Company. Techmango is driven by the mantra Clients Vision is our Mission. We have a tendency to stick on to the current statement. To be the technologically advanced & most loved organization providing prime quality and cost-efficient services with a long-term client relationship strategy. We are operational in the USA - Chicago, Atlanta, Dubai - UAE, in India - Bangalore, Chennai, Madurai, Trichy. Job Title: GCP Data Engineer/Lead/Architect Job Location: Madurai/chennai - Remote Experience: 4-20 Years Notice Period: Immediate Mode: Remote (Initial 15 Days or 1 month need to be work from madurai office) Job Summary We are seeking a hands-on GCP Data Engineer with deep expertise in real-time streaming data architectures to help design, build, and optimize data pipelines in our Google Cloud Platform (GCP) environment. The ideal candidate will have strong architectural vision and be comfortable rolling up their sleeves to build scalable, low-latency streaming data pipelines using Pub/Sub, Dataflow (Apache Beam) , and BigQuery . Key Responsibilities Architect and implement end-to-end streaming data solutions on GCP using Pub/Sub , Dataflow , and BigQuery . Design real-time ingestion, enrichment, and transformation pipelines for high-volume event data. Work closely with stakeholders to understand data requirements and translate them into scalable designs. Optimize streaming pipeline performance, latency, and throughput. Build and manage orchestration workflows using Cloud Composer (Airflow) . Drive schema design, partitioning, and clustering strategies in BigQuery for both real-time and batch datasets. Define SLAs, monitoring, logging, and alerting for streaming jobs using Cloud Monitoring , Error Reporting , and Stackdriver . Experience with the data modeling. Ensure robust security, encryption, and access controls across all data layers. Collaborate with DevOps for CI/CD automation of data workflows using Terraform , Cloud Build , and Git . Document streaming architecture, data lineage, and deployment runbooks. Required Skills & Experience 5+ years of experience in data engineering or architecture. 3+ years of hands-on GCP data engineering experience. Strong expertise in: Google Pub/Sub Dataflow (Apache Beam) BigQuery (including streaming inserts) Cloud Composer (Airflow) Cloud Storage (GCS) Solid understanding of streaming design patterns , exactly-once delivery , and event-driven architecture . Deep knowledge of SQL and NoSQL data modeling. Hands-on experience with monitoring and performance tuning of streaming jobs. Experience using Terraform or equivalent for infrastructure as code. Familiarity with CI/CD pipelines for data workflows. Arumugam Veera Manager - Talent Acquisition & Business Development LinkedIn: Techmango Technology Services MyLinkedIn: Arumugam Veera Website: www.techmango.net Office Locations: USA - Atlanta, GA | UAE - Dubai | India - Chennai, Trichy & Madurai

Data Engineer (GCP) Company: Moptra Infotech Pvt. Ltd. Location: Noida, Sector 63 Website: https://moptra.com/ Work Schedule: 5 days working (Saturday & Sunday fixed off) About Moptra Infotech Moptra Infotech Pvt. Ltd. is a fast-growing IT services and consulting company headquartered in Noida, India. We specialize in delivering data engineering, cloud solutions, business intelligence, and advanced analytics services to global enterprises. With a strong focus on innovation, scalability, and client satisfaction , Moptra partners with leading organizations across industries to build reliable, future-ready digital ecosystems. Job Responsibilities Develop and maintain robust data pipelines and ETL/ELT processes using Python. Design and implement scalable, high-performance applications. Collaborate with cross-functional teams to define requirements and deliver solutions. Build and manage near real-time data streaming solutions (Pub/Sub, Apache Beam). Participate in code reviews, architecture discussions, and continuous improvement. Monitor and troubleshoot production systems to ensure reliability and performance. Basic Qualifications 5+ years of professional software development experience with Python. Strong understanding of software engineering best practices (testing, version control, CI/CD). Hands-on experience in building and optimizing ETL/ELT pipelines. Proficiency in SQL and strong database concepts. Experience with data processing frameworks (Pandas, etc.). Solid understanding of software design patterns and architectural principles. Experience with unit testing and test automation. Hands-on experience with Google Cloud Platform (GCP) . Experience with CI/CD pipelines and Infrastructure as Code (IaC). Hands-on with containerization technologies (Docker, Kubernetes). Bachelors degree in Computer Science, Engineering, or related field (or equivalent experience). Proven track record of delivering complex software projects. Excellent problem-solving, analytical and communication skills. Preferred Qualifications Experience with GCP services (Cloud Run, Dataflow, Pub/Sub). Exposure to big data technologies (Airflow). Knowledge of data visualization tools and libraries. Experience with CI/CD (GitLab) and IaC (Terraform). Familiarity with Snowflake, BigQuery, or Databricks. GCP Data Engineer Certification i Preferred candidate profile Desired Candidates Must Have Good Communication Skills With Minimum 5 Years of Experience as a Data Engineer ( GCP services Cloud Run, Dataflow, Pub/Sub). •Exposure to big data technologies (Airflow). •Knowledge of data visualization tools and libraries. •Experience with CI/CD (GitLab) and IaC (Terraform). •Familiarity with Snowflake, BigQuery, or Databricks. •GCP Data Engineer Certification is a plus. Note - Need Immediate Joiner or Max 30 Days Call / What's App Resume - 9718978697 Email - Siddharth Mathur Manager Talent Acquisition Moptra Infotech

The GCP Data Engineer will be responsible for designing, developing, and maintaining data pipelines and data infrastructure on Google Cloud Platform (GCP). This role requires expertise in data engineering best practices, cloud architecture, and big data technologies. The ideal candidate will work closely with data scientists, analysts, and other stakeholders to ensure the availability, reliability, and efficiency of data systems, enabling data-driven decision-making across the organization. Key Responsibilities Data Pipeline Development Design, develop, and maintain scalable and efficient ETL/ELT pipelines on GCP. Implement data ingestion processes from various data sources (e.g., APIs, databases, file systems). Ensure data quality, integrity, and reliability throughout the data lifecycle. Cloud Architecture Design and implement data architecture on GCP using services such as BigQuery, Dataflow, Pub/Sub, Cloud Storage, and Cloud Composer. Optimize and manage data storage and retrieval processes to ensure high performance and cost efficiency. Ensure data infrastructure is secure, scalable, and aligned with industry best practices. Big Data Processing Develop and manage large-scale data processing workflows using Apache Beam, Dataflow, and other big data technologies. Implement real-time data streaming solutions using Pub/Sub and Dataflow. Optimize data processing jobs for performance and cost. Collaboration and Communication Work closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions that meet business needs. Communicate technical concepts effectively to both technical and non-technical stakeholders. Participate in agile development processes, including sprint planning, stand-ups, and retrospectives. Data Management and Governance Implement and maintain data governance practices, including data cataloging, metadata management, and data lineage. Ensure compliance with data security and privacy regulations. Monitor and manage data quality and consistency. Troubleshooting and Support Debug and resolve technical issues related to data pipelines and infrastructure. Provide support and maintenance for existing data solutions. Continuously monitor and improve data pipeline performance and reliability. Qualifications Education: Bachelors degree in Computer Science, Information Technology, Data Science, or a related field. Experience: Minimum of 4-12 years of experience in data engineering. Proven experience with GCP data services and tools. Technical Skills: Proficiency in GCP services (e.g., BigQuery, Dataflow, Pub/Sub, Cloud Storage, Cloud Composer). Strong programming skills in languages such as Python Familiarity with big data technologies and frameworks (e.g., Apache Beam, Hadoop, Spark). Knowledge of containerization and orchestration tools (e.g., Docker, Kubernetes) is a plus. Key Competencies Strong problem-solving skills and attention to detail. Excellent communication and teamwork skills. Ability to work in a fast-paced, dynamic environment. Self-motivated and able to work independently as well as part of a team. Continuous learning mindset and a passion for staying up-to-date with emerging technologies.

This role is for a GCP Data Engineer who can build cloud analytics platforms to meet expanding business requirements with speed and quality using lean Agile practices. You will work on analysing and manipulating large datasets supporting the enterprise by activating data assets to support Enabling Platforms and Analytics in the GCP. You will be responsible for designing the transformation and modernization on GCP. Experience with large scale solutions and operationalizing of data warehouses, data lakes and analytics platforms on Google Cloud Platform or other cloud environment is a must. We are looking for candidates who have a broad set of technology skills across these areas and who can demonstrate an ability to design right solutions with appropriate combination of GCP and 3rd party technologies for deploying on the Google Cloud Platform. Responsibilities Develop technical solutions for Data Engineering and work between 1 PM and 10 PM IST to enable more overlap time with European and North American counterparts. This role will work closely with teams in US and as well as Europe to ensure robust, integrated migration aligned with Global Data Engineering patterns and standards. Design and deploying data pipelines with automated data lineage. Develop, reusable Data Engineering patterns. Design and build production data engineering solutions to deliver pipeline patterns using Google Cloud Platform (GCP) services: BigQuery, DataFlow, Pub/Sub, BigTable, Data Fusion, DataProc, Cloud Composer, Cloud SQL, Compute Engine, Cloud Functions, and App Engine. Ensure timely migration of Ford Credit Europe FCE Teradata warehouse to GCP and to enable Teradata platform decommissioning by end 2025 with a strong focus on ensuring continued, robust, and accurate Regulatory Reporting capability. Position Opportunities The Data Engineer role within FC Data Engineering supports the following opportunities for successful individuals: Key player in a high priority program to unlock the potential of Data Engineering Products and Services & secure operational resilience for Ford Credit Europe. Explore and implement leading edge technologies, tooling and software development best practices. Experience of managing data warehousing and product delivery within a financially regulated environment. Experience of collaborative development practices within an open-plan, team-designed environment. Experience of working with third party suppliers / supplier management. Continued personal and professional development with support and encouragement for further certification. Qualifications Essential: 5+ years of experience in data engineering, with a focus on data warehousing and ETL development (including data modelling, ETL processes, and data warehousing principles). 5+ years of SQL development experience. 3+ years of Cloud experience (GCP preferred) with solutions designed and implemented at production scale. Strong understanding of key GCP services, especially those related to data processing (Batch/Real Time) leveraging Terraform, BigQuery, Dataflow, DataFusion, Dataproc, Cloud Build, AirFlow, and Pub/Sub, alongside and storage including Cloud Storage, Bigtable, Cloud Spanner. Excellent problem-solving skills, with the ability to design and optimize complex data pipelines. Strong communication and collaboration skills, capable of working effectively with both technical and non-technical stakeholders as part of a large global and diverse team. Experience developing with micro service architecture from container orchestration framework. Designing pipelines and architectures for data processing. Strong evidence of self-motivation to continuously develop own engineering skills and those of the team. Proven record of working autonomously in areas of high ambiguity, without day-to-day supervisory support. Evidence of a proactive mindset to problem solving and willingness to take the initiative. Strong prioritization, co-ordination, organizational and communication skills, and a proven ability to balance workload and competing demands to meet deadlines. Desired: Professional Certification in GCP (e.g., Professional Data Engineer). Data engineering or development experience gained in a regulated, financial environment. Experience with Teradata to GCP migrations is a plus. Strong expertise in SQL and experience with programming languages such as Python, Java, and/or Apache Beam. Experience of coaching and mentoring Data Engineers. Experience with data security, governance, and compliance best practices in the cloud. An understanding of current architecture standards and digital platform services strategy.,

As a Technical Specialist - Data Engineer at Pune, India, you will be responsible for designing, developing, and delivering significant components of engineering solutions to achieve business goals. Your key responsibilities include actively participating in the design and development of new application features, ensuring solutions are maintainable and integrated successfully, and assisting junior team members as needed. You will develop and deploy source code, provide development activities for projects and technical infrastructure components, debug and provide support to the L3 team, and contribute to quality assurance through unit testing. Additionally, you will ensure architectural changes are implemented, conduct problem analysis, and integrate software components following the defined strategy. Your role also involves collaborating with colleagues in different stages of the Software Development Lifecycle, identifying product integration verifications, and managing maintenance of applications. You should have a Bachelor of Science degree in Computer Science or Software Engineering with strong analytical and communication skills. Proficiency in English, the ability to work in virtual teams, and relevant Financial Services experience are essential. Desirable domain-specific skills include expertise in Java, Scala, Apache Spark, GCP Data Engineering services, workflow orchestrators like Airflow, and automation through Python or Terraform. Knowledge of technologies such as AI/ML, data analysis, and Python development is a plus, along with familiarity with tools like HP ALM, Jira, and Agile processes. You will receive training, coaching, and support for career development, as well as access to a range of flexible benefits. As part of Deutsche Bank Group, we encourage a culture of empowerment, responsibility, commercial thinking, and collaboration. We are committed to promoting a positive, fair, and inclusive work environment where every individual is valued and celebrated. Visit our company website for more information: [https://www.db.com/company/company.htm](https://www.db.com/company/company.htm).,

The company StatusNeo is a global consulting firm that specializes in cutting-edge AI, automation, and cloud-first digital solutions. With a focus on product & platform engineering, the team at StatusNeo includes top-notch software engineers and product designers who are experts in their respective fields. Located in Gurugram, StatusNeo is a Great Place To Work certified organization that values innovation and collaboration. As a GCP Data Engineer at StatusNeo, your primary responsibility will be designing, developing, and maintaining scalable data pipelines and architectures on Google Cloud Platform. Working alongside data scientists, analysts, and other stakeholders, you will ensure that data systems are optimized for performance, reliability, and scalability. Your key responsibilities will include designing and implementing data pipelines using GCP services such as BigQuery, Dataflow, Pub/Sub, and Cloud Storage. You will also utilize tools like Apache Beam, Apache Spark, and Dataproc for data ingestion, processing, and transformation. Managing and optimizing cloud-based data storage solutions, ensuring data quality and integrity, will also be part of your role. Collaboration with data scientists and analysts to understand data requirements and translate them into technical solutions is essential. Additionally, monitoring and troubleshooting data pipelines, implementing improvements for efficiency and reliability, and maintaining comprehensive documentation while adhering to best practices in data engineering are crucial aspects of your job. To be successful in this role, you should have a Bachelor's degree in Computer Science, Engineering, or a related field, along with at least 3 years of experience in data engineering or a similar role. Technical proficiency in GCP services such as pyspark, BigQuery, Dataflow, Pub/Sub, Cloud Storage, SQL, data modeling, scripting languages like Python or Java, and data integration tools like Apache Beam is required. Excellent analytical and troubleshooting abilities are essential for solving complex problems efficiently.,

Dear Candidate, Greetings of the day!! My name is Arumugam Veera, and I'm reaching out to you regarding an exciting opportunity with TechMango. You can connect with me on below platform. LinkedIn https://www.linkedin.com/in/arumugamv/ Mobile Not available on calls & WhatsApp Only: (+91) [+91 6369 002 769] Email: arumugam.veera@techmango.net Techmango Technology Services is a full-scale software development services company founded in 2014 with a strong focus on emerging technologies. It holds a primary objective of delivering strategic solutions towards the goal of its business partners in terms of technology. We are a full-scale leading Software and Mobile App Development Company. Techmango is driven by the mantra Clients Vision is our Mission. We have a tendency to stick on to the current statement. To be the technologically advanced & most loved organization providing prime quality and cost-efficient services with a long-term client relationship strategy. We are operational in the USA - Chicago, Atlanta, Dubai - UAE, in India - Bangalore, Chennai, Madurai, Trichy. Job Title: GCP Data Engineer Job Location: Madurai/chennai - Remote Experience: 4+ Years Notice Period: Immediate Mode: Remote (Initial 15 Days or 1 month need to be work from madurai office) Job Summary We are seeking a hands-on GCP Data Engineer with deep expertise in real-time streaming data architectures to help design, build, and optimize data pipelines in our Google Cloud Platform (GCP) environment. The ideal candidate will have strong architectural vision and be comfortable rolling up their sleeves to build scalable, low-latency streaming data pipelines using Pub/Sub, Dataflow (Apache Beam) , and BigQuery . Key Responsibilities Architect and implement end-to-end streaming data solutions on GCP using Pub/Sub , Dataflow , and BigQuery . Design real-time ingestion, enrichment, and transformation pipelines for high-volume event data. Work closely with stakeholders to understand data requirements and translate them into scalable designs. Optimize streaming pipeline performance, latency, and throughput. Build and manage orchestration workflows using Cloud Composer (Airflow) . Drive schema design, partitioning, and clustering strategies in BigQuery for both real-time and batch datasets. Define SLAs, monitoring, logging, and alerting for streaming jobs using Cloud Monitoring , Error Reporting , and Stackdriver . Experience with the data modeling. Ensure robust security, encryption, and access controls across all data layers. Collaborate with DevOps for CI/CD automation of data workflows using Terraform , Cloud Build , and Git . Document streaming architecture, data lineage, and deployment runbooks. Required Skills & Experience 5+ years of experience in data engineering or architecture. 3+ years of hands-on GCP data engineering experience. Strong expertise in: Google Pub/Sub Dataflow (Apache Beam) BigQuery (including streaming inserts) Cloud Composer (Airflow) Cloud Storage (GCS) Solid understanding of streaming design patterns , exactly-once delivery , and event-driven architecture . Deep knowledge of SQL and NoSQL data modeling. Hands-on experience with monitoring and performance tuning of streaming jobs. Experience using Terraform or equivalent for infrastructure as code. Familiarity with CI/CD pipelines for data workflows. Arumugam Veera Manager - Talent Acquisition & Business Development Mobile & WhatsApp Only: (+91) [+91 6369 002 769] LinkedIn: Techmango Technology Services MyLinkedIn: Arumugam Veera Website: www.techmango.net Office Locations: USA - Atlanta, GA | UAE - Dubai | India - Chennai, Trichy & Madurai

You have more than 7 years of IT experience and have expertise in working with at least two structural databases such as SQL, Oracle, or Postgres, and one NoSQL database. You are capable of collaborating with the Presales team to propose optimal solutions and architectures. Additionally, you possess design experience with BQ, Redshift, and Synapse. Your responsibilities include managing the entire product life cycle, from proposal to delivery, and continuously evaluating architecture enhancements with the delivery team. You are well-versed in security protocols for in-transit data, as well as encryption and decryption of PII data. Moreover, you have a strong understanding of analytics tools for efficient data analysis and have previously been involved in production deployment and support teams. Your technical expertise extends to Big Data tools like Hadoop, Spark, Apache Beam, and Kafka. You are proficient in object-oriented and object function scripting languages such as Python, Java, C++, and Scala. Furthermore, you have hands-on experience in ETL processes and Data Warehousing, along with a comprehensive understanding of both relational and non-relational databases like MySQL, MS SQL Server, Postgres, MongoDB, and Cassandra. Your familiarity with cloud platforms such as AWS, GCP, and Azure is an added advantage, and you have experience in workflow management utilizing tools like Apache Airflow. Ideally, you should be knowledgeable about Design Best Practices for OLTP and OLAP Systems and actively contribute to designing databases and pipelines. You should be adept at suggesting appropriate architectures, including Data Warehouse and Datamesh approaches, as well as understanding data sharing and multi-cloud implementation. Additionally, you should have experience in Load testing methodologies, debugging pipelines, and Delta load handling. Prior exposure to heterogeneous migration projects, multiple Cloud platforms, and additional expertise in Load testing methodologies, debugging pipelines, and Delta load handling would be preferred.,

The Senior GCP Data Engineer role at Deutsche Bank in Bangalore, India, offers you the opportunity to be part of an ambitious initiative focused on Sustainable Finance, ESG Risk Mitigation, and Corporate Sustainability. As Climate Change presents new challenges and opportunities, the bank is investing in developing a Sustainability Technology Platform, Sustainability data products, and sustainability applications to support its goals. To contribute to this initiative, we are seeking a highly motivated and experienced Senior GCP Data Engineer who is passionate about Climate Change and eager to leverage their Technology Skillset in Cloud/Hybrid Architecture. In this role, you will be responsible for designing, developing, and maintaining robust data pipelines that transform raw data into valuable insights for the organization. You will work on GCP services like Dataflow, Dataproc, and Pub/Sub to design and implement data ingestion and transformation processes using tools such as Apache Beam and Apache Spark. Additionally, you will manage and optimize data storage solutions on GCP, including Big Query, Cloud Storage, and Cloud SQL, while also implementing data security and access controls using GCP's Identity and Access Management (IAM) and Cloud Security Command Center. Key responsibilities: - Design, develop, and maintain data pipelines using GCP services. - Implement data ingestion and transformation processes with tools like Apache Beam and Apache Spark. - Manage and optimize data storage solutions on GCP. - Implement data security and access controls using GCP's tools. - Monitor and troubleshoot data pipelines and storage solutions. - Collaborate with data experts, analysts, and product teams. - Automate data processing tasks using Python. - Participate in code reviews and contribute to best practices. - Stay updated on advancements in GCP services and technologies. Your skills and experience should include: - 5+ years of experience as a Data Engineer or similar role. - Expertise in designing, developing, and deploying data pipelines. - Proficiency in GCP and its core data services. - Strong skills in Python & SQL for data manipulation. - Experience with distributed data processing frameworks like Apache Beam or Apache Spark. - Familiarity with data security and access control principles. - Excellent communication, collaboration, and problem-solving abilities. - Ability to work independently, manage multiple projects, and meet deadlines. - Knowledge of Sustainable Finance/ESG Risk/CSRD/Regulatory Reporting is a plus. - Knowledge of cloud infrastructure and data governance best practices is a plus. - Knowledge of Terraform is a plus. At Deutsche Bank, you will receive training, coaching, and support to excel in your career. The culture fosters continuous learning, and you can benefit from a range of flexible benefits tailored to suit your needs. The company promotes a positive, fair, and inclusive work environment where all individuals are welcome to apply and contribute to the success of the organization.,

Role & responsibilities (Apache Beam) . Develop and manage orchestration workflows using Cloud Composer (Airflow) . Ingest, transform, and process large-scale data with a focus on performance, scalability, and compliance. Collaborate with business analysts and healthcare SMEs to understand workflows and translate them into data solutions. Optimize data pipelines for cost efficiency, performance, and scalability. Ensure data quality, lineage, and governance across claims datasets. Integrate structured and unstructured data sources into data lakes/warehouses . Implement data security and HIPAA compliance standards in all processes. Build reusable frameworks for ETL/ELT processes in a regulated healthcare environment. Support Agile delivery model by participating in sprint planning, reviews, and retrospectives. Preferred candidate profile Strong experience with Google Cloud Platform services including: Dataflow (Apache Beam) data pipeline development Cloud Composer (Airflow) – workflow orchestration BigQuery, Cloud Storage, Pub/Sub – data storage and streaming Strong SQL programming skills and performance tuning. Experience with ETL/ELT frameworks and large-scale data processing.

As a GCP Data Engineer-Technical Lead at Birlasoft Office in Bengaluru, India, you will be responsible for designing, building, and maintaining scalable data pipelines and platforms on Google Cloud Platform (GCP) to support business intelligence, analytics, and machine learning initiatives. With a primary focus on Python and GCP technologies such as BigQuery, Dataproc, and Data Flow, you will develop ETL and ELT pipelines while ensuring optimal data manipulation and performance tuning. Your role will involve leveraging data manipulation libraries like Pandas, NumPy, and PySpark, along with SQL expertise for efficient data processing in BigQuery. Additionally, your experience with tools such as Dataflow, Cloud Run, GKE, and Cloud Functions will be crucial in this position. A strong foundation in data modeling, schema design, data governance, and containerization (Docker) for data workloads will further enhance your contributions to our data team. With 5-8 years of experience in Data Engineering and Software Development, including a minimum of 3-4 years working directly with Google Cloud Platform, you will play a key role in driving our data initiatives forward.,

As a Data Engineer at our company, you will be responsible for designing scalable and robust AI/ML systems in production, focusing on high-performance and cost-effective solutions. Your expertise in various technologies, including GCP services like BigQuery, Cloud Dataflow, Pub/Sub, Dataproc, and Cloud Storage, along with programming languages such as Python, Java/Scala, and SQL, will be crucial for the success of our projects. Additionally, your experience with data processing tools like Apache Beam, Apache Kafka, and Cloud Dataprep, as well as orchestration tools like Apache Airflow and Terraform, will play a significant role in implementing efficient data pipelines. Knowledge of security protocols such as IAM, Cloud Identity, and Cloud Security Command Center, and containerization technologies like Docker and Kubernetes (GKE) will also be essential in ensuring data integrity and system security. Moreover, your familiarity with machine learning platforms like Google AI Platform, TensorFlow, and AutoML will enable you to develop and deploy cutting-edge AI models. Certification in Google Cloud Data Engineer and Cloud Architect is preferred, demonstrating your commitment to continuous learning and professional growth. In this role, you will collaborate with cross-functional teams, mentor engineers, and provide leadership to ensure that our projects meet business objectives. Your ability to implement MLOps practices, deploy models, monitor performance, and manage version control will be critical for the success of our AI/ML initiatives. Furthermore, your deep understanding of frameworks such as TensorFlow, PyTorch, and Scikit-learn, coupled with experience in data engineering principles, scalable pipelines, and distributed systems like Apache Kafka, Spark, and Kubernetes, will be invaluable assets in designing and deploying advanced machine learning models. The ideal candidate will possess strong leadership and mentorship capabilities, problem-solving skills, project management abilities, and a collaborative mindset. By fostering a positive and productive work environment, you will contribute to the success of our team and the timely delivery of high-quality solutions. At our company, you will have the opportunity to work on cutting-edge projects, collaborate with a highly motivated team, and enjoy a competitive salary, flexible schedule, and a comprehensive benefits package. Join us at Grid Dynamics, a leading provider of technology consulting and engineering services, and be part of our journey to solve complex technical challenges and drive positive business outcomes for our clients worldwide.,

You should have at least 7-10 years of industry experience, with a minimum of 5 years specifically in machine learning roles. Your proficiency should be advanced in Python and common ML libraries such as TensorFlow, PyTorch, and Scikit-learn. Experience in distributed training, model optimization (quantization, pruning), and inference at scale is essential. It is expected that you have hands-on experience with cloud ML platforms like AWS (SageMaker), GCP (Vertex AI), or Azure ML. Additionally, familiarity with MLOps tooling such as MLflow, TFX, Airflow, or Kubeflow, as well as data engineering frameworks like Spark, dbt, or Apache Beam is required. You should possess a strong understanding of CI/CD for ML, model governance, and post-deployment monitoring including aspects like data drift and model decay. Excellent problem-solving skills, effective communication, and good documentation practices are also key qualities for this role.,

You should have at least 10 years of overall development experience with at least 5 years in a Data Engineering role. Your responsibilities will include building and optimizing big data pipelines, architectures, and data sets. You should have a strong background in writing SQL statements and experience in Spring/Spring Boot framework. Additionally, you should have experience in relational databases like Postgres, Oracle, SnowFlake, BigQuery, and other cloud databases. Experience in implementing web services such as SOAP and RESTful web services is required. Knowledge of frontend frameworks like Angular, jQuery, and Bootstrap is also expected. You should be familiar with real-time and batch data processing, ETL frameworks like Google Cloud Data platform or Apache Beam, and analyzing data to derive insights. Leading small to midsize technical teams, customer-facing experience, and managing deliverables are also part of the role. Good verbal and written communication skills are essential, along with an advanced understanding of modern software development methodologies and software testing methodologies, scripting, and tools. You should have a minimum of three or more full SDLC experiences for web application projects. Experience in Agile development environments and messaging platforms like ActiveMQ would be a plus.,

As a GCP DBT Manager, your primary responsibility will be to collaborate with the team in designing, building, and maintaining data pipelines and transformations using Google Cloud Platform (GCP) and the Data Build Tool (dbt). This role will involve utilizing tools such as BigQuery, Cloud Composer, and Python, requiring a strong foundation in SQL skills and knowledge of data warehousing concepts. Additionally, you will play a crucial role in ensuring data quality, optimizing performance, and working closely with cross-functional teams. Your key responsibilities will include: Data Pipeline Development: - Designing, building, and maintaining ETL/ELT pipelines using dbt and GCP services like BigQuery and Cloud Composer. Data Modeling: - Creating and managing data models and transformations with dbt to ensure efficient and accurate data consumption for analytics and reporting. Data Quality: - Developing and maintaining a data quality framework, including automated testing and cross-dataset validation. Performance Optimization: - Writing and optimizing SQL queries to enhance data processing efficiency within BigQuery. Collaboration: - Collaborating with data engineers, analysts, scientists, and business stakeholders to deliver effective data solutions. Incident Resolution: - Providing support for day-to-day incident and ticket resolution related to data pipelines. Documentation: - Creating and maintaining comprehensive documentation for data pipelines, configurations, and procedures. Cloud Platform Expertise: - Leveraging GCP services like BigQuery, Cloud Composer, Cloud Functions, etc. for efficient data operations. Scripting: - Developing and maintaining SQL/Python scripts for data ingestion, transformation, and automation tasks. Preferred Candidate Profile: Requirements: - 7~12 years of experience in data engineering or a related field. - Strong hands-on experience with Google Cloud Platform (GCP) services, particularly BigQuery. - Proficiency in using dbt for data transformation, testing, and documentation. - Advanced SQL skills for data modeling, performance optimization, and querying large datasets. - Understanding of data warehousing concepts, dimensional modeling, and star schema design. - Experience with ETL/ELT tools and frameworks, such as Apache Beam, Cloud Dataflow, Data Fusion, or Airflow/Composer. In this role, you will be at the forefront of data pipeline development and maintenance, ensuring data quality, performance optimization, and effective collaboration across teams to deliver impactful data solutions using GCP and dbt.,

About Us: Fission Labs is a leading software development company headquartered in Sunnyvale, with offices in Dallas and Hyderabad. We specialize in crafting flexible, agile, and scalable solutions that drive businesses forward. Our comprehensive services include product development, cloud engineering, big data analytics, QA, DevOps consulting, and AI/ML solutions, empowering clients to achieve sustainable digital transformation aligned with their business goals. Fission Labs Website: https://www.fissionlabs.com/ Work Location: Hyderabad Notice Period: Immediate to 30 Days Role Overview: Omada is dedicated to developing next-gen intelligent systems that seamlessly integrate real-time APIs, cloud-native infrastructure, and external AI capabilities. We are seeking a talented Python Engineer with expertise in FastAPI, AWS, and practical experience in utilizing GenAI APIs and data pipelines. Key Responsibilities: Backend & API Development - Design, develop, and maintain robust REST APIs using FastAPI and Python. - Construct scalable microservices that interface with AWS services such as Lambda, EC2, EKS, API Gateway, DynamoDB, and S3. - Implement workflow automation and event-driven pipelines employing tools like Step Functions, SQS, and SNS. - Create real-time and streaming APIs using WebSockets or Kinesis as needed. - Integrate with external GenAI APIs including OpenAI (ChatGPT APIs), Google Gemini APIs, and other third-party AI/ML APIs or services. - Design and execute web crawlers or integrate with crawling frameworks/tools to extract and process structured/unstructured data. Required Skills: - 7-9 years of backend development experience with a strong proficiency in Python. - Demonstrated production-level experience utilizing FastAPI. - Extensive expertise in AWS services, particularly Lambda, EC2, EKS, API Gateway, Step Functions, DynamoDB, S3, and SNS/SQS. - Hands-on experience in calling and managing responses from ChatGPT APIs (OpenAI) and Google Gemini APIs. - Familiarity with writing or integrating web crawlers (e.g., BeautifulSoup, Playwright, Scrapy). - Proficiency in Git and GitHub, encompassing branching strategies, pull requests, and code reviews. - Ability to work independently in a dynamic startup environment. - Prior experience working on Chat Agents. Preferred Qualifications: - Bachelor's degree in Computer Science, Engineering, or a related field. - Familiarity with NoSQL and relational databases (DynamoDB, PostgreSQL, etc.). - Experience in CI/CD workflows, Docker, and Kubernetes. - Bonus: Exposure to distributed data processing frameworks like Apache Beam or Spark. - Bonus: Previous experience integrating with external data and media APIs. Why Join Omada: - Contribute to building API-first systems integrated with cutting-edge AI and cloud technologies. - Shape scalable, real-time backend architecture in a greenfield product. - Collaborate with a modern Python + AWS + GenAI stack.,

As a member of the Platform Observability Engineering team within Ford's Data Platforms and Engineering (DP&E) organization, you will contribute to building and maintaining a top-tier platform for monitoring and observability. This platform focuses on the four golden signalslatency, traffic, errors, and saturationproviding essential data to support operations, root cause analysis, continuous improvement, and cost optimization. You will collaborate with platform architects to help design, develop, and maintain a scalable and reliable platform, ensuring smooth integration with systems used across various teams. Your contributions will be key in improving MTTR and MTTX through increased visibility into system performance. Working with stakeholders, you will integrate observability data into their workflows, develop insightful dashboards and reports, continuously improve platform performance and reliability, optimize costs, and stay updated with industry best practices and technologies. The role focuses on building and maintaining a robust platform rather than developing individual monitoring tools, creating a centralized, reliable source of observability data that empowers data-driven decisions and accelerates incident response across the organization. Responsibilities: - Design and Build Data Pipelines: Architect, develop, and maintain scalable data pipelines and microservices supporting real-time and batch processing on GCP. - Service-Oriented Architecture (SOA) and Microservices: Design and implement SOA and microservices-based architectures for modular, flexible, and maintainable data solutions. - Full-Stack Integration: Contribute to the seamless integration of front-end and back-end components, ensuring robust data access and UI-driven data exploration. - Data Ingestion and Integration: Lead the ingestion and integration of data from various sources into the data platform, ensuring standardized and optimized data for analytics. - GCP Data Solutions: Utilize GCP services (BigQuery, Dataflow, Pub/Sub, Cloud Functions, etc.) to build and manage data platforms meeting business needs. - Data Governance and Security: Implement and manage data governance, access controls, and security best practices while leveraging GCP's native security features. - Performance Optimization: Continuously monitor and improve the performance, scalability, and efficiency of data pipelines and storage solutions. - Collaboration and Best Practices: Define best practices, design patterns, and frameworks for cloud data engineering by closely working with data architects, software engineers, and cross-functional teams. - Automation and Reliability: Automate data platform processes to enhance reliability, reduce manual intervention, and improve operational efficiency. Qualifications: - Technical Skills: Proficiency in Java, Angular, or any JavaScript technology with experience in designing and deploying cloud-based data pipelines and microservices using GCP tools like BigQuery, Dataflow, and Dataproc. - Service-Oriented Architecture and Microservices: Strong understanding of SOA, microservices, and their application within a cloud data platform context. Develop robust, scalable services using Java Spring Boot, Python, Angular, and GCP technologies. - Full-Stack Development: Knowledge of front-end and back-end technologies enabling collaboration on data access and visualization layers (e.g., React, Node.js). - Design and develop RESTful APIs for seamless integration across platform services. - Implement robust unit and functional tests to maintain high standards of test coverage and quality. - Database Management: Experience with relational (e.g., PostgreSQL, MySQL) and NoSQL databases, as well as columnar databases like BigQuery. - Data Governance and Security: Understanding of data governance frameworks and implementing RBAC, encryption, and data masking in cloud environments. - CI/CD and Automation: Familiarity with CI/CD pipelines, Infrastructure as Code (IaC) tools like Terraform, and automation frameworks. - Manage code changes with GitHub and troubleshoot and resolve application defects efficiently. - Ensure adherence to SDLC best practices, independently managing feature design, coding, testing, and production releases. - Problem-Solving: Strong analytical skills with the ability to troubleshoot complex data platform and microservices issues. Certifications (Preferred): GCP Data Engineer, GCP Professional Cloud,

You should possess a minimum of 7-10 years of industry experience, out of which a minimum of 5 years should have been in machine learning roles. Your proficiency in Python and popular ML libraries such as TensorFlow, PyTorch, and Scikit-learn should be advanced. Furthermore, you should have hands-on experience in distributed training, model optimization including quantization and pruning, and inference at scale. Experience with cloud ML platforms like AWS (SageMaker), GCP (Vertex AI), or Azure ML is essential. It is expected that you are familiar with MLOps tooling such as MLflow, TFX, Airflow, or Kubeflow, and data engineering frameworks like Spark, dbt, or Apache Beam. A solid understanding of CI/CD for ML, model governance, and post-deployment monitoring (e.g., data drift, model decay) is crucial for this role. In addition to technical skills, problem-solving abilities, effective communication, and strong documentation skills are highly valued in this position.,

As a Data Engineer at Synoptek, you will be responsible for designing, developing, and maintaining robust and scalable data pipelines on the Google Cloud Platform (GCP). You will leverage your hands-on experience with GCP services such as BigQuery, Jitterbit, Cloud Dataflow, Cloud Pub/Sub, and Cloud Storage to build efficient data processing solutions. Collaborating with cross-functional teams, you will translate their data needs into technical requirements, ensuring data quality, integrity, and security throughout the data lifecycle. Your role will involve developing and optimizing ETL/ELT processes to extract, transform, and load data from various sources into data warehouses and data lakes. Additionally, you will build and maintain data models and schemas to support business intelligence and analytics, while troubleshooting data quality issues and performance bottlenecks. To excel in this position, you should have a Bachelor's degree in Computer Science, Engineering, or a related field, along with 3 to 4 years of experience as a Data Engineer focusing on GCP. Proficiency in Python, SQL, and BigQuery is essential, as well as hands-on experience with data ingestion, transformation, and loading tools like Jitterbit and Apache Beam. A strong understanding of data warehousing and data lake concepts, coupled with experience in data modeling and schema design, will be beneficial. The ideal candidate will exhibit excellent problem-solving and analytical skills, working both independently and collaboratively with internal and external teams. Familiarity with acquiring and managing data from various sources, as well as the ability to identify trends in complex datasets and propose business solutions, are key attributes for success in this role. At Synoptek, we value employees who embody our core DNA behaviors, including clarity, integrity, innovation, accountability, and a results-focused mindset. We encourage continuous learning, adaptation, and growth in a fast-paced environment, promoting a culture of teamwork, flexibility, respect, and collaboration. If you have a passion for data engineering, a drive for excellence, and a commitment to delivering impactful results, we invite you to join our dynamic team at Synoptek. Work hard, play hard, and let's achieve superior outcomes together.,

As a Data Pipeline Architect at our company, you will be responsible for designing, developing, and maintaining optimal data pipeline architecture. You will monitor incidents, perform root cause analysis, and implement appropriate actions to ensure smooth operations. Additionally, you will troubleshoot issues related to abnormal job execution and data corruption, and automate jobs, notifications, and reports for efficiency. Your role will also involve optimizing existing queries, reverse engineering for data research and analysis, and calculating the impact of issues on downstream processes for effective communication. You will support failures, address data quality issues, and ensure the overall health of the environment. Maintaining ingestion and pipeline runbooks, portfolio summaries, and DBAR will be part of your responsibilities. Furthermore, you will enable infrastructure changes, enhancements, and updates roadmap, and build the infrastructure for optimal extraction, transformation, and loading of data from various sources using big data technologies, python, or Web-based APIs. Conducting and participating in code reviews with peers, ensuring effective communication, and understanding requirements will be essential in this role. To qualify for this position, you should hold a Bachelor's degree in Engineering/Computer Science or a related quantitative field. You must have a minimum of 8 years of programming experience with python and SQL, as well as hands-on experience with GCP, BigQuery, Dataflow, Data Warehousing, Apache Beam, and Cloud Storage. Experience with massively parallel processing systems like Spark or Hadoop, source code control systems (GIT), and CI/CD processes is required. Involvement in designing, prototyping, and delivering software solutions within the big data ecosystem, developing generative AI models, and ensuring code quality through reviews are key aspects of this role. Experience with Agile development methodologies, improving data governance and quality, and increasing data reliability are also important. Joining our team at EXL Analytics offers you the opportunity to work in a dynamic and innovative environment alongside experienced professionals. You will gain insights into various business domains, develop teamwork and time-management skills, and receive training in analytics tools and techniques. Our mentoring program and growth opportunities ensure that you have the support and guidance needed to excel in your career. Sky is the limit for our team members, and the experiences gained at EXL Analytics pave the way for personal and professional development within our company and beyond.,

As the Architect IT Strategy, VP at Deutsche Bank Group based in Pune, India, your role involves designing applications, defining modernization approaches, developing application code, and implementing technical solutions to address business problems and meet specified requirements and design principles. Under our flexible scheme, you will enjoy benefits such as a best-in-class leave policy, gender-neutral parental leaves, childcare assistance benefit reimbursement, flexible working arrangements, sponsorship for industry-relevant certifications and education, employee assistance program, comprehensive hospitalization and life insurance, and health screening for individuals aged 35 years and above. Your key responsibilities will include managing work across various areas of the bank's IT platform, planning and developing engineering solutions, ensuring reliability and resiliency, fostering maintainability and reusability, reviewing engineering plans and quality, participating in industry forums, and leading an application development team. Key requirements for this role include technical leadership experience, expertise in designing and developing cloud-native services in GCP/AWS, proficiency in Java/J2EE, Spring Boot, scripting languages like Apache Beam or Python, knowledge of GCP/AWS managed services, experience with Agile methodologies, and strong communication skills. To be successful in this position, you should have 15 to 22 years of overall IT experience, an engineering degree or post-graduation, and the ability to manage stakeholders" expectations. Desired behaviors include being a strong team player, result-oriented, conflict resolver, and possessing excellent communication and collaboration skills. A bachelor's or master's degree in Computer Science, Information Systems, or related discipline is required, or an equivalent combination of education and relevant work experience. You will receive training and development, coaching and support from experts in your team, and a culture of continuous learning to aid your career progression. If you are a motivated individual who thrives in a collaborative and diverse environment, eager to excel in your career, and comfortable with navigating ambiguity to extract meaningful insights, we welcome your application for the Architect IT Strategy, VP position at Deutsche Bank Group in Pune, India.,

As a GCP Data Engineer in Australia, you will be responsible for leveraging your experience in Google Cloud Platform (GCP) to handle various aspects of data engineering. Your role will involve working on data migration projects from legacy systems such as SQL and Oracle. You will also be designing and building ETL pipelines for data lake and data warehouse solutions on GCP. In this position, your expertise in GCP data and analytics services will be crucial. You will work with tools like Cloud Dataflow, Cloud Dataprep, Apache Beam/Cloud composer, Cloud BigQuery, Cloud Fusion, Cloud PubSub, Cloud storage, and Cloud Functions. Additionally, you will utilize Cloud Native GCP CLI/gsutil for operations and scripting languages like Python and SQL to enhance data processing efficiencies. Furthermore, your experience with data governance practices, metadata management, data masking, and encryption will be essential. You will utilize GCP tools such as Cloud Data Catalog and GCP KMS tools to ensure data security and compliance. Overall, this role requires a strong foundation in GCP technologies and a proactive approach to data engineering challenges in a dynamic environment.,

The ideal candidate for the position will have the responsibility of designing, developing, and maintaining an optimal data pipeline architecture. You will be required to monitor incidents, perform root cause analysis, and implement appropriate actions to solve issues related to abnormal job execution and data corruption conditions. Additionally, you will automate jobs, notifications, and reports to improve efficiency. You should possess the ability to optimize existing queries, reverse engineer for data research and analysis, and calculate the impact of issues on the downstream side for effective communication. Supporting failures, data quality issues, and ensuring environment health will also be part of your role. Furthermore, you will maintain ingestion and pipeline runbooks, portfolio summaries, and DBAR, while enabling infrastructure changes, enhancements, and updates roadmap. Building the infrastructure for optimal extraction, transformation, and loading data from various sources using big data technologies, python, or web-based APIs will be essential. You will participate in code reviews with peers, have excellent communication skills for understanding and conveying requirements effectively. As a candidate, you are expected to have a Bachelor's degree in Engineering/Computer Science or a related quantitative field. Technical skills required include a minimum of 8 years of programming experience with python and SQL, experience with massively parallel processing systems like Spark or Hadoop, and a minimum of 6-7 years of hands-on experience with GCP, BigQuery, Dataflow, Data Warehousing, Data modeling, Apache Beam, and Cloud Storage. Proficiency in source code control systems (GIT) and CI/CD processes, involvement in designing, prototyping, and delivering software solutions within the big data ecosystem, and hands-on experience in generative AI models are also necessary. You should be able to perform code reviews to ensure code meets acceptance criteria, have experience with Agile development methodologies and tools, and work towards improving data governance and quality to enhance data reliability. EXL Analytics offers a dynamic and innovative environment where you will collaborate with experienced analytics consultants. You will gain insights into various business aspects, develop effective teamwork and time-management skills, and receive training in analytical tools and techniques. Our mentoring program provides guidance and coaching to every employee, fostering personal and professional growth. The opportunities for growth and development at EXL Analytics are limitless, setting the stage for a successful career within the company and beyond.,

As a Founding Engineer at Colrows, you will play a crucial role in shaping and scaling our generative analytics platform. We are dedicated to revolutionizing how enterprises engage with data, aiming to make insights as accessible as conversing with a colleague. Our platform integrates robust data foundations, agentic workflows, and enterprise-grade governance to deliver a seamless, self-serve experience. Join us in driving the development of a powerful execution engine that lies at the core of our platform. In this role, you will serve as a hands-on Principal Software Engineer, leveraging your architectural expertise and practical experience in constructing resilient systems using Java and open-source technologies. Your primary responsibility will be to lead the design and implementation of the backend engine, which processes and transforms large-scale structured data with a focus on precision, performance, and clarity. Key Responsibilities: - Design and construct essential components of the Colrows platform, such as distributed query execution, type-safe data transformation pipelines, and high-performance connectors to various database backends. - Architect and deploy scalable, fault-tolerant services in Java utilizing open-source libraries and tools. - Ensure accuracy in data type mappings, memory representations, and serialization across diverse data systems. - Mentor and guide engineers through code reviews, design discussions, and the implementation of production-quality solutions. - Collaborate with AI, frontend, and infrastructure teams to deliver a cohesive and agentic product experience. - Take charge of technical decision-making by balancing performance, maintainability, and extensibility. You Bring: - Over 10 years of hands-on experience in backend engineering with a specialization in Java. - A solid foundation in computer science fundamentals, particularly in data structures, memory models, and type systems. - Previous experience handling large datasets spanning terabytes of structured data across distributed systems. - Practical knowledge of data type mappings in SQL engines, Java, JSON, Avro, Arrow, and binary formats. - Proficiency in in-memory data representations, caching strategies, and performance optimization. - Demonstrated success in designing and implementing clean, testable, and production-grade code. - Familiarity with tools such as Apache Arrow, Apache Calcite, Apache Ignite, Presto/Trino, Apache Beam, RocksDB, Protobuf, gRPC, and Netty. - Bonus: Experience in contributing to open-source projects or building data platforms, databases, or query engines. What You'll Get: - Opportunity to work on a cutting-edge analytics platform that is reshaping how enterprises engage with data. - Direct collaboration with the founder and a dedicated, technically proficient team. - A culture that promotes ownership, autonomy, and deep engineering involvement. - Competitive compensation, meaningful equity, and the flexibility of a modern remote/hybrid work environment.,