31 Apache Beam Jobs

As a member of the Platform Observability Engineering team within Ford's Data Platforms and Engineering (DP&E) organization, you will contribute to building and maintaining a top-tier platform for monitoring and observability. This platform focuses on the four golden signalslatency, traffic, errors, and saturationproviding essential data to support operations, root cause analysis, continuous improvement, and cost optimization. You will collaborate with platform architects to help design, develop, and maintain a scalable and reliable platform, ensuring smooth integration with systems used across various teams. Your contributions will be key in improving MTTR and MTTX through increased visibility into system performance. Working with stakeholders, you will integrate observability data into their workflows, develop insightful dashboards and reports, continuously improve platform performance and reliability, optimize costs, and stay updated with industry best practices and technologies. The role focuses on building and maintaining a robust platform rather than developing individual monitoring tools, creating a centralized, reliable source of observability data that empowers data-driven decisions and accelerates incident response across the organization. Responsibilities: - Design and Build Data Pipelines: Architect, develop, and maintain scalable data pipelines and microservices supporting real-time and batch processing on GCP. - Service-Oriented Architecture (SOA) and Microservices: Design and implement SOA and microservices-based architectures for modular, flexible, and maintainable data solutions. - Full-Stack Integration: Contribute to the seamless integration of front-end and back-end components, ensuring robust data access and UI-driven data exploration. - Data Ingestion and Integration: Lead the ingestion and integration of data from various sources into the data platform, ensuring standardized and optimized data for analytics. - GCP Data Solutions: Utilize GCP services (BigQuery, Dataflow, Pub/Sub, Cloud Functions, etc.) to build and manage data platforms meeting business needs. - Data Governance and Security: Implement and manage data governance, access controls, and security best practices while leveraging GCP's native security features. - Performance Optimization: Continuously monitor and improve the performance, scalability, and efficiency of data pipelines and storage solutions. - Collaboration and Best Practices: Define best practices, design patterns, and frameworks for cloud data engineering by closely working with data architects, software engineers, and cross-functional teams. - Automation and Reliability: Automate data platform processes to enhance reliability, reduce manual intervention, and improve operational efficiency. Qualifications: - Technical Skills: Proficiency in Java, Angular, or any JavaScript technology with experience in designing and deploying cloud-based data pipelines and microservices using GCP tools like BigQuery, Dataflow, and Dataproc. - Service-Oriented Architecture and Microservices: Strong understanding of SOA, microservices, and their application within a cloud data platform context. Develop robust, scalable services using Java Spring Boot, Python, Angular, and GCP technologies. - Full-Stack Development: Knowledge of front-end and back-end technologies enabling collaboration on data access and visualization layers (e.g., React, Node.js). - Design and develop RESTful APIs for seamless integration across platform services. - Implement robust unit and functional tests to maintain high standards of test coverage and quality. - Database Management: Experience with relational (e.g., PostgreSQL, MySQL) and NoSQL databases, as well as columnar databases like BigQuery. - Data Governance and Security: Understanding of data governance frameworks and implementing RBAC, encryption, and data masking in cloud environments. - CI/CD and Automation: Familiarity with CI/CD pipelines, Infrastructure as Code (IaC) tools like Terraform, and automation frameworks. - Manage code changes with GitHub and troubleshoot and resolve application defects efficiently. - Ensure adherence to SDLC best practices, independently managing feature design, coding, testing, and production releases. - Problem-Solving: Strong analytical skills with the ability to troubleshoot complex data platform and microservices issues. Certifications (Preferred): GCP Data Engineer, GCP Professional Cloud,

You should possess a minimum of 7-10 years of industry experience, out of which a minimum of 5 years should have been in machine learning roles. Your proficiency in Python and popular ML libraries such as TensorFlow, PyTorch, and Scikit-learn should be advanced. Furthermore, you should have hands-on experience in distributed training, model optimization including quantization and pruning, and inference at scale. Experience with cloud ML platforms like AWS (SageMaker), GCP (Vertex AI), or Azure ML is essential. It is expected that you are familiar with MLOps tooling such as MLflow, TFX, Airflow, or Kubeflow, and data engineering frameworks like Spark, dbt, or Apache Beam. A solid understanding of CI/CD for ML, model governance, and post-deployment monitoring (e.g., data drift, model decay) is crucial for this role. In addition to technical skills, problem-solving abilities, effective communication, and strong documentation skills are highly valued in this position.,

As a Data Engineer at Synoptek, you will be responsible for designing, developing, and maintaining robust and scalable data pipelines on the Google Cloud Platform (GCP). You will leverage your hands-on experience with GCP services such as BigQuery, Jitterbit, Cloud Dataflow, Cloud Pub/Sub, and Cloud Storage to build efficient data processing solutions. Collaborating with cross-functional teams, you will translate their data needs into technical requirements, ensuring data quality, integrity, and security throughout the data lifecycle. Your role will involve developing and optimizing ETL/ELT processes to extract, transform, and load data from various sources into data warehouses and data lakes. Additionally, you will build and maintain data models and schemas to support business intelligence and analytics, while troubleshooting data quality issues and performance bottlenecks. To excel in this position, you should have a Bachelor's degree in Computer Science, Engineering, or a related field, along with 3 to 4 years of experience as a Data Engineer focusing on GCP. Proficiency in Python, SQL, and BigQuery is essential, as well as hands-on experience with data ingestion, transformation, and loading tools like Jitterbit and Apache Beam. A strong understanding of data warehousing and data lake concepts, coupled with experience in data modeling and schema design, will be beneficial. The ideal candidate will exhibit excellent problem-solving and analytical skills, working both independently and collaboratively with internal and external teams. Familiarity with acquiring and managing data from various sources, as well as the ability to identify trends in complex datasets and propose business solutions, are key attributes for success in this role. At Synoptek, we value employees who embody our core DNA behaviors, including clarity, integrity, innovation, accountability, and a results-focused mindset. We encourage continuous learning, adaptation, and growth in a fast-paced environment, promoting a culture of teamwork, flexibility, respect, and collaboration. If you have a passion for data engineering, a drive for excellence, and a commitment to delivering impactful results, we invite you to join our dynamic team at Synoptek. Work hard, play hard, and let's achieve superior outcomes together.,

As a Data Pipeline Architect at our company, you will be responsible for designing, developing, and maintaining optimal data pipeline architecture. You will monitor incidents, perform root cause analysis, and implement appropriate actions to ensure smooth operations. Additionally, you will troubleshoot issues related to abnormal job execution and data corruption, and automate jobs, notifications, and reports for efficiency. Your role will also involve optimizing existing queries, reverse engineering for data research and analysis, and calculating the impact of issues on downstream processes for effective communication. You will support failures, address data quality issues, and ensure the overall health of the environment. Maintaining ingestion and pipeline runbooks, portfolio summaries, and DBAR will be part of your responsibilities. Furthermore, you will enable infrastructure changes, enhancements, and updates roadmap, and build the infrastructure for optimal extraction, transformation, and loading of data from various sources using big data technologies, python, or Web-based APIs. Conducting and participating in code reviews with peers, ensuring effective communication, and understanding requirements will be essential in this role. To qualify for this position, you should hold a Bachelor's degree in Engineering/Computer Science or a related quantitative field. You must have a minimum of 8 years of programming experience with python and SQL, as well as hands-on experience with GCP, BigQuery, Dataflow, Data Warehousing, Apache Beam, and Cloud Storage. Experience with massively parallel processing systems like Spark or Hadoop, source code control systems (GIT), and CI/CD processes is required. Involvement in designing, prototyping, and delivering software solutions within the big data ecosystem, developing generative AI models, and ensuring code quality through reviews are key aspects of this role. Experience with Agile development methodologies, improving data governance and quality, and increasing data reliability are also important. Joining our team at EXL Analytics offers you the opportunity to work in a dynamic and innovative environment alongside experienced professionals. You will gain insights into various business domains, develop teamwork and time-management skills, and receive training in analytics tools and techniques. Our mentoring program and growth opportunities ensure that you have the support and guidance needed to excel in your career. Sky is the limit for our team members, and the experiences gained at EXL Analytics pave the way for personal and professional development within our company and beyond.,

As the Architect IT Strategy, VP at Deutsche Bank Group based in Pune, India, your role involves designing applications, defining modernization approaches, developing application code, and implementing technical solutions to address business problems and meet specified requirements and design principles. Under our flexible scheme, you will enjoy benefits such as a best-in-class leave policy, gender-neutral parental leaves, childcare assistance benefit reimbursement, flexible working arrangements, sponsorship for industry-relevant certifications and education, employee assistance program, comprehensive hospitalization and life insurance, and health screening for individuals aged 35 years and above. Your key responsibilities will include managing work across various areas of the bank's IT platform, planning and developing engineering solutions, ensuring reliability and resiliency, fostering maintainability and reusability, reviewing engineering plans and quality, participating in industry forums, and leading an application development team. Key requirements for this role include technical leadership experience, expertise in designing and developing cloud-native services in GCP/AWS, proficiency in Java/J2EE, Spring Boot, scripting languages like Apache Beam or Python, knowledge of GCP/AWS managed services, experience with Agile methodologies, and strong communication skills. To be successful in this position, you should have 15 to 22 years of overall IT experience, an engineering degree or post-graduation, and the ability to manage stakeholders" expectations. Desired behaviors include being a strong team player, result-oriented, conflict resolver, and possessing excellent communication and collaboration skills. A bachelor's or master's degree in Computer Science, Information Systems, or related discipline is required, or an equivalent combination of education and relevant work experience. You will receive training and development, coaching and support from experts in your team, and a culture of continuous learning to aid your career progression. If you are a motivated individual who thrives in a collaborative and diverse environment, eager to excel in your career, and comfortable with navigating ambiguity to extract meaningful insights, we welcome your application for the Architect IT Strategy, VP position at Deutsche Bank Group in Pune, India.,

As a GCP Data Engineer in Australia, you will be responsible for leveraging your experience in Google Cloud Platform (GCP) to handle various aspects of data engineering. Your role will involve working on data migration projects from legacy systems such as SQL and Oracle. You will also be designing and building ETL pipelines for data lake and data warehouse solutions on GCP. In this position, your expertise in GCP data and analytics services will be crucial. You will work with tools like Cloud Dataflow, Cloud Dataprep, Apache Beam/Cloud composer, Cloud BigQuery, Cloud Fusion, Cloud PubSub, Cloud storage, and Cloud Functions. Additionally, you will utilize Cloud Native GCP CLI/gsutil for operations and scripting languages like Python and SQL to enhance data processing efficiencies. Furthermore, your experience with data governance practices, metadata management, data masking, and encryption will be essential. You will utilize GCP tools such as Cloud Data Catalog and GCP KMS tools to ensure data security and compliance. Overall, this role requires a strong foundation in GCP technologies and a proactive approach to data engineering challenges in a dynamic environment.,

The ideal candidate for the position will have the responsibility of designing, developing, and maintaining an optimal data pipeline architecture. You will be required to monitor incidents, perform root cause analysis, and implement appropriate actions to solve issues related to abnormal job execution and data corruption conditions. Additionally, you will automate jobs, notifications, and reports to improve efficiency. You should possess the ability to optimize existing queries, reverse engineer for data research and analysis, and calculate the impact of issues on the downstream side for effective communication. Supporting failures, data quality issues, and ensuring environment health will also be part of your role. Furthermore, you will maintain ingestion and pipeline runbooks, portfolio summaries, and DBAR, while enabling infrastructure changes, enhancements, and updates roadmap. Building the infrastructure for optimal extraction, transformation, and loading data from various sources using big data technologies, python, or web-based APIs will be essential. You will participate in code reviews with peers, have excellent communication skills for understanding and conveying requirements effectively. As a candidate, you are expected to have a Bachelor's degree in Engineering/Computer Science or a related quantitative field. Technical skills required include a minimum of 8 years of programming experience with python and SQL, experience with massively parallel processing systems like Spark or Hadoop, and a minimum of 6-7 years of hands-on experience with GCP, BigQuery, Dataflow, Data Warehousing, Data modeling, Apache Beam, and Cloud Storage. Proficiency in source code control systems (GIT) and CI/CD processes, involvement in designing, prototyping, and delivering software solutions within the big data ecosystem, and hands-on experience in generative AI models are also necessary. You should be able to perform code reviews to ensure code meets acceptance criteria, have experience with Agile development methodologies and tools, and work towards improving data governance and quality to enhance data reliability. EXL Analytics offers a dynamic and innovative environment where you will collaborate with experienced analytics consultants. You will gain insights into various business aspects, develop effective teamwork and time-management skills, and receive training in analytical tools and techniques. Our mentoring program provides guidance and coaching to every employee, fostering personal and professional growth. The opportunities for growth and development at EXL Analytics are limitless, setting the stage for a successful career within the company and beyond.,

As a Founding Engineer at Colrows, you will play a crucial role in shaping and scaling our generative analytics platform. We are dedicated to revolutionizing how enterprises engage with data, aiming to make insights as accessible as conversing with a colleague. Our platform integrates robust data foundations, agentic workflows, and enterprise-grade governance to deliver a seamless, self-serve experience. Join us in driving the development of a powerful execution engine that lies at the core of our platform. In this role, you will serve as a hands-on Principal Software Engineer, leveraging your architectural expertise and practical experience in constructing resilient systems using Java and open-source technologies. Your primary responsibility will be to lead the design and implementation of the backend engine, which processes and transforms large-scale structured data with a focus on precision, performance, and clarity. Key Responsibilities: - Design and construct essential components of the Colrows platform, such as distributed query execution, type-safe data transformation pipelines, and high-performance connectors to various database backends. - Architect and deploy scalable, fault-tolerant services in Java utilizing open-source libraries and tools. - Ensure accuracy in data type mappings, memory representations, and serialization across diverse data systems. - Mentor and guide engineers through code reviews, design discussions, and the implementation of production-quality solutions. - Collaborate with AI, frontend, and infrastructure teams to deliver a cohesive and agentic product experience. - Take charge of technical decision-making by balancing performance, maintainability, and extensibility. You Bring: - Over 10 years of hands-on experience in backend engineering with a specialization in Java. - A solid foundation in computer science fundamentals, particularly in data structures, memory models, and type systems. - Previous experience handling large datasets spanning terabytes of structured data across distributed systems. - Practical knowledge of data type mappings in SQL engines, Java, JSON, Avro, Arrow, and binary formats. - Proficiency in in-memory data representations, caching strategies, and performance optimization. - Demonstrated success in designing and implementing clean, testable, and production-grade code. - Familiarity with tools such as Apache Arrow, Apache Calcite, Apache Ignite, Presto/Trino, Apache Beam, RocksDB, Protobuf, gRPC, and Netty. - Bonus: Experience in contributing to open-source projects or building data platforms, databases, or query engines. What You'll Get: - Opportunity to work on a cutting-edge analytics platform that is reshaping how enterprises engage with data. - Direct collaboration with the founder and a dedicated, technically proficient team. - A culture that promotes ownership, autonomy, and deep engineering involvement. - Competitive compensation, meaningful equity, and the flexibility of a modern remote/hybrid work environment.,

You should have 5+ years of experience in core Java and the Spring Framework. Additionally, you must have at least 2 years of experience in Cloud technologies such as GCP, AWS, or Azure, with a preference for GCP. It is required to have experience in big data processing on a distributed system and in working with databases including RDBMS, NoSQL databases, and Cloud natives. You should also have expertise in handling various data formats like Flat file, JSON, Avro, XML, etc., including defining schemas and contracts. Furthermore, you should have experience in implementing data pipelines (ETL) using Dataflow (Apache Beam) and in working with Microservices and integration patterns of APIs with data processing. Experience in data structure, defining, and designing data models will be beneficial for this role.,

Dear Candidate, Greetings of the day!! I am Kantha, and I'm reaching out to you regarding an exciting opportunity with TechMango. You can connect with me on LinkedIn https://www.linkedin.com/in/kantha-m-ashwin-186ba3244/ Or Email: kanthasanmugam.m@techmango.net Techmango Technology Services is a full-scale software development services company founded in 2014 with a strong focus on emerging technologies. It holds a primary objective of delivering strategic solutions towards the goal of its business partners in terms of technology. We are a full-scale leading Software and Mobile App Development Company. Techmango is driven by the mantra Clients Vision is our Mission. We have a tendency to stick on to the current statement. To be the technologically advanced & most loved organization providing prime quality and cost-efficient services with a long-term client relationship strategy. We are operational in the USA - Chicago, Atlanta, Dubai - UAE, in India - Bangalore, Chennai, Madurai, Trichy. Job Title: GCP Data Engineer Location: Madurai Experience: 5+ Years Notice Period: Immediate Job Summary We are seeking a hands-on GCP Data Engineer with deep expertise in real-time streaming data architectures to help design, build, and optimize data pipelines in our Google Cloud Platform (GCP) environment. The ideal candidate will have strong architectural vision and be comfortable rolling up their sleeves to build scalable, low-latency streaming data pipelines using Pub/Sub, Dataflow (Apache Beam) , and BigQuery . Key Responsibilities Architect and implement end-to-end streaming data solutions on GCP using Pub/Sub , Dataflow , and BigQuery . Design real-time ingestion, enrichment, and transformation pipelines for high-volume event data. Work closely with stakeholders to understand data requirements and translate them into scalable designs. Optimize streaming pipeline performance, latency, and throughput. Build and manage orchestration workflows using Cloud Composer (Airflow) . Drive schema design, partitioning, and clustering strategies in BigQuery for both real-time and batch datasets. Define SLAs, monitoring, logging, and alerting for streaming jobs using Cloud Monitoring , Error Reporting , and Stackdriver . Experience with the data modeling. Ensure robust security, encryption, and access controls across all data layers. Collaborate with DevOps for CI/CD automation of data workflows using Terraform , Cloud Build , and Git . Document streaming architecture, data lineage, and deployment runbooks. Required Skills & Experience 5+ years of experience in data engineering or architecture. 3+ years of hands-on GCP data engineering experience. Strong expertise in: Google Pub/Sub Dataflow (Apache Beam) BigQuery (including streaming inserts) Cloud Composer (Airflow) Cloud Storage (GCS) Solid understanding of streaming design patterns , exactly-once delivery , and event-driven architecture . Deep knowledge of SQL and NoSQL data modeling. Hands-on experience with monitoring and performance tuning of streaming jobs. Experience using Terraform or equivalent for infrastructure as code. Familiarity with CI/CD pipelines for data workflows.

Hiring Java Fullstack Developer_Hyderabad Exp:-12 to 20 Years Location:- Hyderabad Notice period:- Immediate to 15 Days JD:- Responsibilities: 1. Design, develop, and maintain line of business applications using cutting-edge technologies. 2. Write secure, scalable, and maintainable code using programming languages such as React, Java. 3. Implement cloud-native architectures on Google Cloud Platform. 4. Stay abreast of emerging trends and technologies in AI. Apply AI at various phases of the software development life cycle, to increase software development productivity. Suggest AI Enabled features in software for a better and modern experience for the customers. 5. Design for performance, troubleshoot performance issues, and involve in performance tuning to produce highly performing software. 6. Lead Teams or modules and be responsible for delivery of high performing and high quality software. 7. Perform code reviews and Architecture reviews to enforce coding standards and to align with industry best practices. 8. Collaborate with multiple cross-functional teams and stakeholders to understand requirements. Actively engage with stakeholders to provide progress updates and ensure alignment with business objectives and delivery timelines. 9. Mentor and guide junior developers, fostering skill development and professional growth. 10. Solve complex technical challenges by identifying root causes and developing innovative solutions. Interested candidates Please share your resume to mkateru@vsoftconsulting.com

Dear Candidate, Greetings of the day!! I am Kantha, and I'm reaching out to you regarding an exciting opportunity with TechMango. You can connect with me on LinkedIn https://www.linkedin.com/in/kantha-m-ashwin-186ba3244/ Or Email: kanthasanmugam.m@techmango.net Techmango Technology Services is a full-scale software development services company founded in 2014 with a strong focus on emerging technologies. It holds a primary objective of delivering strategic solutions towards the goal of its business partners in terms of technology. We are a full-scale leading Software and Mobile App Development Company. Techmango is driven by the mantra Clients Vision is our Mission. We have a tendency to stick on to the current statement. To be the technologically advanced & most loved organization providing prime quality and cost-efficient services with a long-term client relationship strategy. We are operational in the USA - Chicago, Atlanta, Dubai - UAE, in India - Bangalore, Chennai, Madurai, Trichy. Job Title: Technical GCP Data Architect/Lead Location: Madurai Experience: 12+ Years Notice Period: Immediate Job Summary We are seeking a hands-on Technical GCP Data Architect/Lead with deep expertise in real-time streaming data architectures to help design, build, and optimize data pipelines in our Google Cloud Platform (GCP) environment. The ideal candidate will have strong architectural vision and be comfortable rolling up their sleeves to build scalable, low-latency streaming data pipelines using Pub/Sub, Dataflow (Apache Beam) , and BigQuery . Key Responsibilities Architect and implement end-to-end streaming data solutions on GCP using Pub/Sub , Dataflow , and BigQuery . Design real-time ingestion, enrichment, and transformation pipelines for high-volume event data. Work closely with stakeholders to understand data requirements and translate them into scalable designs. Optimize streaming pipeline performance, latency, and throughput. Build and manage orchestration workflows using Cloud Composer (Airflow) . Drive schema design, partitioning, and clustering strategies in BigQuery for both real-time and batch datasets. Define SLAs, monitoring, logging, and alerting for streaming jobs using Cloud Monitoring , Error Reporting , and Stackdriver . Experience with the data modeling. Ensure robust security, encryption, and access controls across all data layers. Collaborate with DevOps for CI/CD automation of data workflows using Terraform , Cloud Build , and Git . Document streaming architecture, data lineage, and deployment runbooks. Required Skills & Experience 10+ years of experience in data engineering or architecture. 3+ years of hands-on GCP data engineering experience. Strong expertise in: Google Pub/Sub Dataflow (Apache Beam) BigQuery (including streaming inserts) Cloud Composer (Airflow) Cloud Storage (GCS) Solid understanding of streaming design patterns , exactly-once delivery , and event-driven architecture . Deep knowledge of SQL and NoSQL data modeling. Hands-on experience with monitoring and performance tuning of streaming jobs. Experience using Terraform or equivalent for infrastructure as code. Familiarity with CI/CD pipelines for data workflows.

You will be part of a dynamic team at Equifax, where we are seeking creative, high-energy, and driven software engineers with hands-on development skills to contribute to various significant projects. As a software engineer at Equifax, you will have the opportunity to work with cutting-edge technology alongside a talented group of engineers. This role is perfect for you if you are a forward-thinking, committed, and enthusiastic individual who is passionate about technology. Your responsibilities will include designing, developing, and operating high-scale applications across the entire engineering stack. You will be involved in all aspects of software development, from design and testing to deployment, maintenance, and continuous improvement. By utilizing modern software development practices such as serverless computing, microservices architecture, CI/CD, and infrastructure-as-code, you will contribute to the integration of our systems with existing internal systems and tools. Additionally, you will participate in technology roadmap discussions and architecture planning to translate business requirements and vision into actionable solutions. Working within a closely-knit, globally distributed engineering team, you will be responsible for triaging product or system issues and resolving them efficiently to ensure the smooth operation and quality of our services. Managing project priorities, deadlines, and deliverables will be a key part of your role, along with researching, creating, and enhancing software applications to advance Equifax Solutions. To excel in this position, you should have a Bachelor's degree or equivalent experience, along with at least 7 years of software engineering experience. Proficiency in mainstream Java, SpringBoot, TypeScript/JavaScript, as well as hands-on experience with Cloud technologies such as GCP, AWS, or Azure, is essential. You should also have a solid background in designing and developing cloud-native solutions and microservices using Java, SpringBoot, GCP SDKs, and GKE/Kubernetes. Experience in deploying and releasing software using Jenkins CI/CD pipelines, infrastructure-as-code concepts, Helm Charts, and Terraform constructs is highly valued. Moreover, being a self-starter who can adapt to changing priorities with minimal supervision could set you apart in this role. Additional advantageous skills include designing big data processing solutions, UI development, backend technologies like JAVA/J2EE and SpringBoot, source code control management systems, build tools, working in Agile environments, relational databases, and automated testing. If you are ready to take on this exciting opportunity and contribute to Equifax's innovative projects, apply now and be part of our team of forward-thinking software engineers.,

Job Description: Job Title: Apache beam software engineer Work Mode: Remote Base Location: Bengaluru Experience Required: 4 to 6 Years Job Summary: We are looking for a Software Engineer with hands-on experience in Apache Beam , Google Cloud Dataflow , and Dataproc , focusing on building reusable data processing frameworks . This is not a traditional data engineering role. The ideal candidate will have strong software development skills in Java or Python and experience in building scalable, modular data processing components and frameworks for batch and streaming use cases. Key Responsibilities: Design and develop framework-level components using Apache Beam , GCP Dataflow , and Dataproc . Build scalable, reusable libraries and abstractions in Python or Java for distributed data processing. Work closely with architects to implement best practices for designing high-performance data frameworks. Ensure software reliability, maintainability, and testability through strong coding and automation practices. Participate in code reviews, architectural discussions, and performance tuning initiatives. Contribute to internal tooling or SDK development for data engineering platforms. Required Skills: 4 to 6 years of experience as a Software Engineer working on distributed systems or data processing frameworks. Strong programming skills in Java and/or Python . Deep experience with Apache Beam and GCP Dataflow . Hands-on experience with GCP Dataproc , especially for building scalable custom batch or streaming jobs. Solid understanding of streaming vs batch processing concepts. Familiarity with CI/CD pipelines , GitHub , and test automation. Preferred Skills: Experience with workflow orchestration tools such as Airflow (Composer) . Exposure to Pub/Sub and BigQuery (from a system integration perspective). Understanding of observability , logging , and error-handling in distributed data pipelines. Experience building internal libraries, SDKs, or tools to support data teams. Tech Stack: Cloud: GCP (Dataflow, Dataproc, Pub/Sub, Composer) Programming: Java, Python Frameworks: Apache Beam DevOps: GitHub, CI/CD (Cloud Build, Jenkins) Focus Areas: Framework/library development, scalable distributed data processing, component-based architecture

Arista Networks is a renowned industry leader specializing in data-driven, client-to-cloud networking solutions tailored for large data center, campus, and routing environments. With a robust foundation and revenue exceeding $8 billion, Arista stands out for its award-winning platforms delivering Ethernet speeds up to 800G bits per second, setting new standards in scalability, agility, and resilience. As a founding member of the Ultra Ethernet consortium, Arista has globally shipped over 20 million cloud networking ports powered by CloudVision and EOS, an advanced network operating system. Committed to open standards, Arista offers its products worldwide through direct channels and partnerships. Diversity of thought and varied perspectives are highly valued at Arista, as the company believes in fostering an inclusive environment where individuals from diverse backgrounds and experiences can contribute to driving creativity and innovation. Arista's commitment to excellence has been recognized with prestigious awards, such as the Great Place to Work Survey accolades for Best Engineering Team and Best Company for Diversity, Compensation, and Work-Life Balance. The company takes pride in its successful track record and upholds the highest quality and performance standards in all endeavors. Arista is excited to expand the Wi-Fi Team at the Pune Development Center to elevate its Cognitive Wi-Fi solution to new heights. With ambitious growth plans for the Pune-based Development Center in the coming years, this is an opportune moment to join the team and make a significant impact on its evolution and direction. As a Software Engineer at Arista, you will collaborate with the Wi-Fi Data team within the broader Software Engineering team. The Wi-Fi Data team is pivotal to the success of Arista's Cognitive Wi-Fi solution, particularly in managing the increasing volumes of cloud-based data. This team offers ample opportunities for growth and influence due to its small and relatively new structure. Your responsibilities within the Wi-Fi Data team will involve close collaboration with Data Scientists to develop and maintain data and AI/ML pipelines operating at scale. This includes tasks like anomaly detection, root cause analysis, automatic remediation, and analytics use cases. Additionally, you will be involved in creating ELT data pipelines to extract data from multiple Wi-Fi sources and feed them into a data warehouse. You will also spearhead the development and management of CI/CD pipelines for deploying these data pipelines. Depending on the project, you may have the chance to showcase your work to a wider audience through talks and blog posts. Qualifications: - Bachelor's degree in Computer Science or a related field. - Proficiency in Python or Go. - Experience with databases (Relational and/or NoSQL). - Hands-on familiarity with DevOps tools like Jenkins, Git, Docker, Kubernetes, Ansible, and CI/CD pipelines. - Knowledge of data processing libraries such as Apache Beam and data manipulation libraries like Pandas (Python) would be advantageous. Arista prides itself on being an engineering-centric company where leadership, including founders and engineering managers, are engineers who prioritize sound software engineering practices and quality. With a global team, Arista offers engineers full ownership of their projects in a flat and streamlined management structure led by those with deep engineering expertise. Test automation tools are emphasized, and engineers enjoy exposure to diverse areas within the company. Arista's headquarters is in Santa Clara, California, with development offices in Australia, Canada, India, Ireland, and the US, considering all R&D centers equal in significance. Join Arista to be part of shaping the future of networking and contribute to a culture that values innovation, quality, respect, and enjoyment.,

You should have 6 months to 3 years of IT experience. You must have knowledge of Bigquery, SQL, or similar tools. It is essential to be aware of ETL and Data warehouse concepts. Your oral and written communication skills should be good. Being a great team player and able to work efficiently with minimal supervision is crucial. You should also have good knowledge of Java or Python to conduct data cleansing. Preferred qualifications include good communication and problem-solving skills. Experience on Spring Boot would be an added advantage. Being an Apache Beam developer with Google Cloud BigTable and Google BigQuery is desirable. Experience in Google Cloud Platform (GCP) is preferred. Skills in writing batch and stream processing jobs using Apache Beam Framework (Dataflow) are a plus. Knowledge of Microservices, Pub/Sub, Cloud Run, and Cloud Function would be beneficial.,

Arista Networks, a leader in data-driven, client-to-cloud networking solutions, is seeking a Software Engineer to join the Wi-Fi Data team in the Pune Development Center. Arista has a strong commitment to excellence, open standards, and diversity, creating an inclusive environment where innovation thrives. As a Software Engineer at Arista, you will collaborate with Data Scientists to develop and maintain data and AI/ML pipelines for the Cognitive Wi-Fi solution. This role offers significant growth opportunities within a small yet impactful team. You will be responsible for building ELT data pipelines, working on anomaly detection, root cause analysis, automatic remediation, and analytics use cases. Furthermore, you will develop and manage CI/CD pipelines for deployment and have the chance to showcase your work through talks and blog posts. The ideal candidate for this role should have a Bachelor's degree in Computer Science or a related field, proficiency in Python or Go, experience with databases (Relational and/or NoSQL), and hands-on expertise with DevOps tools like Jenkins, Git, Docker, Kubernetes, and Ansible. Familiarity with data processing libraries such as Apache Beam and data manipulation tools like Pandas would be advantageous. Arista Networks is known for its engineering-centric culture, where engineers have ownership of projects and access to various domains within the company. The company is headquartered in Santa Clara, California, with development offices worldwide, including Australia, Canada, India, Ireland, and the US. Arista values quality, respect, and innovation, offering a flat management structure and prioritizing test automation tools. If you are looking to shape the future of networking and be part of a dynamic team that values invention and fun, Arista Networks offers a unique opportunity to make a significant impact in the Wi-Fi space. Join Arista Networks and contribute to the growth and success of the Pune Development Center while working on cutting-edge technologies that redefine networking scalability and agility.,

You should have a strong understanding of the tech stack including GCP Services such as BigQuery, Cloud Dataflow, Pub/Sub, Dataproc, and Cloud Storage. Experience with Data Processing tools like Apache Beam (batch/stream), Apache Kafka, and Cloud Dataprep is crucial. Proficiency in programming languages like Python, Java/Scala, and SQL is required. Your expertise should extend to Orchestration tools like Apache Airflow (Cloud Composer) and Terraform, and Security aspects including IAM, Cloud Identity, and Cloud Security Command Center. Knowledge of Containerization using Docker and Kubernetes (GKE) is essential. Familiarity with Machine Learning platforms such as Google AI Platform, TensorFlow, and AutoML is expected. Candidates with certifications like Google Cloud Data Engineer and Cloud Architect are preferred. You should have a proven track record of designing scalable AI/ML systems in production, focusing on high-performance and cost-effective solutions. Strong experience with cloud platforms (Google Cloud, AWS, Azure) and cloud-native AI/ML services like Vertex AI and SageMaker is important. Your role will involve implementing MLOps practices, including model deployment, monitoring, retraining, and version control. Leadership skills are key to guide teams, mentor engineers, and collaborate effectively with cross-functional teams to achieve business objectives. A deep understanding of frameworks like TensorFlow, PyTorch, and Scikit-learn for designing, training, and deploying models is necessary. Experience with data engineering principles, scalable pipelines, and distributed systems (e.g., Apache Kafka, Spark, Kubernetes) is also required. Nice to have requirements include strong leadership and mentorship capabilities to guide teams towards best practices and high-quality deliverables. Excellent problem-solving skills focusing on designing efficient, high-performance systems are valued. Effective project management abilities are necessary to handle multiple initiatives and ensure timely delivery. Collaboration and teamwork are emphasized to foster a positive and productive work environment.,

Dear Candidate, Greetings of the day!! I am Kantha, and I'm reaching out to you regarding an exciting opportunity with TechMango. You can connect with me on LinkedIn https://www.linkedin.com/in/kantha-m-ashwin-186ba3244/ Or Email: kanthasanmugam.m@techmango.net Techmango Technology Services is a full-scale software development services company founded in 2014 with a strong focus on emerging technologies. It holds a primary objective of delivering strategic solutions towards the goal of its business partners in terms of technology. We are a full-scale leading Software and Mobile App Development Company. Techmango is driven by the mantra Clients Vision is our Mission. We have a tendency to stick on to the current statement. To be the technologically advanced & most loved organization providing prime quality and cost-efficient services with a long-term client relationship strategy. We are operational in the USA - Chicago, Atlanta, Dubai - UAE, in India - Bangalore, Chennai, Madurai, Trichy. Job Title: GCP Data Architect Location: Madurai/Chennai Experience: 12+ Years Notice Period: Immediate About TechMango TechMango is a rapidly growing IT Services and SaaS Product company that helps global businesses with digital transformation, modern data platforms, product engineering, and cloud-first initiatives. We are seeking a GCP Data Architect to lead data modernization efforts for our prestigious client, Livingston, in a highly strategic project. Role Summary As a GCP Data Architect, you will be responsible for designing and implementing scalable, high-performance data solutions on Google Cloud Platform. You will work closely with stakeholders to define data architecture, implement data pipelines, modernize legacy data systems, and guide data strategy aligned with enterprise goals. Key Responsibilities: Lead end-to-end design and implementation of scalable data architecture on Google Cloud Platform (GCP) Define data strategy, standards, and best practices for cloud data engineering and analytics Develop data ingestion pipelines using Dataflow, Pub/Sub, Apache Beam, Cloud Composer (Airflow), and BigQuery Migrate on-prem or legacy systems to GCP (e.g., from Hadoop, Teradata, or Oracle to BigQuery) Architect data lakes, warehouses, and real-time data platforms Ensure data governance, security, lineage, and compliance (using tools like Data Catalog, IAM, DLP) Guide a team of data engineers and collaborate with business stakeholders, data scientists, and product managers Create documentation, high-level design (HLD) and low-level design (LLD), and oversee development standards Provide technical leadership in architectural decisions and future-proofing the data ecosystem Required Skills & Qualifications: 10+ years of experience in data architecture, data engineering, or enterprise data platforms Minimum 3–5 years of hands-on experience in GCP Data Service Proficient in:BigQuery, Cloud Storage, Dataflow, Pub/Sub, Composer, Cloud SQL/Spanner Python / Java / SQL Data modeling (OLTP, OLAP, Star/Snowflake schema) Experience with real-time data processing, streaming architectures, and batch ETL pipelines Good understanding of IAM, networking, security models, and cost optimization on GCP Prior experience in leading cloud data transformation projects Excellent communication and stakeholder management skills Preferred Qualifications: GCP Professional Data Engineer / Architect Certification Experience with Terraform, CI/CD, GitOps, Looker / Data Studio / Tableau for analytics Exposure to AI/ML use cases and MLOps on GCP Experience working in agile environments and client-facing roles What We Offer: Opportunity to work on large-scale data modernization projects with global clients A fast-growing company with a strong tech and people culture Competitive salary, benefits, and flexibility Collaborative environment that values innovation and leadership

As a data engineer, you will be expected to be proficient in Python, SQL, and either Java or Scala, especially for Spark/Beam pipelines. Experience with BigQuery, Dataflow, Apache Beam, Airflow, and Kafka will be beneficial for this role. You will be responsible for building scalable batch and streaming pipelines to support machine learning or campaign analytics. Familiarity with ad tech, bid logs, or event tracking pipelines is considered a plus. Your primary role will involve constructing the foundational data infrastructure to handle the ingestion, processing, and serving of bid logs, user events, and attribution data from various sources. Key responsibilities include building scalable data pipelines for real-time and batch ingestion from DSPs, attribution tools, and order management systems. You will need to design clean and queryable data models to facilitate machine learning training and campaign optimization. Additionally, you will be required to enable data joins across 1st, 2nd, and 3rd-party data sets such as device, app, geo, and segment information. Optimizing pipelines for freshness, reliability, and cost efficiency is crucial, along with supporting event-level logging of auction wins, impressions, conversions, and click paths. The ideal candidate for this role should possess skills in Apache Beam, Airflow, Kafka, Scala, SQL, BigQuery, attribution, Java, Dataflow, Spark, machine learning, and Python. If you are enthusiastic about data engineering and have a background in building scalable data pipelines, this position could be a great fit for you.,

Software Engineer - Lead/Sr.Engineer Bachelor in Computer Science, Engineering, or equivalent experience 7+ years of experience in core JAVA, Spring Framework (Required) 2 years of Cloud experience (GCP, AWS, Azure, GCP preferred ) (Required) Experience in big data processing, on a distributed system. (required) Experience in databases RDBMS, NoSQL databases Cloud natives. (Required) Experience in handling various data formats like Flat file, jSON, Avro, xml etc with defining the schemas and the contracts. (required) Experience in implementing the data pipeline (ETL) using Dataflow (Apache beam) Experience in Microservices and integration patterns of the APIs with data processing. Experience in data structure, defining and designing the data models.

-Design, develop & maintain data pipelines using GCP services: Dataflow, BigQuery, and Pub/Sub -Provisioning infrastructure on GCP using IaC with Terraform -Implement & manage data warehouse solutions -Monitor and resolve issues in data workflows Required Candidate profile -Expertise in GCP, Apache Beam, Dataflow, & BigQuery -Pro in Python, SQL, PySpark -Worked with Cloud Composer for orchestration -Solid understanding of DWH, ETL pipelines, and real-time data streaming

Job Title: Software Engineer GCP Data Engineering Work Mode: Remote Base Location: Bengaluru Experience Required: 4 to 6 Years Job Summary: We are seeking a Software Engineer with a strong background in GCP Data Engineering and a solid understanding of how to build scalable data processing frameworks. The ideal candidate will be proficient in data ingestion, transformation, and orchestration using modern cloud-native tools and technologies. This role requires hands-on experience in designing and optimizing ETL pipelines, managing big data workloads, and supporting data quality initiatives. Key Responsibilities: Design and develop scalable data processing solutions using Apache Beam, Spark, and other modern frameworks. Build and manage data pipelines on Google Cloud Platform (GCP) using services like Dataflow, Dataproc, Composer (Airflow), and BigQuery . Collaborate with data architects and analysts to understand data models and implement efficient ETL solutions. Leverage DevOps and CI/CD best practices for code management, testing, and deployment using tools like GitHub and Cloud Build. Ensure data quality, performance tuning, and reliability of data processing systems. Work with cross-functional teams to understand business requirements and deliver robust data infrastructure to support analytical use cases. Required Skills: 4 to 6 years of professional experience as a Data Engineer working on cloud platforms, preferably GCP . Proficiency in Java and Python with strong problem-solving and analytical skills. Hands-on experience with Apache Beam , Apache Spark , Dataflow , Dataproc , Composer (Airflow) , and BigQuery . Strong understanding of data warehousing concepts and ETL pipeline optimization techniques. Experience in cloud-based architectures and DevOps practices. Familiarity with version control (GitHub) and CI/CD pipelines . Preferred Skills: Exposure to modern ETL tools and data integration platforms. Experience with data governance, data quality frameworks , and metadata management. Familiarity with performance tuning in distributed data processing systems. Tech Stack: Cloud: GCP (Dataflow, BigQuery, Dataproc, Composer) Programming: Java, Python Frameworks: Apache Beam, Apache Spark DevOps: GitHub, CI/CD tools, Composer (Airflow) ETL/Data Tools: Data ingestion, transformation, and warehousing on GCP

Design, develop, test, and maintain Java applications on Google Cloud Platform (GCP) using Spring Boot and Apache Beam Collaborate with cross-functional teams to design and implement scalable solutions using microservices architecture

Job Description Position: GCP Dialogflow Location : Anywhere in India Work Mode : Hybrid/Remote Locations: Bangalore, Pune, Chennai, Hyderabad or Gurgaon then they might have to come to the office. 1-5+ years of experience in software design and development 1 years of experience in the data engineering field is preferred 1+ years of Hands-on experience in GCP cloud data implementation suite such as Big Query, Pub Sub, Data Flow/Apache Beam, Airflow/Composer, Cloud Storage, Experience in dialog flow and java programming is must Strong experience and understanding of very large-scale data architecture, solutioning, and operationalization of data warehouses, data lakes, and analytics platforms. Hands on Strong Experience in the below technology 1. GBQ Query 2. Python 3. Apache Airflow 4. SQL (BigQuery preferred) 5. Dialogflow Extensive hands-on experience working with data using SQL and Python Cloud Functions. Comparable skills in AWS and other cloud Big Data Engineering space is considered. Experience with agile development methodologies Excellent verbal and written communications skills with the ability to clearly present ideas, concepts, and solutions Bachelor's Degree in Computer Science, Information Technology, or closely related discipline