Jobs
Interviews

8521 Pyspark Jobs - Page 42

Setup a job Alert
JobPe aggregates results for easy application access, but you actually apply on the job portal directly.

5.0 - 9.0 years

0 Lacs

jaipur, rajasthan

On-site

As a Senior Data Engineer + AI, you will play a crucial role in designing and optimizing distributed data pipelines using PySpark, Apache Spark, and Databricks to cater to both analytics and AI workloads. Your expertise in PySpark, Apache Spark, and Databricks for batch and streaming data pipelines will be instrumental in contributing to high-impact programs with clients. Your strong SQL skills for data analysis, transformation, and modeling will enable you to drive data-driven decision-making and facilitate rapid insight generation. Your responsibilities will involve supporting RAG pipelines, embedding generation, and data pre-processing for LLM applications, as well as creating and maintaining interactive dashboards and BI reports using tools such as Power BI, Tableau, or Looker. You will collaborate with cross-functional teams, including AI scientists, analysts, and business teams, to ensure the successful delivery of use cases. In this role, you will need to have a solid understanding of data warehouse design, relational databases such as PostgreSQL, Snowflake, SQL Server, as well as data lakehouse architectures. Your familiarity with cloud services for data and AI, such as Azure, AWS, or GCP, will be essential for ensuring data pipeline monitoring, cost optimization, and scalability in cloud environments. Furthermore, your exposure to Generative AI, RAG, embedding models, and vector databases like FAISS, Pinecone, ChromaDB, as well as experience with Agentic AI frameworks such as LangChain, Haystack, CrewAI, will be beneficial. Your knowledge of MLflow, Delta Live Tables, or other Databricks-native AI tools, CI/CD, Git, Docker, and DevOps pipelines will also be advantageous in this role. If you have a background in consulting, enterprise analytics, or AI/ML product development, it will further enhance your ability to excel in this position. Your excellent problem-solving and collaboration skills, coupled with your ability to bridge engineering and business needs, will be key to your success as a Senior Data Engineer + AI.,

Posted 1 week ago

Apply

6.0 - 10.0 years

0 Lacs

hyderabad, telangana

On-site

As a part of Microsoft's Cloud Supply Chain (CSCP) organization, your role will be crucial in supporting the growth of Microsoft's Cloud business which includes AI technologies. The vision of CSCP is to empower customers to achieve more by providing Cloud Capacity Differentiated at Scale. The mission is to deliver capacity for all cloud services predictably through intelligent systems and continuous learning. The responsibilities of CSCP extend beyond traditional supply chain functions to include supportability, decommissioning, and disposition of Data centre assets on a global scale. Within the Cloud Manufacturing Operations and Fulfilment (CMOF) organization, your role will focus on developing scalable and secure data architecture to support analytics and business processes. You will lead the creation of data pipelines, models, and integration strategies to enable analytics and AI capabilities across CMOF. This position plays a critical role in aligning data infrastructure with Microsoft's evolving Security Future Initiative (SFI) and engineering best practices. Key Responsibilities: - Design and develop scalable data ingestion pipelines from multiple sources. - Implement data orchestration using tools like Spark, PySpark, and Python. - Develop ETL jobs to optimize data flow and reliability. - Design logical and physical data models to support near real-time analytics. - Perform data profiling and gap analysis for migration to next-gen platforms. - Ensure data models support scalability, privacy, and governance. - Adhere to Microsoft's SFI guidelines, data residency policies, and data privacy regulations. - Implement data security measures like data masking and encryption. - Collaborate with engineering teams to ensure system updates and data lineage tracking. - Enable self-service BI and analytics using tools like Power BI and Azure Synapse. - Create reusable datasets, data models, and visualizations aligned with business priorities. - Translate business requirements into technical specs for scalable data solutions. Qualifications: Required: - Bachelor's degree in computer science, MIS, Data Engineering, or equivalent. - 5-8 years of experience in building cloud-based data systems and ETL frameworks. - Proficiency in relational databases, cloud-based data systems, and data orchestration tools. - Experience with visualization tools like Microsoft Power Platform and Fabric. Preferred: - Strong foundation in data modeling, warehousing, and data lake architecture. - Familiarity with ERP systems such as SAP and Dynamics 365. - Experience in modern development practices, agile methodologies, and version control. - Hands-on experience in data security, compliance controls, and governance frameworks. - Knowledge of AI applications for automated learning. Key Competencies: - Strong business acumen and strategic alignment of data capabilities. - Deep understanding of data privacy, compliance, and lifecycle management. - Excellent collaboration and communication skills across global teams. - Self-starter mindset with the ability to thrive in a fast-paced environment. - Strong analytical thinking, problem-solving skills, and continuous improvement mindset. - Ability to drive change and promote a data-driven culture within the organization.,

Posted 1 week ago

Apply

0.0 - 6.0 years

0 Lacs

Chennai, Tamil Nadu

On-site

Role: Senior Analyst - Data Engineering Experience: 4 to 6 years Location: Chennai, Tamil Nadu , India (CHN) Job Description: A highly skilled and motivated Senior Engineer with deep expertise in the Databricks platform to join our growing data engineering and analytics team. As a Senior Engineer, you will play a crucial role in designing, building, and optimizing our data pipelines, data lakehouse solutions, and analytics infrastructure on Databricks. You will collaborate closely with data scientists, analysts, and other engineers to deliver high-quality, scalable, and reliable data solutions that drive business insights and decision-making. Job Responsibilities: Design, develop, and maintain scalable and robust data pipelines and ETL/ELT processes using Databricks, Spark (PySpark, Scala), Delta Lake, and related technologies. Architect and implement data lakehouse solutions on Databricks, ensuring data quality, integrity, and performance. Develop and optimize data models for analytical and reporting purposes within the Databricks environment. Implement and manage data governance and security best practices within the Databricks platform, including Unity Catalog and RBAC. Utilize Databricks Delta Live Tables (DLT) to build and manage reliable data pipelines. Implement and leverage Change Data Feed (CDF) for efficient data synchronization and updates. Monitor and troubleshoot data pipelines and system performance on the Databricks platform. Collaborate with data scientists and analysts to understand their data requirements and provide efficient data access and processing solutions. Participate in code reviews, ensuring adherence to coding standards and best practices. Contribute to the development of technical documentation and knowledge sharing within the team. Stay up-to-date with the latest advancements in Databricks and related data technologies. Mentor and guide junior engineers on the team. Participate in the planning and execution of data-related projects and initiatives. Skills Required: Databricks, SQL, Pyspark, Python Data modeling, DE concepts Job Snapshot Updated Date 24-07-2025 Job ID J_3897 Location Chennai, Tamil Nadu, India Experience 4 - 6 Years Employee Type Permanent

Posted 1 week ago

Apply

4.0 years

0 Lacs

Bengaluru, Karnataka

On-site

Category: Software Development/ Engineering Main location: India, Karnataka, Bangalore Position ID: J0725-0904 Employment Type: Full Time Position Description: Founded in 1976, CGI is among the world's largest independent IT and business consulting services firms. With 94,000 consultants and professionals globally, CGI delivers an end-to-end portfolio of capabilities, from strategic IT and business consulting to systems integration, managed IT and business process services, and intellectual property solutions. CGI works with clients through a local relationship model complemented by a global delivery network that helps clients digitally transform their organizations and accelerate results. CGI Fiscal 2024 reported revenue is CA$14.68 billion, and CGI shares are listed on the TSX (GIB.A) and the NYSE (GIB). Learn more at cgi.com. Job Title: Databricks Developer / Engineer Position: SSE / LA Experience: 4+ years of experience Category: Software Development Job location: Bangalore / Chennai / Hyderabad / Pune Position ID: J0725-0904 Work Type: Hybrid Employment Type: Full Time / Permanent Qualification: Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field. Design, develop, and maintain scalable and reliable data pipelines using AWS services, PySpark, and Databricks. Collaborate with cross-functional teams to understand data requirements, identify data sources, and define data ingestion strategies. Implement data extraction, transformation, and loading (ETL) processes to enable efficient data integration from various sources. Hands-on experience in developing and optimizing Databricks data pipelines using PySpark. Proficient in SQL, Python, and ETL processes Optimize and tune data pipelines to ensure high performance, scalability, and data quality. Monitor and troubleshoot data pipelines to identify and resolve issues in a timely manner. Collaborate with data scientists and analysts to provide them with clean, transformed, and reliable data for analysis and modeling. Develop and maintain data documentation, including data lineage, data dictionaries, and metadata management. Level of exp in Databricks: E4 Should have worked in different functional perimeters (e.g. finance, HR, Geology, HSE…), so that they are open-minded and able to adapt to the HR domain which requires to have a deep understanding of functional need to make good quality developments (otherwise the calculation or the developed rule gives wrong results (e.g. negative headcount of employees, wrong columns used to make calculation, calculation in itself that wouldn’t follow the logical HR rule given by the business…) CGI is an equal opportunity employer. In addition, CGI is committed to providing accommodation for people with disabilities in accordance with provincial legislation. Please let us know if you require reasonable accommodation due to a disability during any aspect of the recruitment process and we will work with you to address your needs. #LI-GB9 Skills: PowerBuilder Python What you can expect from us: Together, as owners, let’s turn meaningful insights into action. Life at CGI is rooted in ownership, teamwork, respect and belonging. Here, you’ll reach your full potential because… You are invited to be an owner from day 1 as we work together to bring our Dream to life. That’s why we call ourselves CGI Partners rather than employees. We benefit from our collective success and actively shape our company’s strategy and direction. Your work creates value. You’ll develop innovative solutions and build relationships with teammates and clients while accessing global capabilities to scale your ideas, embrace new opportunities, and benefit from expansive industry and technology expertise. You’ll shape your career by joining a company built to grow and last. You’ll be supported by leaders who care about your health and well-being and provide you with opportunities to deepen your skills and broaden your horizons. Come join our team—one of the largest IT and business consulting services firms in the world.

Posted 1 week ago

Apply

5.0 - 10.0 years

0 Lacs

thiruvananthapuram, kerala

On-site

You are an experienced GCP Data Engineer with 5 to 10 years of experience. You will be responsible for designing and implementing efficient data solutions on the Google Cloud Platform (GCP) for UST. Your role will involve developing end-to-end data pipelines with a focus on scalability and performance, as well as maintaining ETL workflows for seamless data processing. Your expertise in GCP services such as BigQuery, Cloud Storage, and Dataflow will be crucial for effective data engineering. You will also leverage PySpark for data transformations, ensuring high-quality and well-structured output by implementing data cleansing, enrichment, and validation processes. To succeed in this role, you must have proven experience as a Data Engineer with a strong emphasis on GCP, proficiency in GCP services, and expertise in PySpark for data processing and analytics. Experience with data modeling, ETL processes, and data warehousing, as well as proficiency in programming languages such as Python, SQL, or Scala, will be required. Relevant certifications in GCP or data engineering are a plus. Join UST, a global digital transformation solutions provider, and work alongside the world's best companies to make a real impact through transformation. With over 30,000 employees in 30 countries, UST is dedicated to embedding innovation and agility into their clients" organizations, touching billions of lives in the process.,

Posted 2 weeks ago

Apply

3.0 years

0 Lacs

Bhubaneswar, Odisha, India

On-site

Project Role : Application Developer Project Role Description : Design, build and configure applications to meet business process and application requirements. Must have skills : Python (Programming Language) Good to have skills : NA Minimum 3 Year(s) Of Experience Is Required Educational Qualification : 15 years full time education Summary: As an Application Developer, you will engage in the design, construction, and configuration of applications tailored to fulfill specific business processes and application requirements. Your typical day will involve collaborating with team members to understand project needs, developing innovative solutions, and ensuring that applications are optimized for performance and usability. You will also participate in testing and debugging processes to guarantee the quality and functionality of the applications you create, while continuously seeking ways to enhance existing systems and processes. Roles & Responsibilities: - Expected to perform independently and become an SME. - Required active participation/contribution in team discussions. - Contribute in providing solutions to work related problems. - Assist in the documentation of application specifications and user guides. - Engage in code reviews to ensure adherence to best practices and standards. Professional & Technical Skills: - Good to have skills - Pyspark, AWS, Airflow, Databricks, SQL, SCALA - Experience should be 4+ years in Python - Candidate must be a strong Hands-on senior Developer - Candidate must possess good technical / non-technical communication skills to highlight areas of concern/risks - Should have good troubleshooting skills to do RCA of prod support related issues Additional Information: - The candidate should have minimum 3 years of experience in Python (Programming Language). - This position is based at our Bengaluru office. - A 15 years full time education is required. - Candidate must be willing to work in Shift B i.e. daily 9PM/10 PM IST, 15 years full time education

Posted 2 weeks ago

Apply

5.0 years

0 Lacs

Bhubaneswar, Odisha, India

On-site

Project Role : Application Lead Project Role Description : Lead the effort to design, build and configure applications, acting as the primary point of contact. Must have skills : Python (Programming Language), Apache Spark, AWS Architecture, PySpark Good to have skills : NA Minimum 5 Year(s) Of Experience Is Required Educational Qualification : 15 years full time education Summary: As an Application Lead, you will lead the effort to design, build, and configure applications, acting as the primary point of contact. Your typical day will involve collaborating with various teams to ensure project milestones are met, facilitating discussions to address challenges, and guiding your team in implementing effective solutions. You will also engage in strategic planning sessions to align project goals with organizational objectives, ensuring that all stakeholders are informed and involved in the development process. Your role will be pivotal in driving innovation and efficiency within the application development lifecycle, fostering a collaborative environment that encourages team growth and success. Roles & Responsibilities: - Expected to be an SME. - Collaborate and manage the team to perform. - Responsible for team decisions. - Engage with multiple teams and contribute on key decisions. - Provide solutions to problems for their immediate team and across multiple teams. - Facilitate knowledge sharing sessions to enhance team capabilities. - Monitor project progress and implement necessary adjustments to meet deadlines. Professional & Technical Skills: - Must To Have Skills: Proficiency in Python (Programming Language), Apache Spark, AWS Architecture, PySpark. - Strong understanding of software development methodologies. - Experience with application design and architecture. - Familiarity with cloud computing concepts and services. - Ability to troubleshoot and optimize application performance. Additional Information: - The candidate should have minimum 5 years of experience in Python (Programming Language). - This position is based in Mumbai. - A 15 years full time education is required.

Posted 2 weeks ago

Apply

4.0 - 8.0 years

0 Lacs

chennai, tamil nadu

On-site

At EY, you'll have the chance to build a career as unique as you are, with the global scale, support, inclusive culture and technology to become the best version of you. And we're counting on your unique voice and perspective to help EY become even better, too. Join us and build an exceptional experience for yourself, and a better working world for all. The opportunity As a Senior BI Consultant, you will be responsible for supporting and enhancing Business Intelligence and Data Analytics platforms with a primary focus on Power BI and Databricks. You will work across global engagements, helping clients translate complex data into actionable insights. This role involves day-to-day application management, dashboard development, troubleshooting, and stakeholder collaboration to ensure high data quality, performance, and availability. Your Key Responsibilities BI Support & Monitoring: Provide daily application support for Power BI dashboards and Databricks pipelines, resolving incidents, fulfilling service requests, and implementing enhancements. Dashboard Development: Design, develop, and maintain Power BI reports and data models tailored to evolving business requirements. Root Cause Analysis: Investigate and resolve data/reporting issues, bugs, and performance bottlenecks through detailed root cause analysis. Requirement Gathering: Collaborate with business users and technical stakeholders to define BI requirements and translate them into scalable solutions. Documentation: Maintain technical documentation, including data flows, dashboard usage guides, and QA test scripts. On-Call & Shift Support: Participate in shift rotations and be available for on-call support for critical business scenarios. Integration & Data Modeling: Ensure effective data integration from diverse systems and maintain clean, performant data models within Power BI and Databricks. Skills and attributes for success Hands-on expertise in Power BI, including DAX, data modeling, and report optimization Working experience in Databricks, especially with Delta Lake, SQL, and PySpark for data transformation Familiarity with ETL/ELT design, especially within Azure data ecosystems Ability to troubleshoot BI performance issues and manage service tickets efficiently Strong communication skills to interact with global stakeholders and cross-functional teams Ability to manage and prioritize multiple support tasks in a fast-paced environment To qualify for the role, you must have 3-7 years of experience in Business Intelligence and Application Support Strong hands-on skills in Power BI and Databricks, preferably in a global delivery model Working knowledge of ETL processes, data validation, and performance tuning Familiarity with ITSM practices for service request, incident, and change management Willingness to work in rotational shifts and support on-call requirements Bachelor's degree in Computer Science, Engineering, or equivalent work experience Willingness to work in a 24x7 rotational shift-based support environment. No location constraints Technologies and Tools Must haves Power BI: Expertise in report design, data modeling, and DAX Databricks: Experience with notebooks, Delta Lake, SQL, and PySpark Azure Ecosystem: Familiarity with Azure Data Lake and Azure Synapse (consumer layer) ETL & Data Modelling: Good understanding of data integration and modeling best practices ITSM Tools: Experience with ServiceNow or equivalent for ticketing and change management Good to have Data Integration: Experience integrating with ERP, CRM, or POS systems Python: For data transformation and automation scripting Monitoring: Awareness of Azure Monitor or Log Analytics for pipeline health Certifications: Microsoft Certified Data Analyst Associate or Databricks Certified Data Engineer Associate Industry Exposure: Experience in retail or consumer goods industries What we look for People with client orientation, experience and enthusiasm to learn new things in this fast-moving environment. An opportunity to be a part of a market-leading, multi-disciplinary team of hundreds of professionals. Opportunities to work with EY BI application maintenance, practices globally with leading businesses across a range of industries. What we offer EY Global Delivery Services (GDS) is a dynamic and truly global delivery network. We work across six locations - Argentina, China, India, the Philippines, Poland and the UK - and with teams from all EY service lines, geographies and sectors, playing a vital role in the delivery of the EY growth strategy. From accountants to coders to advisory consultants, we offer a wide variety of fulfilling career opportunities that span all business disciplines. In GDS, you will collaborate with EY teams on exciting projects and work with well-known brands from across the globe. We'll introduce you to an ever-expanding ecosystem of people, learning, skills and insights that will stay with you throughout your career. Continuous learning: You'll develop the mindset and skills to navigate whatever comes next. Success as defined by you: We'll provide the tools and flexibility, so you can make a meaningful impact, your way. Transformative leadership: We'll give you the insights, coaching and confidence to be the leader the world needs. Diverse and inclusive culture: You'll be embraced for who you are and empowered to use your voice to help others find theirs. About EY EY | Building a better working world EY exists to build a better working world, helping to create long-term value for clients, people and society and build trust in the capital markets. Enabled by data and technology, diverse EY teams in over 150 countries provide trust through assurance and help clients grow, transform and operate. Working across assurance, consulting, law, strategy, tax and transactions, EY teams ask better questions to find new answers for the complex issues facing our world today. If you can demonstrate that you meet the criteria above, please contact us as soon as possible. The exceptional EY experience. It's yours to build. EY | Building a better working world EY exists to build a better working world, helping to create long-term value for clients, people and society and build trust in the capital markets. Enabled by data and technology, diverse EY teams in over 150 countries provide trust through assurance and help clients grow, transform and operate. Working across assurance, consulting, law, strategy, tax and transactions, EY teams ask better questions to find new answers for the complex issues facing our world today.,

Posted 2 weeks ago

Apply

6.0 years

0 Lacs

Greater Kolkata Area

On-site

Line of Service Advisory Industry/Sector Not Applicable Specialism Microsoft Management Level Senior Associate Job Description & Summary At PwC, our people in software and product innovation focus on developing cutting-edge software solutions and driving product innovation to meet the evolving needs of clients. These individuals combine technical experience with creative thinking to deliver innovative software products and solutions. Those in software engineering at PwC will focus on developing innovative software solutions to drive digital transformation and enhance business performance. In this field, you will use your knowledge to design, code, and test cutting-edge applications that revolutionise industries and deliver exceptional user experiences. Why PWC At PwC, you will be part of a vibrant community of solvers that leads with trust and creates distinctive outcomes for our clients and communities. This purpose-led and values-driven work, powered by technology in an environment that drives innovation, will enable you to make a tangible impact in the real world. We reward your contributions, support your wellbeing, and offer inclusive benefits, flexibility programmes and mentorship that will help you thrive in work and life. Together, we grow, learn, care, collaborate, and create a future of infinite experiences for each other. Learn more about us. At PwC, we believe in providing equal employment opportunities, without any discrimination on the grounds of gender, ethnic background, age, disability, marital status, sexual orientation, pregnancy, gender identity or expression, religion or other beliefs, perceived differences and status protected by law. We strive to create an environment where each one of our people can bring their true selves and contribute to their personal growth and the firm’s growth. To enable this, we have zero tolerance for any discrimination and harassment based on the above considerations. Responsibilities: We are seeking a highly skilled and experienced Python developer with 6-7 years of hands-on experience in software development. Key Responsibilities: - Design, develop, test and maintain robust and scalable backend applications using FastAPI deliver high- performance APIs. - Write reusable efficient code following best practices - Collaborate with cross-functional teams, integrate user-facing elements with server-side logic - Architect and implement distributed, scalable microservices leveraging Temporal workflows for orchestrating complex processes. - Participate in code reviews and mentor junior developers - Debug and resolve technical issues and production incidents - Follow agile methodologies and contribute to sprint planning and estimations - Strong communication and collaboration skills - Relevant certifications are a plus Required Skills: - Strong proficiency in Python 3.x. - Collaborate closely with DevOps to implement CI/CD pipelines for Python projects, ensuring smooth deployment to production environments. - Integrate with various databases (e.g., Cosmos DB,) and message queues (e.g., Kafka, eventhub) for seamless backend operations. - Experience in one or more Python frameworks (Django, Flask, FastAPI) - Develop and maintain unit and integration tests using frameworks like pytest and unittest to ensure code quality and reliability. - Experience with Docker, Kubernetes, and cloud environments (AWS, GCP, or Azure) for deploying and managing Python services. - Familiarity with asynchronous programming (e.g., asyncio, aiohttp) and event-driven architectures. - Strong skill in PySpark for large-scale data processing - Solid understanding of Object-Oriented Programming and design principles - Proficient in using version control systems like Git Mandatory skill sets: Python Developer Preferred skill sets: Experience with Docker, Kubernetes, and cloud environments (AWS, GCP, or Azure) for deploying and managing Years of experience required: 4-7 Years Education qualification: B.Tech/B.E./MCA Education (if blank, degree and/or field of study not specified) Degrees/Field of Study required: Bachelor of Technology, Bachelor of Engineering Degrees/Field of Study preferred: Certifications (if blank, certifications not specified) Required Skills Python (Programming Language) Optional Skills Acceptance Test Driven Development (ATDD), Acceptance Test Driven Development (ATDD), Accepting Feedback, Active Listening, Analytical Thinking, Android, API Management, Appian (Platform), Application Development, Application Frameworks, Application Lifecycle Management, Application Software, Business Process Improvement, Business Process Management (BPM), Business Requirements Analysis, C#.NET, C++ Programming Language, Client Management, Code Review, Coding Standards, Communication, Computer Engineering, Computer Science, Continuous Integration/Continuous Delivery (CI/CD), Creativity {+ 46 more} Desired Languages (If blank, desired languages not specified) Travel Requirements Available for Work Visa Sponsorship? Government Clearance Required? Job Posting End Date

Posted 2 weeks ago

Apply

3.0 - 7.0 years

0 Lacs

chennai, tamil nadu

On-site

You should possess a degree in computer science, engineering, or a related field along with 3-4 years of experience in the field. Your primary skillsets should include AWS, Pyspark, SQL, Databricks, and Python. Additionally, you should have experience with data integration and pipeline development, specifically using AWS Cloud services such as Apache Spark, Glue, Kafka, Kinesis, and Lambda in S3 Redshift, RDS, and MongoDB/DynamoDB ecosystems. Experience with Databricks and Redshift is considered a major plus. You must also have proficiency in SQL, especially in the development of data warehouse projects and applications using Oracle and SQL Server. Furthermore, you should have a strong background in Python development, particularly in PySpark within an AWS Cloud environment. Familiarity with SQL and NoSQL databases like MySQL, Postgres, DynamoDB, and Elasticsearch is essential. Experience with workflow management tools such as Airflow and knowledge of AWS cloud services like RDS, AWS Lambda, AWS Glue, AWS Athena, and EMR are also required. Familiarity with equivalent tools in the GCP stack will be considered a plus. Any experience with ETL tools, Github, and DevOps (CI-CD) will be beneficial for this role. Snowflake and Palantir Foundry knowledge is considered a good-to-have skill. Overall, you will be responsible for designing, developing, testing, and supporting data pipelines and applications as an AWS Data Engineer.,

Posted 2 weeks ago

Apply

5.0 - 9.0 years

0 Lacs

chennai, tamil nadu

On-site

As an offshore Techlead with Databricks engineer experience, your primary responsibility will be to lead the team from offshore. You will be tasked with developing and maintaining a metadata-driven generic ETL framework for automating ETL code. This includes designing, building, and optimizing ETL/ELT pipelines using Databricks (PySpark/SQL) on AWS. Your role will involve ingesting data from various structured and unstructured sources such as APIs, RDBMS, flat files, and streaming. Moreover, you will be expected to develop and maintain robust data pipelines for both batch and streaming data using Delta Lake and Spark Structured Streaming. Implementing data quality checks, validations, and logging mechanisms will also be part of your responsibilities. It will be crucial for you to optimize pipeline performance, cost, and reliability, while collaborating with data analysts, BI, and business teams to deliver fit-for-purpose datasets. You will also support data modeling efforts, including star, snowflake schemas, and de-norm tables approach, as well as assist with data warehousing initiatives. Working with orchestration tools like Databricks Workflows to schedule and monitor pipelines will be essential. Following best practices for version control, CI/CD, and collaborative development is expected from you. In terms of required skills, you should have hands-on experience in ETL/Data Engineering roles and strong expertise in Databricks (PySpark, SQL, Delta Lake), with Databricks Data Engineer Certification being preferred. Experience with Spark optimization, partitioning, caching, and handling large-scale datasets is crucial. Proficiency in SQL and scripting in Python or Scala is required, along with a solid understanding of data lakehouse/medallion architectures and modern data platforms. Additionally, experience working with cloud storage systems like AWS S3, familiarity with DevOps practices (Git, CI/CD, Terraform, etc.), and strong debugging, troubleshooting, and performance-tuning skills are necessary for this role. In summary, as an offshore Techlead with Databricks engineer experience, you will play a vital role in developing and maintaining ETL frameworks, optimizing data pipelines, collaborating with various teams, and ensuring data quality and reliability. Your expertise in Databricks, ETL processes, data modeling, and cloud platforms will be instrumental in driving the success of the projects you undertake. About Virtusa: At Virtusa, we value teamwork, quality of life, and professional and personal development. Joining our team means becoming part of a global workforce of 27,000 individuals who are dedicated to your growth. We offer exciting projects, opportunities, and exposure to state-of-the-art technologies throughout your career with us. We believe in collaboration, a team-oriented environment, and providing a dynamic space for great minds to nurture new ideas and achieve excellence.,

Posted 2 weeks ago

Apply

8.0 - 12.0 years

0 Lacs

karnataka

On-site

Working with data on a day-to-day basis excites you, and you are interested in building robust data architecture to identify data patterns and optimize data consumption for customers who will forecast and predict actions based on data. If this excites you, then working in our intelligent automation team at Schneider AI Hub is the perfect fit for you. As a Lead Data Engineer at Schneider AI Hub, you will play a crucial role in the AI transformation of Schneider Electric by developing AI-powered solutions. Your responsibilities will include expanding and optimizing data and data pipeline architecture, ensuring optimal data flow and collection for cross-functional teams, and supporting software engineers, data analysts, and data scientists on data initiatives. You will be responsible for creating and maintaining optimal data pipeline architecture, designing the right schema to support functional requirements, and building production data pipelines from ingestion to consumption. Additionally, you will create preprocessing and postprocessing for various forms of data, develop data visualization and business intelligence tools, and implement internal process improvements for automating manual data processes. To qualify for this role, you should hold a bachelor's or master's degree in computer science, information technology, or other quantitative fields and have a minimum of 8 years of experience as a data engineer supporting large data transformation initiatives related to machine learning. Strong analytical skills, experience with Azure cloud services, ETLs using Spark, and proficiency in scripting languages like Python and Pyspark are essential requirements for this position. As a team player committed to the success of the team and projects, you will collaborate with various stakeholders to ensure data delivery architecture is consistent and secure across multiple data centers. Join us at Schneider Electric, where we create connected technologies that reshape industries, transform cities, and enrich lives, with a diverse and inclusive culture that values the contribution of every individual. If you are passionate about success and eager to contribute to cutting-edge projects, we invite you to be part of our dynamic team at Schneider Electric in Bangalore, India.,

Posted 2 weeks ago

Apply

2.0 - 6.0 years

0 Lacs

jaipur, rajasthan

On-site

We are searching for a skilled and adaptable Data Engineer with proficiency in PySpark, Apache Spark, and Databricks, combined with knowledge in analytics, data modeling, and Generative AI/Agentic AI solutions. This position suits individuals who excel at the convergence of data engineering, AI systems, and business insights, contributing to impactful programs with clients. Your responsibilities will include designing, constructing, and enhancing distributed data pipelines utilizing PySpark, Apache Spark, and Databricks to cater to both analytics and AI workloads. You will also be tasked with supporting RAG pipelines, embedding generation, and data pre-processing for LLM applications. Additionally, creating and maintaining interactive dashboards and BI reports using tools like Power BI, Tableau, or Looker for business stakeholders and consultants will be part of your role. Furthermore, your duties will involve conducting adhoc data analysis to facilitate data-driven decision-making and rapid insight generation. You will be expected to develop and sustain robust data warehouse schemas, star/snowflake models, and provide support for data lake architecture. Integration with and support for LLM agent frameworks like LangChain, LlamaIndex, Haystack, or CrewAI for intelligent workflow automation will also fall under your purview. In addition, ensuring data pipeline monitoring, cost optimization, and scalability in cloud environments (Azure/AWS/GCP) will be important aspects of your work. Collaboration with cross-functional teams, including AI scientists, analysts, and business teams to drive use-case delivery, is key. Lastly, maintaining robust data governance, lineage, and metadata management practices using tools such as Azure Purview or DataHub will also be part of your responsibilities.,

Posted 2 weeks ago

Apply

5.0 - 9.0 years

0 Lacs

hyderabad, telangana

On-site

We are looking for a highly experienced Senior Data Engineer to lead our data migration projects from On-Premise systems to Azure Cloud, utilizing Azure Databricks, PySpark, SQL, and Python. The successful candidate will be responsible for designing and implementing robust, scalable cloud data solutions to enhance business operations and decision-making processes. Responsibilities: Design and implement end-to-end data solutions using Azure Databricks, PySpark, MS SQL Server, and Python for data migration from on-premise to Azure Cloud. Develop architectural blueprints and detailed documentation for data migration strategies and execution plans. Construct, test, and maintain optimal data pipeline architectures across multiple sources and destinations within Azure Cloud environments. Leverage PySpark within Azure Databricks to perform complex data transformations, aggregations, and optimizations. Ensure seamless migration of large-scale databases from on-premise systems to Azure Cloud, maintaining data integrity and compliance. Handle technical escalations through effective diagnosis and troubleshooting of client queries. Manage and resolve technical roadblocks/escalations as per SLA and quality requirements. If unable to resolve the issues, timely escalate the issues to TA & SES. Provide product support and resolution to clients by performing a question diagnosis while guiding users through step-by-step solutions. Troubleshoot all client queries in a user-friendly, courteous, and professional manner. Offer alternative solutions to clients (where appropriate) with the objective of retaining customers and clients business. Organize ideas and effectively communicate oral messages appropriate to listeners and situations. Follow up and make scheduled call-backs to customers to record feedback and ensure compliance with contract SLAs. Performance Parameters: 1. Process: No. of cases resolved per day, compliance to process and quality standards, meeting process level SLAs, Pulse score, Customer feedback, NSAT/ESAT. 2. Team Management: Productivity, efficiency, absenteeism. 3. Capability development: Triages completed, Technical Test performance. Mandatory Skills: Talend Big Data Experience: 5-8 Years Join us at Wipro as we reinvent our world together. We are an end-to-end digital transformation partner with the boldest ambitions, seeking individuals inspired by reinvention of themselves, their careers, and their skills. Be a part of a business powered by purpose and a place that empowers you to design your reinvention. Come to Wipro. Realize your ambitions. Applications from people with disabilities are explicitly welcome.,

Posted 2 weeks ago

Apply

3.0 - 7.0 years

0 Lacs

noida, uttar pradesh

On-site

We are looking for a skilled and passionate AWS Data Engineer to join our dynamic data engineering team. The ideal candidate will have strong experience in building scalable data pipelines and solutions using AWS, PySpark, Databricks, and Snowflake. Key Responsibilities: Design, develop, and maintain large-scale data pipelines on AWS using PySpark and Databricks. Work with Snowflake to perform data warehousing tasks including data loading, transformation, and optimization. Build efficient and scalable ETL/ELT workflows to support analytics and reporting. Implement data quality checks, monitoring, and performance tuning of ETL processes. Ensure data governance, security, and compliance in all solutions developed. Required Skills & Qualifications: Bachelor's degree in Computer Science, Information Technology, or a related field. 3+ years of experience as a Data Engineer with strong exposure to AWS cloud services (S3, Lambda, Glue, Redshift, etc.). Hands-on experience with PySpark and Databricks for big data processing. Proficiency in working with Snowflake.,

Posted 2 weeks ago

Apply

5.0 - 9.0 years

0 Lacs

maharashtra

On-site

As a Senior Digital Solutions Consultant at Worley, you will have the opportunity to work on the world's most complex projects and be a part of a collaborative and inclusive team. Your role will be varied and challenging, allowing you to contribute to innovative solutions that drive sustainability in projects. Worley, a global professional services company specializing in energy, chemicals, and resources, is at the forefront of bridging the transition to more sustainable energy sources. By partnering with customers, Worley delivers integrated data-centric solutions throughout the lifecycle of assets, from consulting and engineering to decommissioning and remediation. As part of your role, you will be responsible for developing and implementing data pipelines to ingest and collect data from various sources into a centralized data platform. You will work on optimizing and troubleshooting AWS Glue jobs for performance and reliability, using Python and PySpark to handle large data volumes efficiently. Collaboration with data architects to design and implement data models that meet business requirements will be essential. Additionally, you will create and maintain ETL processes using Airflow, Python, and PySpark to move and transform data between systems. Monitoring data pipeline performance, managing and optimizing databases, and proficiency in Infrastructure as Code tools will also be key aspects of your responsibilities. Your expertise in event-driven integrations, batch-based, and API-led data integrations will be valuable, along with proficiency in CICD pipelines. To excel in this role, you should have over 5 years of experience in developing integration projects in an agile or waterfall-based project environment. Proficiency in Python, PySpark, SQL programming, and hands-on experience with AWS services like Glue, Airflow, Dynamo DB, Redshift, and S3 buckets will be required. Familiarity with CI/CD pipelines, web service development, and a degree in Computer Science or related fields are desirable qualifications. At Worley, we are committed to creating a values-inspired culture that fosters innovation, belonging, and connection. We believe in reskilling our workforce and supporting their transition to become experts in low carbon energy infrastructure and technology. Join us to unlock your potential, explore diverse opportunities, and be part of delivering sustainable change.,

Posted 2 weeks ago

Apply

5.0 - 9.0 years

0 Lacs

haryana

On-site

As a Senior Data Engineer (Azure MS Fabric) at Srijan Technologies PVT LTD, located in Gurugram, Haryana, India, you will be responsible for designing and developing scalable data pipelines using Microsoft Fabric. Your role will involve working on both batch and real-time ingestion and transformation, integrating with Azure Data Factory for smooth data flow, and collaborating with data architects to implement governed Lakehouse models in Microsoft Fabric. You will be expected to monitor and optimize the performance of data pipelines and notebooks in Microsoft Fabric, applying tuning strategies to reduce costs, improve scalability, and ensure reliable data delivery. Collaboration with cross-functional teams, including BI developers, analysts, and data scientists, is essential to gather requirements and build high-quality datasets. Additionally, you will need to document pipeline logic, lakehouse architecture, and semantic layers clearly, following development standards and contributing to internal best practices for Microsoft Fabric-based solutions. To excel in this role, you should have at least 5 years of experience in data engineering within the Azure ecosystem, with hands-on experience in Microsoft Fabric, Lakehouse, Dataflows Gen2, and Data Pipelines. Proficiency in building and orchestrating pipelines with Azure Data Factory and/or Microsoft Fabric Dataflows Gen2 is required, along with a strong command of SQL, PySpark, and Python applied to data integration and analytical workloads. Experience in optimizing pipelines and managing compute resources for cost-effective data processing in Azure/Fabric is also crucial. Preferred skills for this role include experience in the Microsoft Fabric ecosystem, familiarity with OneLake, Delta Lake, and Lakehouse principles, expert knowledge of PySpark, strong SQL, and Python scripting within Microsoft Fabric or Databricks notebooks, and understanding of Microsoft Purview, Unity Catalog, or Fabric-native tools for metadata, lineage, and access control. Exposure to DevOps practices for Fabric and Power BI, as well as knowledge of Azure Databricks for Spark-based transformations and Delta Lake pipelines, would be considered a plus. If you are passionate about developing efficient data solutions in a collaborative environment and have a strong background in data engineering within the Azure ecosystem, this role as a Senior Data Engineer at Srijan Technologies PVT LTD could be the perfect fit for you. Apply now to be a part of a dynamic team driving innovation in data architecture and analytics.,

Posted 2 weeks ago

Apply

8.0 years

0 Lacs

Hyderabad, Telangana, India

On-site

The Data Engineer will build and maintain data pipelines and workflows that support ML, BI, analytics, and software products. This individual will work closely with data scientists, engineers, analysts, software developers, and SMEs within the business to deliver new and exciting products and services. The main objectives are to develop data pipelines and fully automated workflows. The primary platform used will be Palantir Foundry. Responsibilities: • Develop high-quality code for the core data stack, including a data integration hub, warehouse, and pipelines. • Build data flows for data acquisition, aggregation, and modeling, using both batch and streaming paradigms • Empower data scientists and data analysts to be as self-sufficient as possible by building core systems and developing reusable library code • Support and optimize data tools and associated cloud environments for consumption by downstream systems, data analysts, and data scientists • Ensure code, configuration, and other technology artifacts are delivered within agreed time schedules, and any potential delays are escalated in advance • Collaborate across developers as part of a SCRUM team, ensuring collective team productivity • Participate in peer reviews and QA processes to drive higher quality • Ensure that 100% of the code is well documented and maintained in the source code repository. • Strive for engineering excellence by simplifying, optimizing, and automating processes and workflows. • Ensures their workstation and all processes and procedures follow organization standards Experience And Skills: • Minimum of 8 years of professional experience as a data engineer • Hands-on experience with Palantir Foundry is Must • Experience with relational and dimensional database modelling (Relational, Kimball, or Data Vault) • Proven experience with aspects of the Data Pipeline (Data Sourcing, Transformations, Data Quality, etc.) • Bachelor’s or Master’s in Computer Science, Information Systems, or an engineering field • Prefer experience with event-driven architectures and data streaming pub/sub technologies such as IBM MQ, Kafka, or Amazon Kinesis. • Strong capabilities in Python, SQL, and stored procedures; interpersonal, communication, problem-solving, and critical thinking skills; agile/Scrum experience. • Prefer travel, transportation, or hospitality experience, especially with fleet management and vehicle maintenance. • Prefer experience with designing application data models for mobile or web applications

Posted 2 weeks ago

Apply

3.0 - 7.0 years

0 Lacs

pune, maharashtra

On-site

At Capgemini Invent, we believe that difference drives change. As inventive transformation consultants, we combine our strategic, creative, and scientific capabilities to collaborate closely with clients in delivering cutting-edge solutions. Join our team to lead transformation customized to address our client's challenges of today and tomorrow, informed and validated by science and data, superpowered by creativity and design, all underpinned by purpose-driven technology. What you will appreciate about working with us: We acknowledge the importance of flexible work arrangements to provide support. Whether it's remote work or flexible work hours, you will find an environment that fosters a healthy work-life balance. At the core of our mission lies your career growth. Our array of career growth programs and diverse professions are designed to assist you in exploring a world of opportunities. Equip yourself with valuable certifications in the latest technologies such as Generative AI. Your Role: We are seeking a skilled PySpark Developer with expertise in Azure Databricks (ADB) and Azure Data Factory (ADF) to become a part of our team. The ideal candidate will have a pivotal role in designing, developing, and implementing data solutions using PySpark for large-scale data processing and analytics. Your Profile: - Design, develop, and deploy PySpark applications and workflows on Azure Databricks for data transformation, cleansing, and aggregation. - Implement data pipelines using Azure Data Factory (ADF) to orchestrate ETL/ELT processes across heterogeneous data sources. - Conduct regular financial risk assessments to identify potential vulnerabilities in data processing workflows. - Collaborate with Data Engineers and Data Scientists to integrate and process structured and unstructured data sets into actionable insights. Capgemini is a global business and technology transformation partner, aiding organizations in accelerating their dual transition to a digital and sustainable world while making a tangible impact for enterprises and society. With a responsible and diverse group of 340,000 team members in more than 50 countries, Capgemini, with its strong over 55-year heritage, is trusted by clients to unlock the value of technology to address the entire breadth of their business needs. It provides end-to-end services and solutions leveraging strengths from strategy and design to engineering, all fueled by its market-leading capabilities in AI, generative AI, cloud, and data, combined with its deep industry expertise and partner ecosystem.,

Posted 2 weeks ago

Apply

2.0 - 10.0 years

0 Lacs

pune, maharashtra

On-site

You are an experienced Data Engineer with expertise in PySpark, Snowflake, and AWS, and you will be responsible for designing, developing, and optimizing data pipelines and workflows in a cloud-based environment. Your main focus will be leveraging AWS services, PySpark, and Snowflake for data processing and analytics. Your key responsibilities will include designing and implementing scalable ETL pipelines using PySpark on AWS, developing and optimizing data workflows for Snowflake integration, managing and configuring AWS services such as S3, Lambda, Glue, EMR, and Redshift, collaborating with data analysts and business teams to understand requirements and deliver solutions, ensuring data security and compliance with best practices in AWS and Snowflake environments, monitoring and troubleshooting data pipelines and workflows for performance and reliability, and writing efficient, reusable, and maintainable code for data processing and transformation. Required skills for this role include strong experience with AWS services (S3, Lambda, Glue, MSK, etc.), proficiency in PySpark for large-scale data processing, hands-on experience with Snowflake for data warehousing and analytics, a solid understanding of SQL and database optimization techniques, knowledge of data lake and data warehouse architectures, familiarity with CI/CD pipelines and version control systems (e.g., Git), strong problem-solving and debugging skills, experience with Terraform or CloudFormation for infrastructure as code, knowledge of Python for scripting and automation, familiarity with Apache Airflow for workflow orchestration, understanding of data governance and security best practices, and a certification in AWS or Snowflake is a plus. For education and experience, a Bachelors degree in Computer Science, Engineering, or related field with 6 to 10 years of experience is required, along with 5+ years of experience in AWS cloud engineering and 2+ years of experience with PySpark and Snowflake. This position falls under the Technology Job Family Group and the Digital Software Engineering Job Family, and it is a full-time role.,

Posted 2 weeks ago

Apply

5.0 - 9.0 years

0 Lacs

karnataka

On-site

You are a Senior Python Data Application Developer with a strong expertise in core Python and data-focused libraries. Your primary responsibility is to design, develop, and maintain data-driven applications optimized for performance and scalability. You will be building robust data pipelines, ETL processes, and APIs for integrating various data sources efficiently within the cloud environment. In this role, you will work on AWS using serverless and microservices architectures, utilizing services such as AWS Lambda, API Gateway, S3, DynamoDB, Kinesis, and other AWS tools as required. Collaboration with cross-functional teams is essential to deliver feature-rich applications that meet business requirements. You will apply software design principles and best practices to ensure applications are maintainable, modular, and highly testable. Your tasks will also involve setting up monitoring solutions to proactively monitor application performance, detect anomalies, and resolve issues. Optimizing data applications for cost, performance, and reliability on AWS is a crucial aspect of your role. To excel in this position, you should have at least 5 years of professional experience in data-focused application development using Python. Proficiency in core Python and data libraries such as Pandas, NumPy, and PySpark is required. You must possess a strong understanding of AWS services like ECS, Lambda, API Gateway, S3, DynamoDB, Kinesis, etc. Experience with building highly distributed and scalable solutions via serverless, micro-service, and service-oriented architecture is essential. Furthermore, you should be familiar with unit test frameworks, code quality tools, and CI/CD practices. Knowledge of database management, ORM concepts, and experience with both relational (PostgreSQL, MySQL) and NoSQL (DynamoDB) databases is desired. An understanding of the end-to-end software development lifecycle, Agile methodology, and AWS certification would be advantageous. Strong problem-solving abilities, attention to detail, critical thinking, and excellent communication skills are necessary for effective collaboration with technical and non-technical teams. Mentoring junior developers and contributing to a collaborative team environment are also part of your responsibilities. This is a full-time position located in Bangalore with a hybrid work schedule. If you have proficiency in Pandas, NumPy, and PySpark, along with 5 years of experience in Python, we encourage you to apply and join our team dedicated to developing, optimizing, and deploying scalable data applications supporting company growth and innovation.,

Posted 2 weeks ago

Apply

9.0 - 13.0 years

0 Lacs

hyderabad, telangana

On-site

You will be leading data engineering activities on moderate to complex data and analytics-centric problems that have a broad impact and require in-depth analysis to achieve desired results. Your responsibilities will include assembling, enhancing, maintaining, and optimizing current data, enabling cost savings, and meeting project or enterprise maturity objectives. Your role will require an advanced working knowledge of SQL, Python, and PySpark. You should also have experience using tools like Git/Bitbucket, Jenkins/CodeBuild, and CodePipeline, as well as familiarity with platform monitoring and alerts tools. Collaboration with Subject Matter Experts (SMEs) is essential for designing and developing Foundry front-end applications with the ontology (data model) and data pipelines supporting these applications. You will be responsible for implementing data transformations to derive new datasets or create Foundry Ontology Objects necessary for business applications. Additionally, you will implement operational applications using Foundry Tools such as Workshop, Map, and/or Slate. Active participation in agile/scrum ceremonies (stand-ups, planning, retrospectives, etc.) is expected from you. Documentation plays a crucial role in this role, and you will create and maintain documentation describing data catalog and data objects. As applications grow in usage and requirements change, you will be responsible for maintaining these applications. A continuous improvement mindset is encouraged, and you will be expected to engage in after-action reviews and share learnings. Strong communication skills, especially in explaining technical concepts to non-technical business leaders, will be essential for success in this role.,

Posted 2 weeks ago

Apply

2.0 - 6.0 years

0 Lacs

karnataka

On-site

The TTS Analytics team provides analytical insights to the Product, Pricing, Client Experience, and Sales functions within the global Treasury & Trade Services business. You will work on business problems focused on driving acquisitions, cross-sell, revenue growth & improvements in client experience. You will extract relevant insights, identify business opportunities, convert business problems into analytical frameworks, use big data tools and machine learning algorithms to build predictive models & other solutions, and design go-to-market strategies for a huge variety of business problems. The role of Spec Analytics Analyst 2 (C10) in the TTS Analytics team involves reporting to the AVP leading the team and working on multiple analyses throughout the year on business problems across the client life cycle acquisition, engagement, client experience, and retention for the TTS business. This will require leveraging multiple analytical approaches, tools, and techniques, working on multiple data sources (client profile & engagement data, transactions & revenue data, digital data, unstructured data like call transcripts, etc.) to provide data-driven insights to business and functional stakeholders. Qualifications: - Bachelors Degree with 3+ years of experience in data analytics or Masters Degree with 2+ years of experience in data analytics - Experience in identifying and resolving business problems in areas such as sales/marketing strategy optimization, pricing optimization, client experience, cross-sell, and retention preferably in the financial services industry - Proficiency in leveraging and developing analytical tools and methods to identify patterns, trends, and outliers in data - Ability to apply Predictive Modeling techniques for a wide range of business problems - Experience in working with data from different sources, with different complexities, both structured and unstructured - Ability to utilize text data to derive business value by leveraging different NLP techniques Skills: - Proficient in formulating analytical methodology, identifying trends and patterns with data - Ability to work hands-on to retrieve and manipulate data from big data environments - Proficiency in Python/R, PySpark and related tools, along with experience in Hive - Proficiency in MS Excel, PowerPoint, and experience with PySpark and Tableau (Good to have) - Strong analytical and problem-solving skills - Excellent communication and interpersonal skills - Organized, detail-oriented, and adaptive to a matrix work environment If you are a person with a disability and need a reasonable accommodation to use search tools and/or apply for a career opportunity, review Accessibility at Citi. View Citis EEO Policy Statement and the Know Your Rights poster.,

Posted 2 weeks ago

Apply

6.0 - 10.0 years

0 Lacs

thiruvananthapuram, kerala

On-site

You are a talented GCP Data Engineer with 6 to 10 years of experience, and you will be joining the UST team to take on a crucial role in designing and implementing efficient data solutions on the Google Cloud Platform (GCP). Your strong data engineering skills, expertise in GCP services, and proficiency in data processing technologies, particularly PySpark, will be essential for this role. Your responsibilities will include: Data Pipeline Development: - Designing, implementing, and optimizing end-to-end data pipelines on GCP with a focus on scalability and performance. - Developing and maintaining ETL workflows for seamless data processing. GCP Cloud Expertise: - Utilizing GCP services such as BigQuery, Cloud Storage, and Dataflow for effective data engineering. - Implementing and managing data storage solutions on GCP. Data Transformation with PySpark: - Leveraging PySpark for advanced data transformations to ensure high-quality and well-structured output. - Implementing data cleansing, enrichment, and validation processes using PySpark. Requirements: - Proven experience as a Data Engineer with a strong emphasis on GCP. - Proficiency in GCP services such as BigQuery, Cloud Storage, and Dataflow. - Expertise in PySpark for data processing and analytics is a must. - Experience with data modeling, ETL processes, and data warehousing. - Proficiency in programming languages such as Python, SQL, or Scala for data processing. - Relevant certifications in GCP or data engineering are a plus. Skills required: - GCP - PySpark,

Posted 2 weeks ago

Apply

4.0 - 8.0 years

0 Lacs

kochi, kerala

On-site

At EY, you'll have the chance to build a career as unique as you are, with the global scale, support, inclusive culture, and technology to become the best version of you. And we're counting on your unique voice and perspective to help EY become even better, too. Join us and build an exceptional experience for yourself, and a better working world for all. We are looking for a Senior Data Engineer to join the EY GDS Team. As part of our EY GDS TechOps team, you will be responsible for supporting, optimizing, and maintaining a modern end-to-end data architecture that spans Informatica CDI, Azure Data Factory, Azure Data Lake, and Databricks. This role involves providing technical leadership and application management expertise to global clients, ensuring seamless operation of data platforms, timely resolution of incidents, and implementation of enhancements aligned with business needs. You will collaborate with cross-functional teams and play a critical role in driving data reliability and value through best practices and innovation. To qualify for the role, you must have a Bachelor's degree in a relevant field (Computer Science, Engineering, Data Analytics, or related) or equivalent work experience. You should have 3-7 years of experience working across modern data ecosystems, with hands-on proficiency in Informatica CDI, Azure Data Factory (ADF), Azure Data Lake (ADLS), and Databricks. Experience providing application management support, strong root cause analysis skills, excellent stakeholder collaboration skills, a solid understanding of data governance, performance tuning, and cloud-based data architecture best practices are required. Experience in global delivery models and distributed teams is also necessary. Ideally, you'll also have experience integrating data from a variety of sources, familiarity with DevOps/CI-CD pipelines in a data engineering context, experience in industries such as retail, finance, or consumer goods, and relevant certifications. Your responsibilities will include providing daily Application Management Support for the full data stack, leading and coordinating the resolution of complex data integration and analytics issues, collaborating with technical and business stakeholders, and maintaining detailed documentation. Passionate, proactive problem solvers with a strong client orientation and professionals eager to learn and grow in a fast-paced, global delivery environment are what we look for. Working at EY offers support, coaching, and feedback, opportunities to develop new skills, progress your career, and the freedom and flexibility to handle your role in a way that's right for you. EY exists to build a better working world, helping to create long-term value for clients, people, and society and build trust in the capital markets. Enabled by data and technology, diverse EY teams in over 150 countries provide trust through assurance and help clients grow, transform, and operate. Working across assurance, consulting, law, strategy, tax, and transactions, EY teams ask better questions to find new answers for the complex issues facing our world today.,

Posted 2 weeks ago

Apply
cta

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.

Job Application AI Bot

Job Application AI Bot

Apply to 20+ Portals in one click

Download Now

Download the Mobile App

Instantly access job listings, apply easily, and track applications.

Featured Companies