28509 Pyspark Jobs - Page 3

Job description Hiring for Databrick Engineer-Female candidate Mandatory Skills: Databrick, Data lake, Azure Responsibilities A day in the life of an Infoscion As part of the Infosys consulting team, your primary role would be to actively aid the consulting team in different phases of the project including problem definition, effort estimation, diagnosis, solution generation and design and deployment You will explore the alternatives to the recommended solutions based on research that includes literature surveys, information available in public domains, vendor evaluation information, etc. and build POCs You will create requirement specifications from the business needs, define the to-be-proc...

You have over 7 years of experience in Big Data with a strong expertise in Spark and Scala. Your mandatory skills include proficiency in Big Data primarily Spark and Scala, along with a strong knowledge in HDFS, Hive, Impala, Unix, Oracle, and Autosys. It is essential to have strong communication skills and experience in Spark streaming, not limited to Spark batch. Additionally, you should have experience with NoSQL databases such as HBase, Mongo, and Couchbase. Your core skills should encompass PySpark, Python, and SQL. You should also have experience in machine learning with ML Libraries, Scikit-learn, PySpark Mlib, statistical methods, regression, classification, and clustering. In terms ...

As a highly skilled and innovative Senior Data Engineer, your role will involve architecting and building scalable data pipelines, enabling intelligent data processing, and driving AI-powered analytics across enterprise datasets. You will primarily work with Azure Databricks, PySpark, and Snowflake, integrating AI tools such as ChatGPT, Copilot, or Databricks AI Functions. **Key Responsibilities:** - Design and develop scalable data pipelines using Azure Databricks, PySpark, and Delta Lake. - Implement and optimize data warehousing solutions using Snowflake. - Integrate AI capabilities into data workflows using tools like ChatGPT, Copilot, Databricks AI Functions, or Snowflake Copilot. - Col...

As an Engineering Manager (Data Engineering) at Amgen, you will play a crucial role in driving the development and implementation of the data strategy with deep expertise in the R&D of Biotech or Pharma domain. Your responsibilities will include leading and mentoring a team of data engineers to build, optimize, and maintain scalable data architectures, data pipelines, and operational frameworks that support real-time analytics, AI-driven insights, and enterprise-wide data solutions. Your technical expertise, team leadership, and understanding of cloud data solutions will be essential in optimizing data-driven decision-making. **Roles & Responsibilities:** - Lead and mentor a team of data eng...

As a Deputy Manager, Business Analyst at our company based in Gurgaon, you will play a crucial role in utilizing analytics tools to conduct unstructured problem-solving and detailed analysis within the collections business. Your primary responsibilities will involve making recommendations to enhance business profitability and operational processes by leveraging your analysis and designing strategies to implement these recommendations effectively. Additionally, you will be tasked with effectively communicating findings to VP-level leaders globally and overseeing the implementation process with necessary support. Key Responsibilities: - Utilize expertise in Azure, Python (including Pandas), Py...

As an experienced AI/ML Architect, you will be responsible for architecting and deploying enterprise-scale AI/ML platforms. Your role will involve working in agile, cross-functional teams to deliver measurable impact. With a Masters or PhD in Computer Science, Artificial Intelligence, Machine Learning, or a related quantitative field, you should have 8+ years of experience in this domain. Your key responsibilities will include: - Designing, implementing, and scaling AI infrastructure using technologies such as Databricks, Azure AI, Snowflake, Spark, and Airflow. - Hands-on experience with model selection, fine-tuning, evaluation, and responsible AI governance. - Strong programming skills in ...

As a Data Engineer - Senior at EY's GDS Tax Technology practice in India, you will play a crucial role in developing, implementing, and integrating technology solutions to better serve clients and engagement teams. Your responsibilities will include: - Strong database programming/ backend development experience using SQL Server/Azure SQL, writing complex queries, stored procedures, views, triggers, cursors, UDFs. - Strong knowledge in data modeling and Datawarehouse concepts. - Experience on Azure Data Factory or any other ETL. - Developing, maintaining, and optimizing all data layer components for new and existing systems, including databases, stored procedures, ETL packages, and SQL querie...

**Job Description:** You will be responsible for developing and maintaining Python scripts and data pipelines using technologies such as dataflow, Data proc, GCP, Data Form, Big Query, TERRAFORM, and Data Fusion. Your experience with Agile Software Development will be valuable in this role. Additionally, having knowledge in Cloud SQL, GCP, and KAFKA would be beneficial. **Key Responsibilities:** - Develop and maintain Python scripts - Build and manage data pipelines using dataflow and Data proc - Utilize GCP and Data Form for data processing - Implement Agile Software Development practices - Work with technologies like Big Query, TERRAFORM, and Data Fusion - Optional: Experience with Cloud S...

Role Overview: As a Lead Programmer Analyst specializing in GCP Data Engineering, you will be responsible for utilizing your extensive experience and skills in data engineering to lead and drive projects related to Google Cloud Platform (GCP). Your primary focus will be on developing solutions using GCP services such as BigQuery, DataProc, Dataflow, and Dataform. Additionally, you will be required to have a strong command over SQL, Python, and Pyspark to effectively fulfill your responsibilities. Key Responsibilities: - Utilize your 5 to 7 years of data engineering experience to design and implement solutions using GCP services. - Develop and maintain data pipelines using tools like BigQuery...

As an FBS Associate Analytics Engineer at Capgemini, you will be part of a dynamic team dedicated to delivering cutting-edge analytics solutions for clients. Your role will involve working with different business teams to help with Data Engineering, Architecture, and Solutioning. You will be responsible for building complex data products and data assets to support the organization's D&A needs. Your strong technical skills in Python, SQL, PySpark, and cloud technologies will be crucial in solving business challenges related to data and data products. Key Responsibilities: - Collaborate with different business teams and IT to ensure the required data is available to support their needs - Becom...

Job Description: You have 4-6 years of experience in the analytics domain, focusing on developing and deploying ML/AI solutions. Additionally, you have at least 1 year of experience working on NLP projects, particularly text mining, using LLMs, working with Hugging Face/Transformer models, and utilizing publicly available LLM APIs. Your level of understanding in machine learning and deep learning methods is proficient. You possess excellent programming skills in Python, Pandas, PySpark, and SQL. Your communication skills are top-notch, allowing you to effectively describe findings to both technical and non-technical audiences. Furthermore, you have a good knowledge of the MS Office suite, es...

As a Data Engineer, your role will involve designing, developing, and maintaining scalable and efficient data pipelines using PySpark and Databricks. You will be responsible for performing data extraction, transformation, and loading (ETL) from diverse structured and unstructured data sources. Additionally, you will develop and maintain data models, data warehouses, and data marts in Databricks. Key responsibilities for this role include: - Proficiency in Python, Apache Spark, and PySpark - Integrating third party data from multiple sources with internal data - Writing and optimizing complex SQL queries for high performance and scalability across large datasets - Collaborating closely with d...

As a Lead Senior Software Engineer specializing in Python and React, you will leverage your 7+ years of industry experience to deliver enterprise products, with at least 3 years in a senior engineer role. Your responsibilities will include building and scaling platforms, APIs, or enterprise-grade SaaS products using Python and React. You will lead high-performing engineering teams of 5+ members, ensuring timely delivery of features with a focus on performance, scalability, and maintainability. Your logical mindset and problem-solving aptitude will guide the team towards effective resolutions. Additionally, you will actively participate in architectural discussions, facilitate healthy debates...

In this role, you will be responsible for performing AWS Glue ETL development using Python and PySpark for data transformation. Your proficiency in AWS services such as S3, Redshift, Athena, Lambda, and EMR will be crucial. You must have experience working with data formats like Parquet, Avro, JSON, CSV, and ORC. Advanced SQL skills are required for data querying and transformation. Additionally, you should possess a deep understanding of ETL/ELT design patterns, best practices, and data modeling concepts including dimensional modeling and star/snowflake schemas. Experience with version control using Git/Bitbucket is important for code management. Key Responsibilities: - Hands-on experience ...

Role Overview: At Citi, you will be part of a team that is shaping the future of banking by developing high-quality data products to support the Banks regulatory requirements and data-driven decision-making. As a Data Engineer, you will work closely with customers, set an example for the team, and contribute to business outcomes on an agile team by applying your knowledge of data architecture standards, data warehousing, data structures, and business intelligence. Key Responsibilities: - 5-8 years of relevant experience in Big Data platforms such as Hadoop, Hive, or Snowflake for data storage and processing - Proficiency in data modeling techniques, design, optimization, and maintenance of d...

As an MS Fabric Architect at our company, you will play a crucial role in leading the design and architecture of the MS fabric-based platform and related applications on the Medallion framework. With your expertise and experience, you will contribute to the successful delivery of projects by engaging with customer architects and business stakeholders, mentoring team members, and ensuring smooth deliveries. Additionally, you will have the opportunity to work with cutting-edge technologies and be part of a culture focused on talent development and inclusivity. **Key Responsibilities:** - Lead the design and architecture of the MS fabric-based platform and related applications on the Medallion ...

Job Summary: You are sought after for your deep understanding of the credit cards business, preferably for the US geography, with a focus on credit cards Marketing or Portfolio analytics. Your role demands strong business acumen, a grasp of credit cards P&L drivers, and customer lifecycle knowledge. You will be expected to take ownership of understanding business challenges, devising data-backed solutions, presenting them to senior business stakeholders, and overseeing their implementation. Your enthusiasm for learning in a dynamic environment and leadership capabilities to manage clients, partners, and inspire a team of analysts are crucial for this role. Key Responsibilities: - Identify op...

Looking for a Data Engineer (Python+ Linux Developer) with a strong background in Python, data engineering, data modeling, SQL Server, and SDLC *Must have : Hands-on experience on Linux with Python, PySpark, Airflow, MySql, Linux* Hybrid work

Sundial Overview Deep Comprehensible Insights from any Data Sundial is a top VC-backed early-stage startup headquartered at San Francisco Bay Area, US with a second office in Bengaluru, India. Our founders are industry veterans Chandra Narayanan, previously Chief Data Scientist at Sequoia Capital, and Julie Zhuo, previously VP, Design and Research at Facebook, author of a bestselling management book. We are a small team of top talent, high caliber Engineers, Data Scientists, Designers and PMs (currently 46 in India, 8 in US) and rapidly growing. We are on a mission to help builders make meaningful use of data to fulfil their vision. Sundial automatically diagnoses a product's data to explain...

Line of Service Advisory Industry/Sector Not Applicable Specialism SAP Management Level Manager Job Description & Summary A career within Application and Emerging Technology services, will provide you with a unique opportunity to help our clients identify and prioritise emerging technologies that can help solve their business problems. We help clients design approaches to integrate new technologies, skills, and processes so they can get the most out of their technology investment and drive business results and innovation. Why PWC At PwC, you will be part of a vibrant community of solvers that leads with trust and creates distinctive outcomes for our clients and communities. This purpose-led ...

ITIDATA an EXl Company Tasks Cypher or Gremlin query languages, Neo4J, Python, PySpark, Hive, Hadoop graph theory, Experience in creating and managing knowledge graphs using Neo4J. Requirements Looking for an exciting opportunity in graph databases ? We are hiring Neo4J Developers with 7-10 years of experience in data engineering , including 2-3 years of hands-on experience with Neo4J . Benefits 🔹 Key Skills & Responsibilities: ✅ Expertise in Cypher or Gremlin query languages ✅ Strong understanding of graph theory ✅ Experience in creating and managing knowledge graphs using Neo4J ✅ Optimizing performance and scalability of graph databases ✅ Researching & implementing new technology solutions...

Job Description We have an exciting and rewarding opportunity for you to take your software engineering career to the next level. Elevate your software engineering career with us, where you'll collaborate with a dynamic team to build impactful applications. We value your Python and Data Bricks expertise and provide a supportive environment for skill expansion. Join a culture that celebrates innovation, diversity, and continuous learning. Job Responsibilities Execute software solutions, design, development, and technical troubleshooting to build innovative applications Create secure, high-quality production code and maintain efficient algorithms Produce architecture and design artifacts for c...

Project Role : BI Engineer Project Role Description : Develop, migrate, deploy, and maintain data-driven insights and knowledge engine interfaces that drive adoption and decision making. Integrate security and data privacy protection. Must have skills : PySpark Good to have skills : NA Minimum 5 Year(s) Of Experience Is Required Educational Qualification : 15 years full time education Summary: As a BI Engineer, you will engage in the development, migration, deployment, and maintenance of data-driven insights and knowledge engine interfaces. Your typical day will involve collaborating with various teams to ensure that the solutions you create drive adoption and informed decision-making while ...

Project Role : Data Engineer Project Role Description : Design, develop and maintain data solutions for data generation, collection, and processing. Create data pipelines, ensure data quality, and implement ETL (extract, transform and load) processes to migrate and deploy data across systems. Must have skills : Databricks Unified Data Analytics Platform Good to have skills : Microsoft Azure Databricks, Microsoft Azure Analytics Services, PySpark Minimum 5 Year(s) Of Experience Is Required Educational Qualification : 15 years full time education Summary: As a Data Engineer, you will design, develop, and maintain data solutions that facilitate data generation, collection, and processing. Your ...

Project Role : Data Engineer Project Role Description : Design, develop and maintain data solutions for data generation, collection, and processing. Create data pipelines, ensure data quality, and implement ETL (extract, transform and load) processes to migrate and deploy data across systems. Must have skills : Databricks Unified Data Analytics Platform, Microsoft Azure Databricks, Microsoft Azure Data Services, PySpark Good to have skills : NA Minimum 7.5 Year(s) Of Experience Is Required Educational Qualification : 15 years full time education Summary: As a Data Engineer, you will design, develop, and maintain data solutions that facilitate data generation, collection, and processing. Your...