267 Parquet Jobs

Description: Understanding of data modeling concepts data warehousing tools and databases Experience in Snowflake SQL Cloud (AWS) Unix Technologies experience Must have hands on experience on SQL Snowflake Experience in AWS S3 and other common aws services. Should have experience working with multiple file formats especially with Parquet and Avro Should have Knowledge on Jenkins Git Udeploy CICD tools. Should have Experience on SQL writting and Unix commands Good communication skill and good team player Good to have Agile knowledge Named Job Posting? (if Yes - needs to be approved by SCSC) Additional Details Global Grade :B Level :To Be Defined Named Job Posting? (if Yes - needs to be approv...

JD -7+ years of hands on experience in Python especially dealing with Pandas and Numpy Good hands-on experience in Spark PySpark and Spark SQL Hands on experience in Databricks Unity Catalog Delta Lake Lake house Platform Medallion Architecture Azure Data Factory ADLS Experience in dealing with Parquet and JSON file format Knowledge in Snowflake.

You will be responsible for architecting and implementing scalable offline data pipelines for manufacturing systems, including AMHS, MES, SCADA, PLCs, vision systems, and sensor data. You will design and optimize ETL/ELT workflows using Python, Spark, SQL, and orchestration tools like Airflow to transform raw data into actionable insights. Leading database design and performance tuning across SQL and NoSQL systems will be part of your responsibilities, optimizing schema design, queries, and indexing strategies for manufacturing data. It will be essential for you to enforce robust data governance by implementing data quality checks, lineage tracking, access controls, security measures, and re...

Job Title: Senior Azure Data Engineer Experience: 912 Years Location: Noida Job Description We are looking for a Senior Azure Data Engineer with extensive experience in building and optimizing large-scale data pipelines and architectures using the Microsoft Azure ecosystem . The ideal candidate should have deep hands-on expertise in Azure Data Factory (ADF) , Azure Databricks , and Unity Catalog , along with a strong understanding of data governance, security, and performance optimization . The role involves designing scalable, reliable, and secure data platforms that enable data-driven decision-making across the enterprise. Key Responsibilities Design and implement end-to-end data pipelines...

Hiring: Data Analyst Location: Pan India (Hybrid) Experience: 6+ Years Notice Period: Immediate 30 Day Job Description We are seeking an experienced Data Analyst with a strong background in Databricks, SQL, and the Banking domain . The ideal candidate will have hands-on experience in developing and managing data pipelines , performing data analysis , and delivering insights to support business decisions. Key Responsibilities Develop, manage, and optimize data pipelines and data infrastructure within Azure. Perform data analysis to support reporting, MIS, and decision-making functions. Work on end-to-end Data Product deployment , from design to production. Collaborate with cross-functional te...

Meet Our Team: The Software Engineer works with a team of developers of varied skillsets (together known as Caterpillars scrum team) to design and develop the best of breed software offerings for the CRM industry. The D&AI team combine their industry knowledge and Pegas innovative technology to create products that will be used to solve business problems or inefficiencies for our customers. We are seeking an experienced technical professional who can demonstrate creativity, teamwork, a results-driven attitude, and a dedication to customer success. Picture Yourself at Pega: Role Overview We are seeking a highly skilled and motivated Software Engineer to join our Machine Learning & AI team. Th...

SUMMARY Key Responsibilities - Debug and resolve data ingestion and mapping issues for clients. - Understand current data mappings for file ingestions (FinTech files) for each client. - Work with ETL/ELT pipelines and messaging systems (Kafka or any custom ingestion services). - Validate and process data in common formats (CSV, JSON, Parquet). - Monitor and optimize data pipelines with Prometheus/Grafana alerts. - Collaborate with software engineers on schema evolution and data contracts. - Contribute automation scripts in Python/Shell to reduce repetitive tasks. - Participate in on-call for production data issues. RequirementsRequired Skills - Strong knowledge of SQL and PostgreSQL. - Exper...

About The Role Project Role : Data Engineer Project Role Description : Design, develop and maintain data solutions for data generation, collection, and processing. Create data pipelines, ensure data quality, and implement ETL (extract, transform and load) processes to migrate and deploy data across systems. Must have skills : PySpark Good to have skills : NA Minimum 5 year(s) of experience is required Educational Qualification : 15 years full time education Summary :As a Data Engineer, you will design, develop, and maintain data solutions that facilitate data generation, collection, and processing. Your typical day will involve creating data pipelines, ensuring data quality, and implementing...

We are hiring a Data Engineer with strong experience in AWS cloud services, Python, and data pipeline development. The ideal candidate will work on designing scalable data solutions, building ETL pipelines, and collaborating with cross-functional teams to deliver high-quality data products. Key Responsibilities: Design and develop data pipelines using AWS Glue, Lambda, Airflow, and S3. Build dimensional data models and dashboards using Power BI. Implement CI/CD pipelines for automated deployments. Work with APIs, JSON/XML data formats, and data governance practices. Collaborate with data scientists and business teams to deliver insights.

Key Responsibilities Architect and Develop Data Pipelines: Design and implement data pipelines that can handle the 4Vs of data (Volume, Variety, Velocity, Veracity) from diverse sources. Implement CDC: Utilize Change Data Capture (CDC) to efficiently and reliably synchronize data between source and target systems. Build a Medallion Architecture: Design and implement a data platform following the Medallion Architecture (Raw, Refined, Insights) to ensure data quality, manage transformations, and support analytical needs. Orchestrate and Manage Workflows: Develop and manage data orchestration processes to automate data ingestion, transformation, and delivery. Data Strategy & Business Value: Art...

You have a strong background in data engineering or a related field with over 8 years of experience. Your expertise includes hands-on experience with Azure Databricks, Spark, Python/Scala, CICD, and scripting for data processing. You have worked with multiple file formats such as Parquet, Delta, and Iceberg, and have knowledge of streaming technologies like Kafka for real-time data ingestion. Additionally, you are experienced in data governance and data security in Azure, and have a deep understanding of Azure Data Services. Key Responsibilities: - Utilize Azure Databricks, Spark, Python, and Scala for scalable and distributed data processing. - Work with various file formats like Parquet, D...

At JPMorgan Chase, you'll have the opportunity to advance your career and push the boundaries of data engineering by developing innovative, secure solutions that transform financial services. As a Data Engineer at JPMorgan Chase within the Employee Platforms team, you will become an integral part of an agile team dedicated to enhancing, building, and delivering trusted market-leading technology products in a secure, stable, and scalable way. Job Responsibilities Execute creative software and data solutions, including design, development, and technical troubleshooting, by thinking beyond routine or conventional approaches to build solutions or break down technical problems. Develop secure, hi...

About Us At SentinelOne, we're redefining cybersecurity by pushing the limits of what's possibleleveraging AI-powered, data-driven innovation to stay ahead of tomorrow's threats. From building industry-leading products to cultivating an exceptional company culture, our core values guide everything we do. We're looking for passionate individuals who thrive in collaborative environments and are eager to drive impact. If you're excited about solving complex challenges in bold, innovative ways, we'd love to connect with you. What Are We Looking For SentinelOne is seeking a Senior Software Engineer to join the Observo AI team in India, our cutting-edge AI-driven data pipeline optimization platfor...

Key Responsibilities Architect and Develop Data Pipelines: Design and implement data pipelines that can handle the 4Vs of data (Volume, Variety, Velocity, Veracity) from diverse sources. Implement CDC: Utilize Change Data Capture (CDC) to efficiently and reliably synchronize data between source and target systems. Build a Medallion Architecture: Design and implement a data platform following the Medallion Architecture (Raw, Refined, Insights) to ensure data quality, manage transformations, and support analytical needs. Orchestrate and Manage Workflows: Develop and manage data orchestration processes to automate data ingestion, transformation, and delivery. Data Strategy & Business Value: Art...

Computer science or similar education background. 7+ years total work experience. 5+ years hands on development experience in big data (Hadoop (cloudera preferred), spark, hive, pyspark, beeline, Scala, YARN, spark-submit, spark-java, parquet, ranger). Good understanding of big data storage concepts like partitioning, query and storage optimization, backup, restore and data replication. 5+ years experience in SDLC models (waterfall, and agile). 2+ years java coding experience. 2+ years experience in unix scripting and RDBMS Sql (e.g. sql server) Nice-to-have kills : Capital markets work experience Reference data (securities, trading books, client data, etc) Other systems involving large data...

Shift: 2:00 PM 11:00 PM IST Experience: 8+ years of hands-on Data Engineering experience About The Role We are looking for experienced Data Engineers who can design, build, and optimize large-scale data pipelines. This role is for individual contributors who love coding, problem-solving, and working with cutting-edge big data technologies. Key Responsibilities Design, develop, and maintain scalable data pipelines for ETL processes using Spark (PySpark/Scala). Optimize data flow and automate existing processes for improved performance. Collaborate with cross-functional teams Data Science, Analytics, and Product to support data infrastructure needs. Work on data quality, reliability, and perfo...

As a Data Engineer, your role will involve collaborating with the Lead Data Engineer and Data Architect to design and implement end-to-end data solutions. You will create, test, and implement enterprise-level apps with Snowflake, as well as design and implement features for identity and access management. Your responsibilities will also include creating authorization frameworks for better access control and implementing novel query optimization and major security competencies with encryption. Additionally, you will be solving performance issues and scalability issues in the system, managing transaction management with distributed data processing algorithms, and possessing ownership right fro...

Job Title Data Engg - Snowflake Role Summary & Role Description We are seeking a talented Senior Software Engineer to join our agile team. The successful candidate will play a crucial role in ensuring the quality and reliability of our software products, with a focus on developing Azure-based applications, data warehouse and ETL backend systems. Proficiency in Data Warehousing and Snowflake is essential for this role. Candidate must have minimum of 8 years of relevant experience in Data Warehousing and Database application development. Candidate must have hands on experience in Snowflake and Azure. Analyze, Design and Code snowflake objects and views. Create Database Views/materialized views...

Job Title Data Engg - Snowflake Role Summary & Role Description We are seeking a talented Senior Software Engineer to join our agile team. The successful candidate will play a crucial role in ensuring the quality and reliability of our software products, with a focus on developing Azure-based applications, data warehouse and ETL backend systems. Proficiency in Data Warehousing and Snowflake is essential for this role. Candidate must have minimum of 8 years of relevant experience in Data Warehousing and Database application development. Candidate must have hands on experience in Snowflake and Azure. Analyze, Design and Code snowflake objects and views. Create Database Views/materialized views...

Location : Bangalore, Chennai, Pune & Hyderabad Must have skills: ETL/ SQL with pyspark Experience and expertise in pipeline/ETL (Extract, Transform, Load) processes, API integration, scripting languages (Python), and big data technologies (Trio, Iceberg, DuckDB/Parquet). Database design, data modeling, and data warehousing. SQL and at least one cloud platform. Analytical tools (Superset, Power BI), statistical analysis, and SQL. To derive insights from data. Data governance principles and data quality. Data management, data integration projects using IPAAS tools like IBM Sterling, Mulesoft, Dell Boomi About Finacle Finacle is an industry leader in digital banking solutions. We partner with ...

where youll be supported and inspired by a collaborative community of colleagues around the world, where youll be supported and inspired bya collaborative community of colleagues around the world, and where youll be able to reimagine whats possible. Join us and help the worlds leading organizationsunlock the value of technology and build a more sustainable, more inclusive world. Your Role As a senior software engineer with Capgemini, you will have 3 + years of experience in Scala with strong project track record. Hands-on expertise in Scala and Spark, strong SQL skills on DB2, and proficiency in handling diverse file formats including JSON, Parquet, AVRO, ORC, and XML. Proven experience in H...

As an AWS Glue Developer with 6 to 9 years of experience, you will be responsible for the following: - Hands-on experience in AWS Glue ETL development for at least 3 years - Strong programming skills in Python and PySpark for data transformation - Proficiency in AWS services such as S3, Redshift, Athena, Lambda, and EMR - Experience working with data formats like Parquet, Avro, JSON, CSV, and ORC - Advanced SQL skills for data querying and transformation - Deep understanding of ETL/ELT design patterns and best practices - Knowledge of dimensional modeling, star/snowflake schemas - Experience with Git/Bitbucket for code management Preferred skills for this role include: - Experience with Vang...

Role Overview: You are an experienced Azure Databricks + SQL Developer / Big Data Engineer responsible for designing, developing, and maintaining scalable data solutions on Azure. Your primary focus will be on building efficient ETL/ELT pipelines, optimizing SQL queries, and leveraging Databricks and other Azure services for advanced data processing, analytics, and data platform engineering. Your strong background in traditional SQL development and modern big data technologies on Azure will be crucial for this role. Key Responsibilities: - Develop, maintain, and optimize ETL/ELT pipelines using Azure Databricks (PySpark/Spark SQL). - Write and optimize complex SQL queries, stored procedures,...

You will thoroughly analyze the existing MATLAB codebase, algorithms, data structures, and functionalities to gain a comprehensive understanding of their purpose, implementation details, and underlying mathematical/scientific principles. You will document the intricacies of the MATLAB tools, identifying core logic, dependencies, and potential challenges for migration. Collaborate with end-users and domain experts of the current MATLAB tools to capture implicit requirements, usage patterns, and critical performance characteristics. Develop detailed migration plans, including phased approaches, risk assessments, and resource estimates for converting MATLAB components to Python. Design the arch...

We have an opportunity to impact your career and provide an adventure where you can push the limits of what's possible. As a Lead Software Engineer at JPMorgan Chase within the Corporate & Investment Bank Technology team, you are an integral part of an agile team that works to enhance, build, and deliver trusted market-leading technology products in a secure, stable, and scalable way. As a core technical contributor, you are responsible for conducting critical technology solutions across multiple technical areas within various business functions in support of the firm's business objectives Job responsibilities Executes creative software solutions, design, development, and technical troublesh...