Data Engineer

Hi,

Role: : Data Engineer – Oil and Gas Domain

Location: Remote

Note: Minimum 2+ years in the Oil & Gas domain, preferably with upstream or midstream exposure

• Data Engineer with hands-on experience in building robust data pipelines and managing data solutions within the Oil and Gas domain. The ideal candidate will have a strong background in data engineering, data modelling, and cloud-based data platforms with a clear understanding of upstream/downstream data structures, exploration data, and production workflows.
• This role involves working closely with stakeholders, business analysts, and domain experts to build a reliable, scalable, and secure data infrastructure that supports analytics, reporting, and AI/ML solutions.

Key Responsibilities:

• Design, develop, and maintain data ingestion, transformation, and orchestration pipelines using technologies like PySpark, SQL, Databricks, and Airflow.
• Integrate data from various sources including SCADA, seismic systems, IoT sensors, drilling logs, and ERP systems.
• Perform data quality checks, data gap analysis, and resolve data inconsistencies across systems.
• Collaborate with stakeholders to manage master data mapping and align it with transactional data (e.g., well data, asset data).
• Work on data modeling for time-series data, production volumes, well logs, and reservoir data.
• Implement and optimize Delta Lake and data lake architectures on Azure or AWS platforms.
• Partner with data scientists and business teams to provision cleansed, validated, and transformed data for analytics use cases.
• Prioritize stakeholder requests using structured frameworks (e.g., MoSCoW, RICE) and ensure delivery based on business impact.
• Create visual prototypes or dashboards using Power BI/Tableau/Looker when specifications are vague and iterate based on feedback.
• Handle stakeholder conflicts with professionalism and use data-backed reasoning to arrive at resolutions.

Must-Have Skills:

• Strong programming skills in Python, SQL, and PySpark
• Proficiency with Databricks, Delta Lake, and data lakehouse concepts
• Experience working with cloud platforms: Azure Data Factory / AWS Glue / EMR
• Data modelling experience for Oil & Gas data types (wells, assets, production logs, seismic data)
• Familiarity with data quality frameworks and master data management (MDM)
• Version control (Git), CI/CD for data pipelines
• Excellent communication skills and stakeholder management

Good-to-Have Skills:

• Experience with Terraform or other Infrastructure-as-Code (IaC) tools
• Exposure to Palantir Foundry, especially for large-scale E&P data integration
• Knowledge of industry standards such as PPDM, WITSML
• Understanding of analytics, ML pipelines, or time-series forecasting
• Familiarity with data governance, access control, and compliance

Domain Experience:

• Minimum 2+ years in the Oil & Gas domain, preferably with upstream or midstream exposure
• Understanding of business processes like exploration & production (E&P), reservoir management, drilling operations, asset lifecycle, and real-time monitoring

Education:

• Bachelor’s or Master’s in Computer Science, Data Engineering, Petroleum Engineering, or a related field.