Data Management Engineer

Softtek

0 years

0 Lacs

bengaluru karnataka india

Posted:1 day ago| Platform:

Apply

Skills Required

data management orchestration workflow apache airflow sensors ui logic etl relational scripting python sql writing linux monitoring datadog storage spark pyspark scala

Work Mode

On-site

Job Type

Full Time

Job Description

The Data Operations Engineer is the guardian of the data lifecycle. They ensure that data flows from source to destination without interruption, maintaining the "plumbing" of the organizations data infrastructure
1. Orchestration & Workflow Engines
Apache Airflow: Deep understanding of Operators, Sensors, Hooks, and the Airflow UI to manage complex task dependencies.
Workflow Logic: Knowledge of how to handle retries, branching, and ""backfilling"" historical data.
2. ETL/ELT Methodologies
Data Ingestion: Familiarity with moving data from APIs, logs, and relational databases into data warehouses (like Snowflake, BigQuery, or Redshift).
Transformation: Understanding how data is cleaned and restructured during transit.
3. Scripting & Command Line
Python: Ability to read and perform ""hotfixes"" or minor adjustments to existing scripts.
SQL: Proficiency in writing queries to validate data quality and troubleshoot ingestion errors.
Linux/Shell: Comfort navigating servers and checking system logs.
4. System Monitoring & Reliability
SLA/SLO Management: Understanding the business impact of data delays.
Observability: Using tools (like Grafana, Datadog, or Airflow logs) to monitor system health and data throughput
GCS (Google Cloud Storage) (Medium)
BigQuery required
Dataproc (Medium)
Spark (Pyspark/Scala) (Medium)
Airflow required
Python Medium

More Jobs at Softtek

Intune Admin

Bengaluru

2 - 4 yrs

INR 2 - 5 Lacs

Java Developer

Bengaluru

2 - 4 yrs

INR 4 - 8 Lacs

Multi-factor Authentication Leader

Bengaluru

4.0 - 6.0 yrs

INR 8 - 12 Lacs

Lead Service Desk Engineer

Bengaluru, Karnataka, India

Experience: Not specified

Salary: Not disclosed

Hybrid Tester

Bengaluru

4.0 - 6.0 yrs

INR 5 - 8 Lacs

Mock Interview

Practice Video Interview with JobPe AI

Start Python Interview

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.