Data Engineer

1 / 2

Position Summary:

We are seeking a skilled and detail-oriented Data Engineer to join our team and support live data migration projects. This role focuses on building, optimizing, and maintaining dynamic data pipelines that can adapt to various data structures from similar software platforms. The ideal candidate will ensure smooth data transformation and transportation processes while collaborating with cross-functional teams to deliver reliable, high-performance data solutions.

Key Responsibilities:

Data Pipeline Development & Optimization

· Design, develop, and maintain scalable and dynamic data pipelines for migrating and transforming large datasets.

· Ensure pipeline resiliency, reusability, and adaptability to handle multiple source systems and structures.

· Continuously monitor and optimize performance for low-latency, high-throughput data processing.

Data Architecture & Infrastructure

· Build and manage data architecture including data lakes, data warehouses, and relational/non-relational databases.

· Implement efficient storage strategies for structured and unstructured data across cloud/on-premises systems.

ETL/ELT & Integration

· Develop and manage ETL/ELT workflows integrating multiple sources (APIs, databases, third-party systems).

· Ensure data accuracy, consistency, and readiness for analytics and business intelligence.

Data Governance, Security & Compliance

· Implement data security best practices including encryption, access control, and privacy protocols.

· Ensure compliance with standards such as GDPR, HIPAA, and internal data policies.

Collaboration & Delivery

· Work closely with data scientists, analysts, DevOps, and software engineers to understand requirements and deliver robust data solutions.

· Support deployment and infrastructure management in cloud environments (AWS, Azure) or on-premises setups.

Documentation & Best Practices

· Maintain clear, up-to-date documentation of data workflows, architecture, and system configurations.

· Follow CI/CD practices, version control (Git), and agile methodologies for development and deployment.

2 / 2

Required Skills & Experience:

Technical Expertise

· Programming: Proficiency in Python and SQL.

· Big Data Frameworks: Experience with Apache Spark, Hadoop, Kafka, Flink, etc.

· Cloud Platforms: Hands-on with AWS, Azure, or equivalent.

· Data Tools: Experience with ETL tools (e.g., Airflow, Talend), data warehousing (e.g., Snowflake, Redshift), and both SQL/NoSQL databases.

Data Modeling

· Strong skills in data modeling, schema design, and managing metadata across systems.

DevOps & Automation

· Experience with CI/CD pipelines, automated deployment, and Git-based version control.

Soft Skills

· Excellent problem-solving abilities, attention to detail, and the ability to work in a fast-paced, collaborative environment.

Professional Qualifications:

· Education: B.Tech. (computer science)

· Experience: 3-5 years of relevant data engineering experience, preferably in data migration or transformation-heavy projects.

Preferred Attributes:

· Demonstrated experience in live data migration projects.

· Managed financial data migrations and worked with various accounting software platforms.

· Ability to create parameterized and reusable pipelines adaptable to multiple systems.

· Familiarity with real-time streaming and batch processing architectures.