Jobs
Interviews
Sirius AI logo

Manager - Data Lake and Data Architecture

6 years

0 Lacs

gurugram haryana india

Posted:2 weeks ago| Platform: Linkedin logo

Apply

Skills Required

data architecture management design pipeline analytics reporting aws azure gcp dataflow apache spark integration etl extract consistency reliability engineering kubernetes processing hive governance security compliance controls encryption monitoring devops automation code development deployment support testing query indexing strategies efficiency resolve collaboration documentation storage hadoop kafka sql nosql postgresql mysql containerization orchestration docker terraform visualization power tableau looker troubleshooting certifications

Work Mode

On-site

Job Type

Full Time

Job Description

Key Responsibilities:

  • Data Architecture & Management:

    Design and implement scalable, cloud-agnostic Data Lake, Data LakeHouse, Data Mesh and Data Fabric architectures to efficiently store, process, and manage structured and unstructured data from various sources.

  • Data Pipeline Development

    : Design, develop, and maintain robust data pipelines to ingest, process, and transform data from multiple sources into usable formats for analytics and reporting using services like AWS Glue, Azure Data Factory, GCP Dataflow, Apache Spark, or Apache Airflow.

  • Data Integration and ETL:

    Develop and optimize Extract, Transform, Load (ETL) and ELT processes to integrate disparate data sources into the data lake, ensuring high data quality, consistency, and reliability across multiple cloud platforms.

  • Cloud-Agnostic Data Engineering:

    Develop data solutions that are cloud-agnostic, leveraging open-source technologies like Apache Spark, Delta Lake, Presto, and Kubernetes, ensuring compatibility across AWS, Azure, and GCP.

  • Big Data Processing & Analytics:

    Utilize big data technologies such as Apache Spark, Hive, and Presto for distributed computing, enabling large-scale data transformations and analytics.

  • Data Governance and Security

    : Implement robust data governance policies, security frameworks, and compliance controls, including role-based access control (RBAC), encryption, and monitoring to meet industry standards (GDPR, HIPAA, PCI-DSS).

  • DevOps Integration for Data Platforms:

    Leverage cloud-agnostic DevOps tools and practices for source control, build automation, release management, and Infrastructure as Code (IaC) to streamline the development, deployment, and management of data lake and data architecture solutions across multiple cloud providers. Solutions should support CI/CD pipelines, automated testing, and scalable data workflows.

  • Continuous Integration and Deployment (CI/CD

    ): Establish automated CI/CD pipelines to streamline deployment, testing, and monitoring of data infrastructure and workflows.

  • Performance Optimization

    : Optimize data workflows and query performance using indexing, caching, and partitioning strategies to improve efficiency and cost-effectiveness.

  • Monitoring and Troubleshooting

    : Implement observability solutions using tools like Prometheus, Grafana, or cloud-native monitoring services to proactively detect and resolve data pipeline issues.

  • Collaboration and Documentation:

    Work with cross-functional teams, including data scientists, analysts, and business stakeholders, to design and implement scalable data solutions. Maintain comprehensive documentation of data architectures, processes, and best practices.


Job Qualifications:

  • Bachelor's degree in Computer Science, Engineering, or a related field.
  • 6+ years of experience as a Data Engineer, specializing in cloud-agnostic data solutions and data lake architectures.
  • Strong expertise in cloud data platforms such as AWS, Azure, and Google Cloud, with hands-on experience in services like AWS S3, Azure Data Lake, Google Cloud Storage, and related data processing tools.
  • Proficiency in big data technologies such as Apache Spark, Hadoop, Kafka, Delta Lake, or Presto.
  • Experience with SQL and NoSQL databases, including PostgreSQL, MySQL, and DynamoDB.
  • Expertise in containerization and orchestration platforms such as Docker and Kubernetes.
  • Experience implementing DevOps and CI/CD practices using Terraform, CloudFormation, or other Infrastructure as Code (IaC) tools.
  • Knowledge of data visualization tools such as Power BI, Tableau, or Looker for presenting insights and reports.
  • Strong problem-solving and troubleshooting skills with a proactive approach to identifying and resolving issues.
  • Experience leading teams of 5+ cloud engineers.
  • Preferred certifications in AWS, Azure, or Google Cloud data engineering.

More Jobs at Sirius AI

Exciting Internship Opportunities at SiriusAI – Join Us in Shaping the Future

Gurugram, Haryana, India

Experience: Not specified

|

Salary: Not disclosed

Executive Assitant

Gurugram, Haryana, India

6.0 - 6.0 yrs
|

Salary: Not disclosed

Presales Consultant

gurugram, haryana, india

1.0 - 3.0 yrs
|

Salary: Not disclosed

Manager - Data Science

gurugram, haryana, india

4.0 - 7.0 yrs
|

Salary: Not disclosed

Sr.Consultant - Data Science

gurugram, haryana, india

2.0 - 5.0 yrs
|

Salary: Not disclosed

Mock Interview

Practice Video Interview with JobPe AI

Start DevOps Interview
cta

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.

Job Application AI Bot

Job Application AI Bot

Apply to 20+ Portals in one click

Download Now

Download the Mobile App

Instantly access job listings, apply easily, and track applications.

Google Play App Store
coding practice

Enhance Your Skills

Practice coding challenges to boost your skills

Start Practicing Now
Sirius AI logo
Sirius AI

RecommendedJobs for You

Sirius AI logo

Manager - Data Lake and Data Architecture

Sirius AI

gurugram, haryana, india

Sirius AI logo

Manager - Data Lake and Data Architecture

Sirius AI

gurugram, haryana, india