Job
Description
We are preferring employees from Product organisation and premium Engineering Institutes. We are hiring for our client, who is an Indian multinational technology services company based in Pune. It is primarily engaged in cloud computing, internet of things, endpoint security, big data analytics and software product engineering services. Data Platform Engineer: Assisting team members in designing and building data infrastructure at scale. We handle PB of data each day through streaming and batch processing. You will be helping to deliver data to our data lake for use by our Data Warehouse team, Analytics teams and Data Scientists. Work on Data Lakehouse system architecture and ingesting/pipelining of data, and tools to automateand orchestrate in delivering performance, reliability, and operational efficiency • Define both batch and streaming data-parallel processing pipelines and distributed processing back-ends Build CI/CD Pipelines and manage configuration management Build tools and services that run on k8s that are part of our data ecosystem Routinely write efficient, legible, and well-commented Python Clear communication skills to deliver on complex, technical topics Help scale our data warehouse (we use Snowflake) for clean data-ready delivery for analysis • Work closely with Analytic Engineers and Data Analysts for the collection/analysis of raw data for models that empower end users. Build and scale our warehouse platform for data ingest, logging, search, aggregation, viewing, and analysis What we're looking for: 5+ years of professional experience working and developing using Python and/or Java • 3+ years of professional experience working with Python scripting experience (Unix, bash, python) AWS Certification or equivalent experience Terraform or other IaC Tools (Terraform Preferred) Experience with Streaming Data Apache Beam, Fink, Spark, and Kafka Experience with modern data technologies such as Airflow, Snowflake, Redshift, Spark • Knowledge of source control, gitflow, gitlabflow, CI/CD (gitlab, circleci) Knowledge/Experience working with Kubernetes, Docker, Helm Experience with automation and orchestration tools Bachelors degree or equivalent in computer science, information systems, or a combination of education and related experience required Data Engineer: What You'll Do Lead the building of scalable, fault tolerant pipelines with built in data quality checks that transform, load and curate data from various internal and external systems Provide leadership to cross-functional initiatives and projects. Influence architecture design and decisions. Build cross-functional relationships with Data Scientists, Product Managers and Software Engineers to understand data needs and deliver on those needs. Improve engineering processes and cross-team collaboration. Ruthlessly prioritize work to align with company priorities. Provide thought leadership to grow and evolve DE function and implementation of SDLC best practices in building internal-facing data products by staying up-to-date with • industry trends, emerging technologies, and best practices in data engineering What we're looking for: 3-12 years of experience in BI and Data Warehousing. Minimum 3 years of experience leading data teams in a high-volume environment • Minimum 4 years of experience with dbt, Airflow and snowflake Experience with Apache Iceberg tables Experience and knowledge of building data-lakes in AWS (i.e. Spark/Glue, Athena), • Including data modeling, data quality best practices, and self-service tooling. • Experience mentoring data professionals from junior to senior levels Demonstrated success leading cross functional initiatives Passionate about data quality, code quality, SLAs and continuous improvement • Deep understanding of data system architecture Deep understanding of ETL/ELT patterns Development experience in at least one object-oriented language (Python,R,Scala, etc.). • Comfortable with SQL and related tooling
