Data Engineer

JOB DETAILS:

job title

• Data Pipeline Development: Design, develop, test, and deploy robust, high-performance, and scalable ETL/ELT data pipelines using Scala and Apache Spark to ingest, process, and transform large volumes of structured and unstructured data from diverse sources.
• Big Data Expertise: Leverage expertise in the Hadoop ecosystem (HDFS, Hive, etc.) and distributed computing principles to build efficient and fault-tolerant data solutions.
• Advanced SQL: Write complex, optimized SQL queries and stored procedures.
• Performance Optimization: Continuously monitor, analyze, and optimize the performance of data pipelines and data stores. Troubleshoot complex data-related issues, identify bottlenecks, and implement solutions for improved efficiency and reliability.
• Data Quality & Governance: Implement data quality checks, validation rules, and reconciliation processes to ensure the accuracy, completeness, and consistency of data. Contribute to data governance and security best practices.
• Automation & CI/CD: Implement automation for data pipeline deployment, monitoring, and alerting using tools like Apache Airflow, Jenkins, or similar CI/CD platforms.
• Documentation: Create and maintain comprehensive technical documentation for data architectures, pipelines, and processes.

Required Skills & Qualifications:

• Bachelor's or Master's degree in Computer Science, Engineering, or a related quantitative field.
• Minimum 5 years of professional experience in Data Engineering, with a strong focus on big data technologies.
• Proficiency in Scala for developing big data applications and transformations, especially with Apache Spark.
• Expert-level proficiency in SQL; ability to write complex queries, optimize performance, and understand database internals.
• Extensive hands-on experience with Apache Spark (Spark SQL, DataFrames, RDDs) for large-scale data processing and analytics.
• Solid understanding of distributed computing concepts and experience with the Hadoop ecosystem (HDFS, Hive).
• Experience with building and optimizing ETL/ELT processes and data warehousing concepts.
• Strong understanding of data modeling techniques (e.g., Star Schema, Snowflake Schema).
• Familiarity with version control systems (e.g., Git).
• Excellent problem-solving, analytical, and communication skills.
• Ability to work independently and collaboratively in an Agile team environment.