Data Engineer

As a Data Engineer with over 4 years of experience, you will be responsible for designing, developing, and maintaining scalable data pipelines to facilitate efficient data extraction, transformation, and loading (ETL) processes. Your role will involve architecting and implementing data storage solutions such as data warehouses, data lakes, and data marts that align with the organization's business needs. It will be crucial for you to implement robust data quality checks and cleansing techniques to ensure data accuracy and consistency. Your responsibilities will also include optimizing data pipelines for enhanced performance, scalability, and cost-effectiveness. Collaboration with data analysts and data scientists to understand data requirements and translate them into technical solutions will be a key aspect of your role. Additionally, you will be required to develop and maintain data security measures to ensure data privacy and regulatory compliance. Automation of data processing tasks using scripting languages like Python and Bash, as well as big data frameworks such as Spark and Hadoop, will be part of your daily tasks. Monitoring data pipelines and infrastructure for performance, as well as troubleshooting any issues that may arise, will be essential. Staying up to date with the latest trends and technologies in data engineering, including cloud platforms like AWS, Azure, and GCP, will also be expected. Documentation of data pipelines, processes, and data models for maintainability and knowledge sharing, as well as contribution to the overall data governance strategy and best practices, will be integral to your role. Qualifications for this position include a strong understanding of data architectures, data modeling principles, and ETL processes. Proficiency in SQL (e.g., MySQL, PostgreSQL) and experience with big data querying languages like Hive and Spark SQL are required. Experience with scripting languages for data manipulation and automation, familiarity with distributed data processing frameworks like Spark and Hadoop, and knowledge of cloud platforms for data storage and processing will be advantageous. Candidates should also possess experience with data quality tools and techniques, excellent problem-solving, analytical, and critical thinking skills, as well as strong communication, collaboration, and teamwork abilities. Key Skills: Spark, Hadoop, Python, Windows Azure, AWS, SQL, HiveQL,