Data Engineer 2

As a Data Engineer at our company, you will be an integral part of a skilled Data Engineering team focused on developing reusable capabilities and tools to automate various data processing pipelines. Your responsibilities will include contributing to data acquisition, ingestion, processing, monitoring pipelines, and validating data. Your role is pivotal in maintaining the smooth operation of data ingestion and processing pipelines, ensuring that data in the data lake is up-to-date, valid, and usable at all times. With a minimum of 3 years of experience in data engineering, you should be proficient in Python programming and have a strong background in working with both RDBMS and NoSQL systems. Experience in the AWS ecosystem, including components like Airflow, EMR, Redshift, S3, Athena, and PySpark, is essential. Additionally, you should have expertise in developing REST APIs using Python frameworks such as flask and fastapi. Familiarity with crawling libraries like BeautifulSoup in Python would be advantageous. Your skill in writing complex SQL queries to retrieve key metrics and working with various data lake storage formats will be key to your success in this role. Key Responsibilities: - Design and implement scalable data pipelines capable of handling large data volumes. - Develop ETL/ELT pipelines to extract data from upstream sources and synchronize it with data lakes in formats like parquet, iceberg, and delta. - Optimize and maintain data pipelines to ensure smooth operation and business continuity. - Collaborate with cross-functional teams to source data for various business use cases. - Stay informed about emerging data technologies and trends to enhance our data infrastructure and architecture continuously. - Adhere to best practices in data querying and manipulation to uphold data integrity. If you are a motivated Data Engineer with a passion for building robust data pipelines and ensuring data quality, we invite you to join our dynamic team and contribute to the success of our data engineering initiatives.,