Data Engineer

As a Data Engineer at DXFactor, a US-based tech company with a global customer base, you will play a crucial role in designing, developing, and maintaining scalable data pipelines for both batch and streaming workflows. Your expertise in SnowFlake, AWS, and Python will be essential for implementing robust ETL/ELT processes to extract data from various sources and load them into data warehouses. Your responsibilities will also include optimizing database schemas, creating documentation for data flows, collaborating with cross-functional teams, monitoring data pipelines for performance, and implementing data quality checks and validation processes. Additionally, you will build and maintain CI/CD workflows for data engineering projects while staying updated with emerging technologies to suggest improvements to existing systems. Key Responsibilities: - Design, develop, and maintain scalable data pipelines for batch and streaming workflows - Implement robust ETL/ELT processes for extracting and loading data - Optimize database schemas and create documentation for data flows - Collaborate with cross-functional teams to translate business requirements into technical solutions - Monitor and troubleshoot data pipelines for optimal performance - Implement data quality checks and validation processes - Build and maintain CI/CD workflows for data engineering projects - Stay current with emerging technologies and recommend improvements to existing systems Qualifications Required: - Bachelor's degree in Computer Science, Information Technology, or related field - Minimum 4+ years of experience in data engineering roles - Strong proficiency in Python programming and SQL query writing - Hands-on experience with relational databases (e.g., PostgreSQL, MySQL) and NoSQL databases (e.g., MongoDB, Cassandra) - Experience with data warehousing technologies (e.g., Snowflake, Redshift, BigQuery) - Proven track record in building efficient and scalable data pipelines - Practical knowledge of batch and streaming data processing approaches - Experience implementing data validation, quality checks, and error handling mechanisms - Working experience with cloud platforms, particularly AWS (S3, EMR, Glue, Lambda, Redshift) and/or Azure (Data Factory, Databricks, HDInsight) - Understanding of different data architectures including data lakes, data warehouses, and data mesh - Demonstrated ability to debug complex data flows and optimize underperforming pipelines - Strong documentation skills and ability to communicate technical concepts effectively,