Data Engineer - Databricks

As a Data Engineer at AuxoAI, you will be an integral part of our dynamic team, utilizing your 7-10 years of experience in data engineering to collaborate with cross-functional teams. Your primary role will involve designing, building, and optimizing data pipelines and infrastructure using Databricks. **Key Responsibilities:** - Designing, building, and operating scalable on-premises or cloud data architecture. - Analyzing business requirements and translating them into technical specifications. - Designing, developing, and implementing data engineering solutions using Databricks on cloud platforms (AWS, Azure). - Developing and maintaining scalable data pipelines and ETL processes. - Collaborating with data scientists and analysts to implement solutions that support analytics and machine learning initiatives. - Optimizing data storage and retrieval mechanisms for performance, reliability, and cost-effectiveness. - Implementing data governance and security best practices to ensure compliance and data integrity. - Troubleshooting and debugging data pipeline issues, providing timely resolution and proactive monitoring. - Staying updated on emerging technologies and industry trends, recommending innovative solutions to enhance data engineering capabilities. **Qualifications:** - Bachelor's or Master's degree in Computer Science, Engineering, or a related field. - 3-6 years of prior experience in data engineering, focusing on designing and building data pipelines. - 3+ years of experience in Databricks and Spark with at least two end-to-end data engineering implementations. - Comprehensive understanding of Databricks products and ecosystem. - Strong programming skills in Python, Scala, or SQL. - Experience with data modeling, ETL processes, and data warehousing concepts. - Familiarity with containerization technologies and orchestration tools is a plus.,