Senior Data Engineer (Databricks) Remote (Mexico)
As a Senior Data Engineer specializing in Databricks, you will play a critical role within our remote consulting team based in Mexico. Your primary responsibility will be to conceptualize, develop, and enhance large-scale data processing systems utilizing Databricks and cutting-edge data engineering technologies. Your collaborative efforts with data scientists, analysts, and technical teams will be pivotal in delivering robust and scalable data platforms. Your duties will encompass the design, construction, and maintenance of resilient data pipelines tailored for processing both structured and unstructured data. Moreover, you will be tasked with the creation and administration of data lakes and data warehouses optimized for efficient analytics. Your expertise will be crucial in streamlining data workflows to ensure optimal performance, scalability, and cost-effectiveness. In close collaboration with stakeholders, you will elicit data requirements and transform them into scalable solutions that align with the organization's objectives. Implementation of data governance, data quality standards, and security protocols will be essential components of your role. Furthermore, you will be involved in the migration of legacy data processes, such as those from SAS, to contemporary platforms. Key Responsibilities: - Design, develop, and maintain robust data pipelines for processing structured and unstructured data. - Build and manage data lakes and data warehouses optimized for analytics. - Optimize data workflows for enhanced performance, scalability, and cost-efficiency. - Collaborate with stakeholders to gather data requirements and translate them into scalable solutions. - Implement data governance, data quality, and security best practices. - Migrate legacy data processes to modern platforms. - Document architecture, data models, and pipelines. Required Qualifications: - Minimum of 5 years of experience in data engineering or related fields. - At least 3 years of hands-on experience with Databricks. - Proficiency in SQL and hands-on experience with PostgreSQL, MySQL, or NoSQL databases. - Programming skills in Python, Java, or Scala. - Experience with ETL processes, orchestration frameworks, and data pipeline automation. - Familiarity with Spark, Kafka, or similar big data tools. - Experience working on cloud platforms such as AWS, Azure, or GCP. - Prior experience in migrating from SAS would be advantageous. - Excellent communication skills in English. - Must be based in Mexico and eligible to work as a contractor.,
