Data Engineer II (Databricks)

Immediate to 30 days notice period candidates preferred

Key Responsibilities:

• Design and implement general architecture for complex data systems.
• Translate business requirements into functional and technical specifications.
• Design and implement lakehouse architecture.
• Develop and manage cloud-based data architecture and reporting solutions.
• Apply data modelling principles for relational and dimensional data structures.
• Design Data Warehouses following established principles (e.g., Kimball, Inmon).
• Create and manage source-to-target mappings for ETL/ELT processes.
• Mentor junior engineers and contribute to architectural decisions and code reviews.

Minimum Qualifications:

• Bachelors Degree in Computer Science, Computer Engineering, MIS, or related field.
• 5+ years of experience with Microsoft SQL Server and strong proficiency in T- SQL, SQL performance tuning (Indexing, Structure, Query Optimization).
• 5+ years of experience in Microsoft data platform development and implementation.
• 5+ years of experience with Power BI or other competitive technologies.
• 3+ years of experience in consulting, with a focus on analytics and data solutions.
• 2+ years of experience with Databricks, including Unity Catalog, Databricks SQL, Workflows, and Delta Sharing.
• Proficiency in Python and Apache Spark.
• Develop and manage Databricks notebooks for data transformation, exploration, and model deployment.
• Expertise in Microsoft Azure services, including Azure SQL, Azure Data Factory (ADF), Azure Data Warehouse (Synapse Analytics), Azure Data Lake, and Stream Analytics

Preferred Qualifications:

• Experience with Microsoft Fabric.
• Familiarity with CI/CD pipelines and infrastructure-as-code tools like Terraform or Azure Resource Manager (ARM).
• Knowledge of taxonomies, metadata management, and master data management.
• Familiarity with data stewardship, ownership, and data quality management.
• Expertise in Big Data technologies and tools:

o Big Data Platforms: HDFS, MapReduce, Pig, Hive.

o General DBMS experience with Oracle, DB2, MySQL, etc.

o NoSQL databases such as HBase, Cassandra, DataStax, MongoDB, CouchDB, etc.

o Experience with non-Microsoft reporting and BI tools, such as Qlik, Cognos,

MicroStrategy, Tableau, etc.