Data Engineer

Develop Scalable Data Pipelines:

o Utilize Databricks and PySpark to design, develop, and maintain data processing pipelines.

o Implement ETL (Extract, Transform, Load) processes, ensuring efficient data extraction, transformation, and loading.

o Deep knowledge and hands-on of Databricks features and services - Feature Engineering, Lakeflow, Lakebase, Spark

declarative pipeline

2. Data Quality Implementation:

o Establish data quality checks within Azure Databricks.

o Ensure data accuracy, consistency, and adherence to business rules.

o Monitor data quality metrics and address anomalies promptly.

3. Source System Integration:

o Integrate Azure Databricks with various source systems (databases, data lakes, APIs).

o Efficiently ingest data from diverse sources into Azure Databricks.

o Handle schema evolution and changes in source data.

4. Pyspark Coding and Optimization:

o Write efficient PySpark code for data transformations, aggregations, and analytics.

o Optimize Spark jobs for performance and resource utilization.

o Troubleshoot and debug PySpark scripts as needed.

5. Collaboration and Communication:

o Work closely with data scientists, engineers, and other stakeholders.

o Gather requirements, understand business needs, and deliver effective solutions.

o Participate in cross-functional teams to ensure successful project outcomes.

6 Delta Lake Expertise:

o Understand and utilize Delta Lake, which provides ACID transactions and time travel capabilities on top of data lakes.

o Implement Delta Lake tables for reliable data storage and versioning.

7 Stored Procedure Conversion in Databricks:

• Convert existing stored procedures (e.g., from SQL Server) into Databricks-compatible code.

• Optimize and enhance stored procedures for better performance within Databricks

Education: Bachelor's degree in Computer Science, Information Technology, or related field.

• Experience:

o Minimum 3-4 years of experience working with Databricks and PySpark.

o Familiarity with big data technologies and distributed computing.

• Certifications (preferred):

o Microsoft Certified: Azure Data Engineer Associate or similar.

o Databricks Certified Associate Developer for Apache Spark.

• Soft Skills:

o Strong problem-solving abilities and quick troubleshooting skills.

o Excellent communication and collaboration skills.

o Adaptability to work in a dynamic, team-oriented environment.