Data Engineer

Data Engineer

Azure Data Factory, SQL, Python, Databricks, ETL

Data Engineer

.Net + Azure Data Factory

Data Engineer

Key Responsibilities

• Design, build, and maintain data pipelines to collect, process, and store data efficiently from various sources.

• Work with cross-functional teams to understand business requirements and deliver data solutions that meet the needs of the organization.

• Optimize data storage and retrieval methods to enhance system performance and data quality.

• Integrate data from multiple sources (databases, data lakes, cloud storage, etc.) to build comprehensive data sets.

• Build and maintain data infrastructure, ensuring it is scalable, reliable, and secure.

• Implement and manage data governance policies, ensuring data accuracy, consistency, and compliance.

• Conduct data modeling and provide insights into the data through advanced analytics tools and reports.

• Perform data transformations and data wrangling tasks to prepare data for analysis.

• Troubleshoot data issues and collaborate with stakeholders to resolve technical challenges.

• Ensure the integrity of data pipelines and systems by conducting routine testing and validation.

Required Skills and Qualifications

• Bachelors degree in Computer Science, Engineering, Information Systems, or related field.

• Proven experience as a Data Engineer or in a similar role, with a minimum of [X] years of experience.

• SQL

• Experience working with cloud platforms (AWS, Azure, GCP) and data storage solutions (e.g., Redshift, Snowflake, BigQuery).

• Solid understanding of data modeling, database design, and cloud data architectures.

• Hands-on experience with data warehousing concepts and tools (e.g., Apache Hive, Apache Spark).

• Apache Airflow

• Knowledge of real-time data streaming technologies (e.g., Kafka, Kinesis) is a plus.

• Strong problem-solving skills and ability to troubleshoot complex data issues.

• Familiarity with data security best practices and data privacy regulations (e.g., GDPR, HIPAA).