Data Engineer

As a Data Engineer, your role is crucial for ensuring the smooth operation of the Data Platform in Azure/AWS Databricks. You will be responsible for the continuous development, enhancement, support, and maintenance of data availability, data quality, performance enhancement, and stability of the system. **Key Responsibilities:** - Designing and implementing data ingestion pipelines from various sources using Azure Databricks - Ensuring the efficient and smooth running of data pipelines and adhering to security, regulatory, and audit control guidelines - Driving optimization, continuous improvement, and efficiency in data processes **Qualifications Required:** - Minimum of 5 years of experience in the data analytics field - Hands-on experience with Azure/AWS Databricks - Proficiency in building and optimizing data pipelines, architectures, and data sets - Excellent skills in Scala or Python, PySpark, and SQL - Troubleshooting and optimizing complex queries on the Spark platform - Knowledge of structured and unstructured data design/modelling, data access, and data storage techniques - Expertise in designing and deploying data applications on cloud solutions such as Azure or AWS - Practical experience in performance tuning and optimizing code running in Databricks environment - Demonstrated analytical and problem-solving skills, particularly in a big data environment In terms of technical/professional skills, proficiency in Azure/AWS Databricks, Python/Scala/Spark/PySpark, HIVE/HBase/Impala/Parquet, Sqoop, Kafka, Flume, SQL, RDBMS, Airflow, Jenkins/Bamboo, Github/Bitbucket, and Nexus will be advantageous for executing the responsibilities effectively.,