Data Engineer

Role Overview: As a Freelance Data Engineer at ThreatXIntel, your main responsibilities will include designing, developing, and maintaining ETL/ELT pipelines using Java and Pyspark. You will be tasked with building and optimizing data processing workflows on Databricks and Azure cloud platforms. Additionally, you will design SQL queries for data extraction, transformation, and analysis, and deploy containerized workloads using Kubernetes. Collaborating with teams to integrate data pipelines with Power BI dashboards and utilizing Git for version control and CI/CD best practices will be crucial aspects of your role. You will also be responsible for monitoring and troubleshooting production data pipelines for performance and reliability. Key Responsibilities: - Develop, maintain, and optimize ETL/ELT pipelines using Java and Pyspark. - Build and maintain data processing workflows on Databricks and Azure cloud platforms. - Design and maintain SQL queries for data extraction, transformation, and analysis. - Deploy and manage containerized workloads using Kubernetes. - Collaborate with teams to integrate data pipelines with Power BI dashboards for reporting. - Use Git for version control and CI/CD best practices in data engineering workflows. - Monitor and troubleshoot production data pipelines for performance and reliability. Qualification Required: - Strong experience in Java and Pyspark for data processing. - Hands-on expertise with SQL and relational databases. - Experience with Databricks and Azure cloud services. - Experience with ETL development and orchestration of data workflows. - Familiarity with Kubernetes and containerized deployments. - Proficiency with Power BI for data visualization. - Experience using Git for version control and collaborative development. Additional Company Details: ThreatXIntel is a company that values expertise in data engineering and provides opportunities to work with cutting-edge technologies such as Databricks, Azure cloud platforms, and Power BI. The company encourages collaboration among teams and emphasizes the importance of monitoring and optimizing data pipelines for peak performance and reliability. Joining ThreatXIntel means being part of a dynamic environment where your skills in data engineering will be put to the test, and where continuous learning and growth are supported.,