3 Hdinsight Jobs

The job involves designing backend systems, stream processors, and data pipelines using SQL, Azure, and DevOps. Responsibilities include optimizing processes, delivering insights, and leading code reviews while collaborating on Azure solutions. Required Candidate profile CS Engineer with 5 years exp as Data Engineer. Proficient in Azure big data tools (Databricks, Synapse, HDInsight, ADLS) and cloud services (VM, Databricks, SQL DB).

We are seeking an experienced PySpark Developer / Data Engineer to design, develop, and optimize big data processing pipelines using Apache Spark and Python (PySpark). The ideal candidate should have expertise in distributed computing, ETL workflows, data lake architectures, and cloud-based big data solutions. Key Responsibilities: Develop and optimize ETL/ELT data pipelines using PySpark on distributed computing platforms (Hadoop, Databricks, EMR, HDInsight). Work with structured and unstructured data to perform data transformation, cleansing, and aggregation. Implement data lake and data warehouse solutions on AWS (S3, Glue, Redshift), Azure (ADLS, Synapse), or GCP (BigQuery, Dataflow). Optimize PySpark jobs for performance tuning, partitioning, and caching strategies. Design and implement real-time and batch data processing solutions. Integrate data pipelines with Kafka, Delta Lake, Iceberg, or Hudi for streaming and incremental updates. Ensure data security, governance, and compliance with industry best practices. Work with data scientists and analysts to prepare and process large-scale datasets for machine learning models. Collaborate with DevOps teams to deploy, monitor, and scale PySpark jobs using CI/CD pipelines, Kubernetes, and containerization. Perform unit testing and validation to ensure data integrity and reliability. Required Skills & Qualifications: 6+ years of experience in big data processing, ETL, and data engineering. Strong hands-on experience with PySpark (Apache Spark with Python). Expertise in SQL, DataFrame API, and RDD transformations. Experience with big data platforms (Hadoop, Hive, HDFS, Spark SQL). Knowledge of cloud data processing services (AWS Glue, EMR, Databricks, Azure Synapse, GCP Dataflow). Proficiency in writing optimized queries, partitioning, and indexing for performance tuning. Experience with workflow orchestration tools like Airflow, Oozie, or Prefect. Familiarity with containerization and deployment using Docker, Kubernetes, and CI/CD pipelines. Strong understanding of data governance, security, and compliance (GDPR, HIPAA, CCPA, etc.). Excellent problem-solving, debugging, and performance optimization skills.

Proven experience as a MS Fabric Data Architect Comfortable developing and implementing a delivery plan with key milestones based on requirements Strong working knowledge of Microsoft Fabric core features, including setup, configuration, and use of: Azure Data Lake (OneLake) for Big Data storage. Azure Synapse Data Warehouse for database management. Azure Synapse Data Engineering and Data Factory for data integration. Microsoft Purview (preview for Fabric) for data governance. Azure Data Science for analytics and AI workloads. Event stream and Data Activator for real time data flows. Strong understanding of data modelling, including relational and NoSQL data models. Ability to interpret an organisations information needs. Experience collaborating with Azure Cloud Architects to achieve platform goals. Proven experience designing Data architecture to support self serve analytics and AI development. Knowledge of dimensional modelling and Data Warehousing techniques. Expertise in Data partitioning, indexing, and optimisation strategies for large datasets Solution/technical architecture in the cloud Big Data/analytics/information analysis/database management in the cloud IoT/event driven/microservices in the cloud Experience with private and public cloud architectures, pros/cons, and migration considerations. Extensive hands on experience implementing data migration and data processing using Azure services:, Serverless Architecture, Azure Storage, Azure SQL DB/DW, Data Factory, Azure Stream Analytics, Azure Analysis Service, HDInsight, Databricks Azure Data Catalog, Cosmo Db, ML Studio, AI/ML, Azure Functions, ARM Templates, Azure DevOps, CI/CD etc. Cloud migration methodologies and processes including tools like Azure Data Factory, Event Hub, etc. Familiarity with the Technology stack available in the industry for data management, data ingestion, capture, processing and curation: Kafka, StreamSets, Attunity, GoldenGate, Map Reduce, Hadoop, Hive, Hbase, Cassandra, Spark, Flume, Hive, Impala, etc. Familiarity with Networking, Windows/Linux virtual machines, Container, Storage, ELB, AutoScaling is a plus Nice to Have Certifications AZ 303: Microsoft Azure Architect Technologies AZ 304: Microsoft Azure Architect Design DP 200 Implementing an Azure Data Solution DP 201 Designing an Azure Data Solution Nice to Have Skills/Qualifications: DevOps on an Azure platform Experience developing and deploying ETL solutions on Azure Strong in Power BI, C##, Spark, PySpark, Unix shell/Perl scripting Familiarity with the technology stack available in the industry for metadata management: Data Governance, Data Quality, MDM, Lineage, Data Catalog etc. Multi cloud experience a plus Azure, AWS, Google Professional Skill Requirements Proven ability to build, manage and foster a team oriented environment Proven ability to work creatively and analytically in a problem solving environment Desire to work in an information systems environment Excellent communication (written and oral) and interpersonal skills Excellent leadership and management skills Excellent organizational, multi tasking, and time management skills Proven ability to work independently