Aws Data Engineer

Location: Hyderabd

Experience Range: 5 to 10

AWS Databricks Developer

The role requires strong experience in developing reusable Spark frameworks and libraries using Scala, building JAR files, optimizing distributed workloads, and implementing enterprise-grade data pipelines on AWS.

Key Responsibilities

• Design and develop scalable data-processing applications using

  Apache Spark and Scala

  .
• Develop reusable frameworks, utilities, and libraries and package them as

  JAR files

  .
• Deploy and execute Scala-based JAR applications on AWS Databricks clusters.
• Build and maintain high-volume batch and streaming data pipelines.
• Develop ETL and ELT workflows using Databricks, Spark, Scala, and SQL.
• Integrate Databricks with AWS services such as S3, Glue, IAM, Secrets Manager, CloudWatch, and Redshift.
• Implement data ingestion and transformation solutions for structured, semi-structured, and unstructured data.
• Perform Spark performance tuning, including partitioning, caching, join optimization, and memory management.
• Develop and maintain Delta Lake tables and data-processing workflows.
• Configure Databricks jobs, workflows, clusters, libraries, and access controls.
• Implement error handling, logging, auditing, monitoring, and restart mechanisms.
• Write unit and integration tests for Scala and Spark applications.
• Build CI/CD pipelines to automate JAR creation, testing, and deployment.
• Troubleshoot data pipeline failures, cluster issues, and production defects.
• Participate in solution design, code reviews, release planning, and production support.
• Collaborate with data architects, analysts, DevOps teams, and business stakeholders.

Mandatory Technical Skills

Programming and Big Data

• Scala
• Apache Spark
• Spark SQL
• SQL
• Core Java concepts
• Object-oriented and functional programming
• Distributed data-processing concepts

Databricks

• AWS Databricks
• Databricks notebooks and jobs
• Databricks Workflows
• Cluster configuration and management
• Databricks Asset Bundles or equivalent deployment practices
• Delta Lake
• Unity Catalog
• Databricks libraries
• Spark performance optimization
• Job monitoring and troubleshooting

JAR Development

• Scala-based JAR development
• Building executable and reusable Spark libraries
• JAR packaging and dependency management
• Maven or SBT
• Creating assembly or fat JAR files
• Managing third-party dependencies
• Deploying JARs to Databricks clusters and jobs
• Versioning and release management

AWS Services

• Amazon S3
• AWS Glue
• AWS IAM
• Amazon CloudWatch
• AWS Secrets Manager
• Amazon Redshift
• Amazon EMR
• AWS Lambda
• Amazon SNS and SQS
• AWS Step Functions

DevOps and Development Tools

• Git
• Maven or SBT
• Jenkins, GitHub Actions, or AWS CodePipeline
• CI/CD pipeline development
• SonarQube
• JUnit, ScalaTest, or equivalent testing framework
• Jira and Agile development tools