AWS Data Engineer with DataBricks

Job Description: AWS Data Engineer (Databricks Focus)

Position Overview: We are seeking a dynamic and innovative AWS Data Engineer with strong Databricks expertise to join our growing team. The ideal candidate will have a balanced mix of data engineering proficiency and application development skills, ready to own tasks, innovate solutions, and proactively drive technical advancements.

Key Responsibilities:

Design, develop, and optimize data pipelines using Databricks (PySpark) to manage and transform data efficiently.

Implement APIs leveraging AWS AppSync and Lambda functions to interact effectively with AWS Neptune and AWS OpenSearch.

Collaborate closely with front-end developers (React on AWS CloudFront) and cross-functional teams to enhance the overall application architecture.

Ensure adherence to best practices around AWS security services, particularly IAM, ensuring secure and efficient application development.

Proactively research, recommend, and implement innovative solutions to enhance performance, scalability, and reliability of data solutions.

Own assigned tasks and projects end-to-end, demonstrating autonomy and accountability.

Qualifications:

Proven 10year of experience in Databricks and PySpark for building scalable data pipelines.

Experience with AWS Neptune and AWS OpenSearch.

Solution-level understanding of AWS AppSync.

Solid understanding of AWS ecosystem services including Lambda, IAM, and CloudFront.

Strong capabilities in API development and integration within cloud architectures.

Experience or familiarity with React.js-based front-end applications is beneficial.

Excellent analytical, problem-solving, and debugging skills.

Ability to independently research and develop innovative solutions.

Strong communication skills with a proactive mindset and ability to collaborate effectively across teams.

Preferred Qualifications:

AWS Certification (Solutions Architect Associate, Developer Associate, Data Analytics Specialty) preferred.

• Experience with graph databases and search platforms in a production environment.