Senior Data Engineer I

About LUMI:
LUMI is the company’s largest Big Data Platform, purpose-built for computationally and data-intensive processing applications. Whether data is processed in batch, online, or streaming modes, LUMI provides robust, scalable, and cost-efficient capabilities to handle enterprise workloads at scale.

As a hub of passionate Big Data engineers working with the most exciting and upcoming technologies, LUMI is the cornerstone platform where engineers are challenged every day to build world-class data products.

As we embark on our journey to the public cloud (GCP), you will play a senior engineering role in a fast-paced Agile team—designing, developing, testing, troubleshooting, and optimizing solutions that simplify and accelerate access to Amex’s Big Data Platform.

Focus Areas:

• Design, develop, and optimize enterprise-scale data pipelines and platforms.
• Provide technical leadership across design, debugging, performance optimization, and cloud migration.
• Lead efforts to integrate GenAI, Agentic AI, and data observability frameworks into platform capabilities.
• Build solutions that meet the needs of customer-facing applications, business platforms, and internal tools.

Organizational Context:

Member of an engineering delivery team reporting to an Engineering Manager or Director, with accountability for driving engineering excellence and mentoring junior engineers.

Responsibilities:

• Architect and implement enterprise-grade data migration solutions using Java and Python, enabling seamless data transfers from on-premises to GCP (Cloud Storage, BigQuery, Pub/Sub) using Apache Airflow and Google Cloud Composer.
• Build secure, scalable, and optimized data architectures leveraging GCP services such as Cloud Storage, Pub/Sub, Dataproc, Dataflow, and BigQuery.
• Design and implement automated frameworks for data delivery, monitoring, and troubleshooting.
• Develop data observability frameworks to ensure quality, lineage, and reliability across pipelines.
• Proactively monitor system performance, identify bottlenecks, and optimize pipelines for efficiency, scalability, and cost.
• Troubleshoot and resolve complex technical issues in distributed systems and cloud environments.
• Drive best practices in documentation of tools, architecture, processes, and solutions.
• Mentor junior engineers, conduct design/code reviews, and influence engineering standards.
• Collaborate with cross-functional teams to enable AI/ML and GenAI-driven use cases on LUMI.

Minimum Qualifications:

• 8+ years of experience in data engineering, software engineering, or platform development.
• Strong programming expertise in Java, Python, and Shell scripting.
• Advanced knowledge of SQL, data modeling, and performance optimization.
• Deep expertise in Google Cloud Platform services: Cloud Storage, BigQuery, Pub/Sub, Dataproc, Dataflow.
• Strong background in RDBMS (Oracle, Postgres, MySQL) and exposure to NoSQL DBs (Cassandra, MongoDB, or similar).
• Proven track record in CI/CD pipelines, Git workflows, and Agile development.
• Demonstrated experience in building and scaling production-grade data pipelines.
• Strong problem-solving and troubleshooting skills in distributed and cloud-native systems.

Preferred Qualifications:

• Hands-on experience with DevOps best practices, automation, and infrastructure as code.
• Exposure to platform engineering (networking, security, IAM, firewalls).
• Experience designing and implementing data observability frameworks (monitoring, lineage, anomaly detection).
• Hands-on or exposure to GenAI integrations (LLMs, RAG, AI-driven data engineering workflows).
• Proven ability to mentor, influence, and lead engineering discussions.

This is a senior technical role where you’ll help shape the future of LUMI, driving enterprise-scale data transformation on GCP while embedding GenAI and data observability into the next generation of data platforms.

• Competitive base salaries 
• Bonus incentives 
• Support for financial-well-being and retirement 
• Comprehensive medical, dental, vision, life insurance, and disability benefits (depending on location) 
• Flexible working model with hybrid, onsite or virtual arrangements depending on role and business need 
• Generous paid parental leave policies (depending on your location) 
• Free access to global on-site wellness centers staffed with nurses and doctors (depending on location) 
• Free and confidential counseling support through our Healthy Minds program 
• Career development and training opportunities