Field Data Engineer

About Prophecy

Prophecy is a rapidly growing startup enabling all the data users to visually build data pipelines with modern software practices including code on Github using its Low-Code Data Engineering Platform.

Prophecy is trusted by top Fortune 500 firms to replace their legacy ETL tools as they re-platform to the Cloud or Apache Spark. Were very well funded, backed by top VCs, and on the path to establishing ourselves as the leader on the cloud.

Prophecy is a Core Technology and Deep-IP company with engineering centered in India. Prophecy engineers often say that they have never worked in a more productive, higher-horsepower organization in their careers. The engineers love their work, are being challenged, and are doing the best work of their careers. To learn more, visit us on LinkedIn .

Position Summary

As a Field Data Engineer, you will be the technical advisor for enterprise customers building, running, and troubleshooting data pipelines on Prophecy s platform. You ll handle everything from pipeline failures and Spark debugging to CI/CD automation and architectural guidance enabling customers to deliver reliable data solutions at scale. This is a hybrid, individual-contributor role.

The Impact You Will Have

• Diagnose and resolve Spark pipeline failures : OOM/skew, schema mismatches, incorrect write modes, data validation errors, test/UDF issues.


• Troubleshoot runtime configurations,cluster policies, job parameters, library conflicts, fabric settings, and Prophecy-specific misconfigurations.


• Assist customers with UI-related issues, code-gen errors, schema transformations, visual gem stability, input/output validation.


• Resolve identity and authentication issues within pipelines ,tokens, scoped credentials, login errors, permissions.


• Fix data format and storage issues , Delta, Parquet, partitioning, checkpointing, atomic writes, schema drift.


• Provide performance tuning guidance , partitioning, caching, joins, shuffle footprint, Spark resource optimization.


• Build data quality and testing frameworks , unit tests, data diff checks, golden dataset validation, reconciliation strategies.


• Support CI/CD and release automation , CLI workflows, GitHub Actions, pipeline deployments.


• Enable governance and extensibility , Git workflows, code reviews, reusable templates, custom UDFs, SDK usage.


• Guide ETL migrations, advise on migrating from Informatica, Alteryx, BODS, and optimizing converted jobs.


• Serve as a trusted advisor, share best practices, architectural trade-offs, and cost / performance optimization.

What We Look For

• 4+ years in data engineering support, pipeline runtime troubleshooting, or enterprise data infrastructure roles.


• Deep understanding of Spark internals, job lifecycle, and failure modes.


• Strong familiarity with data warehouse runtimes (e.g., Snowflake), and SQL-based ETL errors.


• Hands-on experience with Prophecy platform or similar environments.


• Ability to diagnose UI and pipeline config issues rapidly.


• Proven ability to provide advisory support across performance tuning, data quality, and migration scenarios.


• Excellent written and verbal communication capable of articulating technical solutions to both technical and non-technical stakeholders.


• Technical degree in CS or equivalent practical experience.

What We d Love You to Have

• Exposure to CI/CD frameworks Jenkins, GitHub Actions, automated deployments.


• Background in testing frameworks for data pipelines (unit tests, golden datasets).


• Prior migration experience from ETL tools like Informatica or BODS to modern pipelines.


• Knowledge of DevOps practices in data engineering contexts.

What Youll Have At Prophecy

• Great company culture


• Competitive compensation


• Fair and Open Equity awards for everyone


• Private medical insurance


• Flexible hybrid work environment


• Learning and career development opportunities


• End-to-end project ownership and high-growth career path

Our Commitment to Diversity and Inclusion

At Prophecy, we hire for merit and foster an inclusive culture where people from diverse backgrounds can excel and do their best work. We take great care to ensure that our hiring practices are inclusive and meet equal employment opportunity standards. Individuals looking for employment at Prophecy are considered without regard to age, color, disability, ethnicity, family or marital status, gender identity or expression, language, national origin, physical and mental ability, political affiliation, race, religion, sexual orientation, socio-economic status, veteran status, and any other protected characteristics under applicable laws.