Big Data Engineer - ETL

As a Data Engineer, you will be responsible for building data pipelines, Big data processing solutions, and data lake infrastructure using various Big data and ETL technologies. You will assemble and process large, complex data sets to meet both functional and non-functional business requirements. ETL tasks will involve extracting data from a variety of sources such as MongoDB, S3, Server-to-Server, Kafka, and processing it using SQL and big data technologies. Additionally, you will develop analytical tools to offer actionable insights into customer acquisition, operational efficiency, and other key business performance metrics. Your role will also include constructing interactive and ad-hoc query self-serve tools for analytics use cases. Furthermore, you will be responsible for creating data models and data schema to ensure performance, scalability, and meet functional requirements. You will build processes that support data transformation, metadata, dependency, and workflow management. As part of your responsibilities, you will conduct research, experiments, and prototypes with new tools/technologies to ensure their successful implementation. **Must-Have Skills:** - Strong proficiency in Python/Scala - Experience with Big data technologies such as Spark, Hadoop, Athena/Presto, Redshift, Kafka, etc. - Proficiency in various file formats like parquet, JSON, Avro, orc, etc. - Familiarity with workflow management tools like Airflow - Experience in batch processing, streaming, and message queues - Knowledge of visualization tools such as Redash, Tableau, Kibana - Working experience with structured and unstructured data sets - Strong problem-solving skills **Good-to-Have Skills:** - Exposure to NoSQL databases like MongoDB - Familiarity with Cloud platforms like AWS, GCP, etc. - Understanding of Microservices architecture - Knowledge of Machine learning techniques This role is full-time and offers full benefits. The company is headquartered in the San Francisco Bay Area with its India headquarters located in Bengaluru.,