Data Engineer

As a Data Engineer at Tata Electronics Private Limited (TEPL), you will play a crucial role in architecting and implementing scalable offline data pipelines for manufacturing systems, ensuring the smooth functioning of various components such as AMHS, MES, SCADA, PLCs, vision systems, and sensor data. Your expertise in designing and optimizing ETL/ELT workflows using tools like Python, Spark, SQL, and Airflow will be instrumental in transforming raw data into actionable insights. You will lead database design and performance tuning activities across SQL and NoSQL systems, optimizing schema design, queries, and indexing strategies to enhance the efficiency of manufacturing data processing. Your responsibilities will also include enforcing robust data governance through the implementation of data quality checks, lineage tracking, access controls, security measures, and retention policies. In addition, you will be involved in optimizing storage and processing efficiency by strategically utilizing formats like Parquet, ORC, compression techniques, partitioning, and indexing for high-performance analytics. Implementing streaming data solutions using Kafka or RabbitMQ to handle real-time data flows and synchronization across control systems will be a key aspect of your role. Collaborating cross-functionally with Platform Engineers, Data Scientists, Automation teams, IT Operations, Manufacturing, and Quality departments will be essential to ensure consistency across manufacturing systems and enable data consumption by downstream applications. You will also mentor junior engineers, establish best practices, documentation standards, and foster a data-driven culture within the organization. Your essential attributes should include expertise in Python programming for building robust ETL/ELT pipelines, proficiency with the Hadoop ecosystem, hands-on experience with Apache Spark (PySpark), strong SQL skills, and proficiency in using Apache Airflow for orchestrating complex data workflows. Experience with real-time data streaming using Kafka or RabbitMQ, both SQL and NoSQL databases, data lake architectures, containerization, CI/CD practices, and data governance principles will be highly valued. Qualifications: - BE/ME Degree in Computer Science, Electronics, Electrical Desired Experience Level: - Masters with a minimum of 2 years of relevant experience - Bachelors with a minimum of 4 years of relevant experience Experience in the semiconductor industry would be considered a plus for this role.,