Job
Description
Key Responsibilities: • Data Pipeline Development: Designing, building, and maintaining robust data pipelines to move data from various sources (e.g., databases, external APIs, logs) to centralized data systems, such as data lakes or warehouses. • Data Integration : Integrating data from multiple sources and ensuring it's processed in a consistent, usable format. This involves transforming, cleaning, and validating data to meet the needs of products, analysts and data scientists. • Database Management : Creating, managing, and optimizing databases for storing large amounts of structured and unstructured data. Ensuring high availability, scalability, and security of data storage solutions. • Performance Optimization: Identifying and resolving issues related to the speed and efficiency of data systems. This could include optimizing queries, storage systems, and improving overall system architecture. • Automation : Automating routine tasks, such as data extraction, transformation, and loading (ETL), to ensure smooth data flows with minimal manual intervention. • Collaboration with Data Teams : Working closely with Work closely with product managers, UX/UI designers, and other stakeholders to understand data requirements and ensure data is in the right format for analysis and modeling. • Data Governance and Quality : Ensuring data integrity and compliance with data governance policies, including data quality standards, privacy regulations (e.g., GDPR), and security protocols. • Monitoring and Troubleshooting : Continuously monitoring data pipelines and databases for any disruptions or errors and troubleshooting any issues that arise to ensure continuous data flow. • Tool and Technology Management : Staying up to date with emerging data tools, technologies, and best practices in order to improve data systems and infrastructure. • Documentation and Reporting : Documenting data systems, pipeline processes, and data architectures, providing clear instructions for the team to follow, and ensuring that the architecture is understandable for stakeholders. Required Skills & Experience: • Knowledge of data bases, relational DBs such as Postgres as well as NoSQL systems such as MongoDB, Kafka • Knowledge of Big Data systems such as Hadoop, Hive/Pig, Trino etc • Experience in SQL like query languages (SQL, MQL, HQL etc) • Experience in building data pipelines • Experience in Software lifecycle tools for CI/CD and version control system such as GIT • Familiarity with Agile methodologies is a plus. - General: - Strong problem-solving skills and the ability to troubleshoot complex software issues. - Familiarity with version control systems, particularly Git. - Experience with Agile methodologies (e.g., Scrum, Kanban). - Excellent communication skills, both verbal and written, with the ability to collaborate in a team environment. - Bachelor's degree in Computer Science, Engineering, or a related field, or equivalent work experience. Preferred Qualifications: • Experience work in GCP and familiarity with Kubernetes, Big Query, GCS, Airflow • Problem-Solving: Strong analytical and problem-solving skills. • Communication: Excellent verbal and written communication skills, with the ability to convey technical concepts to non-technical stakeholders. • Team Player: Ability to work collaboratively in a team-oriented environment. • Adaptability: Flexibility to adapt to changing business needs and priorities.worlHi Mugesh
