Job
Description
Data Pipeline Development : Designing, building, and maintaining robust data pipelines to move data from various sources (e.g., databases, external APIs, logs) to centralized data systems, such as data lakes or warehouses. Data Integration : Integrating data from multiple sources and ensuring it's processed in a consistent, usable format. This involves transforming, cleaning, and validating data to meet the needs of products, analysts and data scientists. Database Management : Creating, managing, and optimizing databases for storing large amounts of structured and unstructured data. Ensuring high availability, scalability, and security of data storage solutions. Performance Optimization : Identifying and resolving issues related to the speed and efficiency of data systems. This could include optimizing queries, storage systems, and improving overall system architecture. Automation: Automating routine tasks, such as data extraction, transformation, and loading (ETL), to ensure smooth data flows with minimal manual intervention. Collaboration with Data Teams: Working closely with Work closely with product managers, UX/UI designers, and other stakeholders to understand data requirements and ensure data is in the right format for analysis and modeling. Data Governance and Quality : Ensuring data integrity and compliance with data governance policies, including data quality standards, privacy regulations (e.g., GDPR), and security protocols. Monitoring and Troubleshooting : Continuously monitoring data pipelines and databases for any disruptions or errors and troubleshooting any issues that arise to ensure continuous data flow. Tool and Technology Management : Staying up to date with emerging data tools, technologies, and best practices in order to improve data systems and infrastructure. Documentation and Reporting : Documenting data systems, pipeline processes, and data architectures, providing clear instructions for the team to follow, and ensuring that the architecture is understandable for stakeholders. Required Skills & Experience: Knowledge of data bases, relational DBs such as Postgres as well as NoSQL systems such as MongoDB, Kafka Knowledge of Big Data systems such as Hadoop, Hive/Pig, Trino etc Experience in SQL like query languages (SQL, MQL, HQL etc) Experience in building data pipelines Experience in Software lifecycle tools for CI/CD and version control system such as GIT Familiarity with Agile methodologies is a plus. - General: - Strong problem-solving skills and the ability to troubleshoot complex software issues. - Familiarity with version control systems, particularly Git. - Experience with Agile methodologies (e.g., Scrum, Kanban). - Excellent communication skills, both verbal and written, with the ability to collaborate in a team environment. - Bachelor's degree in Computer Science, Engineering, or a related field, or equivalent work experience. Preferred Qualifications: Experience work in GCP and familiarity with Kubernetes, Big Query, GCS, Airflow Problem-Solving: Strong analytical and problem-solving skills. Communication: Excellent verbal and written communication skills, with the ability to convey technical concepts to non-technical stakeholders. Team Player: Ability to work collaboratively in a team-oriented environment. Adaptability: Flexibility to adapt to changing business needs and priorities.
