Job
Description
Job Category: Data Engineer Job Type: Hybrid Job Location: Bangalore Job Experience: 4-7 years We are seeking a Data Engineer to join our growing team. The Data Engineer will be responsible for designing, developing, and maintaining our ETL pipelines and managing our database systems. The ideal candidate should have a strong background in SQL, database design, and ETL processes. Key Responsibilities Responsibilities for the job Analyse the different source systems, profile data, understand, document & fix Data Quality issues. Gather requirements and business process knowledge to transform the data in a way that is geared towards the needs of end users. Write complex SQLs to extract & format source data for ETL/data pipeline. Design, implement, and maintain systems that collect and analyze business intelligence data. Design and architect an analytical data store or cluster for the enterprise and implement data pipelines that extract, transform, and load data into an information product that helps the organization reach strategic goals. Create design documents, Source to Target Mapping documents and any supporting documents needed for deployment/migration. Design, Develop and Test ETL/Data pipelines. Design & build metadata-based frameworks needs for data pipelines. Write Unit Test cases, execute Unit Testing and document Unit Test results. Manage and maintain the database, warehouse, & cluster with other dependent infrastructure. Expertise in managing and optimizing Spark clusters, along with other implementations of Spark. Strong programming skills in Python and Py-spark. Strong proficiency in SQL and experience with relational databases (PostgreSQL, MySQL, Oracle, etc.) and NoSQL databases (MongoDB, Cassandra, DynamoDB). Knowledge of data modelling techniques such as star/snowflake, data vault, etc. Knowledge of semantic modelling. Strong problem-solving skills Be able to hone business acumen with a capacity for straddling between macro business strategy to micro tangible data and AI products. Perform data cleaning, transformation, and validation to ensure accuracy and consistency across various data sources. Technologies preferred: Azure, Databricks Eligibility Criteria for the Job Education B.E/B.Tech in any specialization, BCA, MTech in any specialization, MCA. Primary Skill SQL Databricks Any one of the cloud experiences (AWS, Azure, GCP). Python, Pyspark Management Skills Ability to handle given tasks and projects simultaneously in an organized and timely manner. Soft Skills Good communication skills, verbal and written. Attention to details. Positive attitude and confidence. Show more Show less
