Data Engineer (Databricks)

Role Overview: You will be responsible for designing and building optimized data pipelines using cutting-edge technologies in a cloud environment to drive analytical insights. Your role will involve constructing infrastructure for efficient ETL processes from various sources and storage systems, leading the implementation of algorithms and prototypes, and architecting database pipeline architectures. Additionally, you will create data validation methods, ensure compliance with data governance and security policies, interpret data trends, develop analytical tools, and continuously explore opportunities to enhance data quality and reliability. Key Responsibilities: - Design and build optimized data pipelines using cutting-edge technologies in a cloud environment - Construct infrastructure for efficient ETL processes from various sources and storage systems - Lead the implementation of algorithms and prototypes to transform raw data into useful information - Architect, design, and maintain database pipeline architectures - Create innovative data validation methods and data analysis tools - Ensure compliance with data governance and security policies - Interpret data trends and patterns to establish operational alerts - Develop analytical tools, programs, and reporting mechanisms - Conduct complex data analysis and present results effectively - Prepare data for prescriptive and predictive modeling - Continuously explore opportunities to enhance data quality and reliability - Apply strong programming and problem-solving skills to develop scalable solutions Qualifications Required: - Experience in Big Data technologies such as Hadoop, Spark, Nifi, Impala - 5+ years of hands-on experience designing, building, deploying, testing, maintaining, monitoring, and owning scalable, resilient, and distributed data pipelines - High proficiency in Scala/Java and Spark for applied large-scale data processing - Expertise with big data technologies, including Spark, Data Lake, and Hive Role Overview: You will be responsible for designing and building optimized data pipelines using cutting-edge technologies in a cloud environment to drive analytical insights. Your role will involve constructing infrastructure for efficient ETL processes from various sources and storage systems, leading the implementation of algorithms and prototypes, and architecting database pipeline architectures. Additionally, you will create data validation methods, ensure compliance with data governance and security policies, interpret data trends, develop analytical tools, and continuously explore opportunities to enhance data quality and reliability. Key Responsibilities: - Design and build optimized data pipelines using cutting-edge technologies in a cloud environment - Construct infrastructure for efficient ETL processes from various sources and storage systems - Lead the implementation of algorithms and prototypes to transform raw data into useful information - Architect, design, and maintain database pipeline architectures - Create innovative data validation methods and data analysis tools - Ensure compliance with data governance and security policies - Interpret data trends and patterns to establish operational alerts - Develop analytical tools, programs, and reporting mechanisms - Conduct complex data analysis and present results effectively - Prepare data for prescriptive and predictive modeling - Continuously explore opportunities to enhance data quality and reliability - Apply strong programming and problem-solving skills to develop scalable solutions Qualifications Required: - Experience in Big Data technologies such as Hadoop, Spark, Nifi, Impala - 5+ years of hands-on experience designing, building, deploying, testing, maintaining, monitoring, and owning scalable, resilient, and distributed data pipelines - High proficiency in Scala/Java and Spark for applied large-scale data processing - Expertise with big data technologies, including Spark, Data Lake, and Hive