Job
Description
About the Role: We are looking for a highly skilled Data Engineering Architect with strong Data Engineering pipeline implementation experience to serve as the lead Solution/Technical Architect and Subject Matter Expert for customer experience data solutions across multiple data sources. The ideal candidate will collaborate with the Enterprise Architect and the client IT team to establish and implement strategic initiatives. Responsibilities and Technical Skills : 12+ years of relevant experience in designing and Architecting ETL, ELT, Reverse ETL, Data Management or Data Integration, Data Warehouse, Data Lake, and Data Migration. Must have expertise in building complex ETL pipelines and large Data Processing, Data Quality and Data security Experience in delivering quality work on time with multiple, competing priorities. Excellent troubleshooting and problem-solving skills must be able to consistently identify critical elements, variables and alternatives to develop solutions. Experience in identifying, analyzing and translating business requirements into conceptual, logical and physical data models in complex, multi-application environments. Experience with Agile and Scaled Agile Frameworks. Experience in identifying and documenting data integration issues, and challenges such as duplicate data, non-conformed data, and unclean data. Multiple platform development experience. Strong experience in performance tuning of ETL processes using Data Platforms Must have experience in handling Data formats like Delta Tables, Parquet files, Iceberg etc. Experience in Cloud technologies such as AWS/Azure or Google Cloud. Apache Spark design and development experience using Scala, Java, Python or Data Frames with Resilient Distributed Datasets (RDDs). Development experience in databases like Oracle, AWS Redshift, AWS RDS, Postgres Databricks and/or Snowflake. Hands-on professional work experience with Python is highly desired. Experience in Hadoop ecosystem tools for real-time or batch data ingestion. Strong communication and teamwork skills to interface with development team members, business analysts, and project management. Excellent analytical skills. Identification of data sources, internal and external, and defining a plan for data management as per business data strategy. Collaborating with cross-functional teams for the smooth functioning of the enterprise data system. Managing end-to-end data architecture, from selecting the platform, designing the technical architecture, and developing the application to finally testing and implementing the proposed solution. Planning and execution of big data solutions using Databricks, Big Data, Hadoop, Big Query, Snowflake, MongoDB, DynamoDB, PostgreSQL and SQL Server Hands-on experience in defining and implementing various Machine Learning models for different business needs. Integrating technical functionality, ensuring data accessibility, accuracy, and security. Programming / Scripting Languages like Python / Java / Go, Microservices Machine Learning / AI tools like Scikit-learn / TensorFlow / PyTorch
