Sr. Software Engineer

Technical Skills:(Primary) Overall, 6 8 yrs experience, with relevant experience on Azure Databricks for 4+ yrs Primary Technologies:Azure Databricks, spark, python Database Technologies:SQL, MS SQL Server, Relational Databases Experience in conducting Code Reviews for offshore team members Strong experience on ETL concepts Areas on Primary Skills Good understanding of databricks as a platform to write and execute notebooks.
Should be able to create Databricks workflows and schedule them with dependencies Understands delta table as a concept and versioning to update/delete/merge the existing dataset and for DML operations Good knowledge of Spark framework, its internals and its distributed processing capability to process huge data volumes Good working knowledge of Spark dataframes , operations on it, data manipulations and transformations, not limited to reading or writing data in various modes Have experience in reading complex data structures in csv / xml / json format and parse/transform using Pyspark Understanding of data loader and incremental data processing Knowledge of spark cluster key terms like driver, executor, memory management Solid understanding of SQL concepts Thorough understanding of ETL concepts, data loading strategies and ETL design Understands of CI/CD push/pull code to/from source control repository, source version control, basic operations and terminologies