Data Engineer

Job Description

JOB DESCRIPTION Developing big data-based applications, leveraging large and complex datasets. Good understanding of the big data ecosystem and design principles. Applying transformations on structured, unstructured and semi-structured data to extract relevant and organized datasets. In-depth understanding of working with relational databases and SQL to support data extraction, transformation and loading operations. Process structured and unstructured data while leveraging data warehouse infrastructure tools and scripting languages. Managing data streaming while implementing messaging services using the pub sub model. Good understanding of data transformation frameworks. Leveraging data clusters, hosted locally and on a cloud platform. Ability to analyze the business requirements and contribute towards a viable solution. Troubleshooting issues and problem solving, as necessary. SKILLS Working with programming languages like Python(Pandas Data Frames, SQLAlchemy, cx_Oracle, NumPy) or Scala to perform operations on underlying complex datasets. Good understanding of the Hadoop ecosystem with emphasis on HDFS and YARN. Test driven development writing unit test cases using PyTest, PyUnit, pytest-cov libraries. Unix scripting Python, Perl, Shell (bash or zsh) Working with data transformation frameworks and technologies such as MapReduce, Spark RDD, SparkSQL, Spark DataFrames, DataSets etc. Good working knowledge on RDBMS and NoSQL databases such as Postgres, DynamoDB, MongoDB, Oracle, MySQL etc. Good exposure to data processing and scripting frameworks like Hive. High level exposure to working with data streaming services like Kafka, Spark Streaming etc. Good understanding of ETL fundamentals and OLAP/OLTP systems. Good knowledge of Data warehousing Concepts/Big Data/Hadoop. Basic Knowledge on Agile development frameworks (Scrum). Working knowledge of leveraging data clusters on cloud Platforms (AWS, GCP, Azure)