Data Engineer and Architect (AWS/Azure/GCP/Oracle)

Role & responsibilities

• Build and deploy data platforms, data warehouses, and big data solutions across industries (BFSI, Manufacturing, Healthcare, eCommerce, IoT, Digital Twin, etc)
• Integrating, transforming, and consolidating data from various structured and unstructured data systems.
• Expert in data ingestion, transformation, storage, and analysis, often using Azure services and migration from legacy on-premise services
• Essential skills include SQL, Python, R and knowledge of ETL/ELT processes and big data technologies like Apache Spark, Scala, PySpark.
• Maintain data integrity, resolve data-related issues, and ensure the reliability and performance of data solutions
• Work with stakeholders to provide real-time data analytics, monitor data pipelines, and optimize performance and scalability
• Strong understanding of data management fundamentals, data warehousing, and data modeling.
• Bigdata technologies HDFS, Spark, Hbase, Hive, Sqoop, Kafka, RabbitMQ, Flink
• Implement seamless data integration solutions between Azure/AWS/GCP and Snowflake platforms.
• Identify and resolve performance bottlenecks, optimize queries, and ensure the overall efficiency of data pipelines.
• Lead the development and management of data infrastructure, including tools, dashboards, queries, reports, and scripts, ensuring automation of recurring tasks while maintaining data quality and integrity
• Implement and maintain data security measures, ensuring compliance with industry standards and regulations.
• Ensure data architecture aligns with business requirements and best practices.
• Experience in Power BI /Tableau /Looker
• Management, administration, and maintenance with data streaming tools such as Kafka/Confluent Kafka, Flink
• Experience in Test Driven Development, building libraries, and proficiency in Pandas, NumPy, Elasticsearch, Apache Beam
• Familiarity with CI/CD pipelines, monitoring, and infrastructure-as-code (e. g., Terraform, CloudFormation).
  Proficient in query optimization, data partitioning, indexing strategies, and caching mechanisms.
• Ensure GDPR, SOX, and other regulatory compliance across data workflows
• Essential skills include SQL, Python, and knowledge of ETL/ELT processes and big data technologies like Spark.
• Exposure in working on Event/File/Table Formats such as Avro, Parquet, Iceberg, Delta

Must have Skills (Atleast one or two)

• Azure Data Factory (ADF), Databricks, Synapse, Data Lake Storage, Timeseries Insights, Azure SQL Database, SQL Server, Presto, SSIS
• AWS data services such as S3, Glue Studio, Redshift, Athena, and EMR, Redshift , Airflow, IAM, DBT, Lambda, RDS, DynamoDB, Neo4j, Amazon Neptune
• GCP Big query, SQL, Composer, Dataflow, Dataform, DBT, /Python, Cloud functions, Dataproc+pyspark, Python injection, Dataflow, Cloud Storage, Pub/Sub, and Vertex AI, GKE, Cloud Functions
• OCI Object Storage, OCI Data Integration, Oracle Database, Oracle Analytics Cloud, Oracle Analytics Cloud (OAC), Autonomous Data Warehouse (ADW), NetSuite Analytics Warehouse (NSAW), PL/SQL, Exadata