Job
Description
As a Prinicipal Integration Engineer, you will play a critical role in designing, developing, and maintaining integrations between data source and sink. Your expertise will be instrumental in setting up cloud infrastructure needed for receiving and processing events from various sources, integrating with external APIs for real-time decision-making, and categorizing events for further analysis and storage in data lake. Additionally, you will also contribute to the development and improvement of DevOps practices that support these integrations. Responsibilities: Develop and maintain high-performance event-driven applications using languages such as .Net, Python, Go, or Node.js Design and implement robust event handling mechanisms, including message queues (e.g., Kafka, RabbitMQ), event streams (e.g., Apache Kafka, Azure Event Hubs), and pub/sub systems Integrate with external APIs and services (e.g., RESTful APIs) to exchange data and trigger actions Develop and maintain data pipelines for ingesting, transforming, and loading data into our data lake (e.g., Azure Data Lake Storage, AWS S3) Perform data analysis and generate insights from event data Ensure data quality and integrity throughout the data lifecycle Enable versioning, logging and monitoring for platform and application observability Troubleshoot and resolve issues related to data pipelines and Azure DevOps integrations Collaborate with cross-functional teams (e.g., data engineers, data scientists, product managers) to understand business requirements and translate them into technical solutions Contribute to the development and improvement of CI/CD pipelines (e.g., using Jenkins, Azure DevOps, GitLab CI) Implement and maintain infrastructure as code (IaC) using tools like Terraform or Ansible Participate in code reviews and contribute to the improvement of development processes Stay up-to-date with the latest technologies and best practices in event-driven architectures and data processing Mentor and guide junior team members DevOps and CI/CD practices Qualifications: Bachelor's degree in Computer Science, Computer Engineering, or a related field 7+ years of professional experience in software development Strong experience with at least one programming language such as Java, Python, Go, or Node.js Proficiency in configuration languages (Terraform, YAML) Experience with message queuing systems (e.g., Kafka, RabbitMQ) or event streaming platforms (e.g., Apache Kafka, Azure Event Hubs) Experience with API integration and RESTful services Experience with data processing frameworks (e.g., Spark) Experience with cloud platforms (e.g., Azure, AWS, GCP) Experience with containerization technologies (e.g., Docker, Kubernetes) Experience with or knowledge of Azure technologies (e.g., Azure Functions, Azure Event Grid, Azure Service Bus) Experience with or knowledge of data warehousing and data lake technologies (e.g., Azure Data Lake Storage, AWS S3) Familiarity with data quality and validation tools Strong analytical and problem-solving skills Excellent communication and collaboration skills Ability to work independently and as part of a team
