4 Sparkml Jobs

This role is for one of Weekday's clients Min Experience: 8 years Location: Pune JobType: full-time Title: Principal Data Engineer Job Description: Job Summary: Lead the design, development, and optimization of enterprise-scale data pipelines and architectures to drive data-driven decision-making. Architect efficient data tables and schemas to enhance query performance, ensuring seamless integration with cross-functional teams Key Responsibilities: Data Pipeline Design and implement high-performance data pipelines using ETL processes, batch/streaming frameworks (e.g., Apache Spark, Airflow), and cloud platforms (AWS). Optimize data ingestion, transformation, and storage workflows to meet scalability, reliability, and latency requirements. Data Architecture & Query Efficiency Architect database schemas and dimensional models (e.g., star/snowflake) to maximize query performance and reduce latency. Implement indexing strategies, partitioning, and materialized views to optimize data retrieval. Collaboration & Governance Partner with data scientists, product managers, and engineers to align data infrastructure with business needs. Establish data governance frameworks, ensuring compliance with security, privacy, and quality standards. Mentor junior engineers and foster best practices in data engineering Innovation & Leadership Stay ahead of industry trends (e.g., AI/ML integration, real-time analytics) and advocate for emerging technologies. Lead technical strategy for data infrastructure, balancing innovation with operational stability. Technical Expertise: 8+ years of experience in data engineering, with a focus on ETL, data modelling, and cloud-based architectures. Proficiency in SQL, Python, and tools like Spark and Snowflake. Strong knowledge of database design, dimensional modeling, and query optimization. Leadership & Collaboration Proven ability to lead teams, mentor engineers, and drive technical initiatives. Experience translating business requirements into scalable technical solutions. Preferred qualification Expertise in machine learning model deployment or real-time analytics. Familiarity with DevOps practices (CI/CD, Docker, Kubernetes). Our Core Values Data Fanatics: Our edge is always found in the data Partner Obsessed: We are obsessed with partner success Team of Doers: We have a bias for action Game Changers: We encourage innovation

Job Type: Full time Job description: Experience of at least 3 years in developing models using Python/PySpark libraries. Experience with Databricks or Dataiku DSS (Data Science Studio) would be an added advantage. Strong experience with Scala/Python/Java and Spark is required. Strong proficiency in developing, training, and evaluating state-of-the-art machine learning models and deploying them is required. Proficiency in statistical and probabilistic methods such as SVM, Decision-Trees, Bagging and Boosting Techniques, and Clustering is needed. Proficiency in core NLP techniques such as Text Classification, Named Entity Recognition (NER), Topic Modeling, and Sentiment Analysis is required. Understanding Generative AI, Large Language Models, and Transformers would be an added advantage. Have hands-on experience in Python data-science and math packages such as NumPy, Pandas, Sklearn, Seaborn, PyCaret, and Matplotlib. Proficiency in Python and common Machine Learning frameworks such as TensorFlow, NLTK, Stanford NLP, PyTorch, Ling Pipe, Caffe, Keras, SparkML, and OpenAI is required. Experience of working in large teams and using collaboration tools like GIT, Jira, and Confluence is a must. Good understanding of any of the cloud platforms such as AWS, Azure, or GCP is required. Understanding of Commercial Pharma landscape and Patient Data/Analytics would be a huge plus. Should have an attitude of willingness to learn, accepting the challenging environment, and confidence in delivering the results within timelines.

Experience: 15+ years in solution designs for data and analytics deliveries. Skills: • Experience with a range of Azure-based big data and analytics platforms like ADLS+ ADF+ DataBricks+ Azure Data Warehouse+ Power BI+ Azure Cosmo DB+ Azure ML+ etc. • Hands-on experience in the design of Near Real-time Data Processing solutions using DataBricks+ In-Stream architecture+ and modular programming. • Hands-on experience in SparkSQL+ SparkML+ PySpark+ Python • Extensive experience consulting with business; translating business use cases into data analytics specifications and designing big data solutions to deliver these use cases. • Conducting review sessions with Architecture+ Solution Design & Project Development resources as required during the Design and Development Phases. • Certifications: Certification in Designing an Azure Data Solution or relevant to Azure Architect role is mandatory. Expertise: • Solution/technical architecture in the cloud for infrastructure migration+ Big Data/analytics+ information analysis+ database management in the cloud+ IoT/event-driven/microservices in the cloud. • Experience with private public cloud architectures+ pros/cons+ and migration considerations. • Extensive hands-on experience implementing infra migration+ data migration+ and data processing using Azure services: Networking+ Windows/Linux virtual machines+ Container+ Storage+ ELB+ AutoScaling+ Azure Functions+ Serverless Architecture+ ARM Templates+ Azure SQL DB/DW+ Data Factory+ Azure Stream Analytics+ Azure Analysis Service. • DevOps on an Azure platform+ developing and deploying ETL solutions on Azure. • Strong in Power BI+ Java+ C+ Spark+ PySpark+ Unix shell/Perl scripting. • Familiarity with the technology stack available in the industry for metadata management: Data Governance+ Data Quality+ MDM+ Lineage

Responsibilities Responsibilities : Construct machine learning lifecycle management including data collection, normalization, and standardization within a data pipeline construction Architect technology components and build applications and interfaces for customer's consumption Experiment, develop and productionize high quality machine learning services and platforms to make huge technology and business impact. Develop hosting platform for machine learning models. Create pipelines to query and retrieve and update data for existing applications to keep them updated Supervise the scaling and management of the machine learning modeling ecosystem Work alongside data scientists and product owners to improve aspects of their lines of business through machine learning What Were Looking For: Possess excellent verbal & written communication skills Expertise in application, data, and infrastructure architecture disciplines Advanced knowledge of architecture and design across all systems Proficiency in multiple machine learning programming languages including Python, PySpark, or Scala 5 + years of experience with big-data technologies such as Hadoop, Spark, SparkML, etc Able to understand various data structures and common methods in data transformation Knowledge of industry-wide technology trends and best practices Ability to work in large, collaborative teams to achieve organizational goals Passionate about building an innovative culture Familiarity with MLOps and ModelOps Docker, Kubernetes, AWS, Python Familiarity with or clear interest in learning about financial Markets.