8 Airflow Pyspark Jobs

We are looking for an experienced Senior Software Engineer with strong expertise in full-stack development and cloud technologies. The ideal candidate will be responsible for designing, developing, testing, and maintaining scalable software applications and products. You will be deeply involved in the entire software development lifecycleright from architecture design to deployment—while collaborating with cross-functional teams and driving user-centric solutions. Key Responsibilities Engage with customers to understand use cases, pain points, and requirements, and advocate for user-focused solutions. Design, develop, and deliver software applications using various tools, frameworks, and methodologies (Agile). Assess business and technical requirements to determine the best technology stack, integration methods, and deployment strategy. Create high-level software architecture designs, including structure, components, and interfaces. Collaborate closely with product owners, designers, and architects to align on solutions. Define and implement software testing strategies, policies, and best practices. Continuously optimize application performance and adopt new technologies to enhance efficiency. Apply programming practices such as Test-Driven Development (TDD), Continuous Integration (CI), and Continuous Delivery (CD). Implement secure coding practices, including encryption and anonymization of user data. Develop user-friendly, interactive front-end interfaces and robust back-end services (APIs, microservices). Leverage cloud platforms and emerging technologies to build future-ready solutions. Skills Required Programming & Data Engineering: Python, PySpark, API Development, SQL/Postgres Cloud Platforms & Tools: Google Cloud Platform (BigQuery, Cloud Run, Dataflow, Dataproc, Data Fusion, Cloud SQL), IBM WebSphere Application Server Infrastructure & DevOps: Terraform, Tekton, Airflow Other Expertise: MDM (Master Data Management), application optimization, microservices Experience Required 10+ years of experience in IT, with 8+ years in software development. Strong practical experience in at least 2 programming languages OR advanced expertise in 1 language. Experience mentoring and guiding engineering teams.

Overview: TekWissen is a global workforce management provider throughout India and many other countries in the world. The below clientis a global company with shared ideals and a deep sense of family. From our earliest days as a pioneer of modern transportation, we have sought to make the world a better place one that benefits lives, communities and the planet Job Title: Data Engineering Engineer II Location: Chennai Work Type: Hybrid Position Description: Employees in this job function are responsible for designing, building, and maintaining data solutions including data infrastructure, pipelines, etc. for collecting, storing, processing and analyzing large volumes of data efficiently and accurately Key Responsibilities: Collaborate with business and technology stakeholders to understand current and future data requirements Design, build and maintain reliable, efficient and scalable data infrastructure for data collection, storage, transformation, and analysis Plan, design, build and maintain scalable data solutions including data pipelines, data models, and applications for efficient and reliable data workflow Design, implement and maintain existing and future data platforms like data warehouses, data lakes, data lakehouse etc. for structured and unstructured data Design and develop analytical tools, algorithms, and programs to support data engineering activities like writing scripts and automating tasks Ensure optimum performance and identify improvement opportunities Skills Required: Google Cloud Platform - Biq Query, Data Flow, Dataproc, Data Fusion, TERRAFORM, Tekton,Cloud SQL, AIRFLOW, POSTGRES, Airflow PySpark, Python, API Skills Preferred: GenAI Experience Required: Engineer 2 Exp: 4+ years Data Engineering work experience Experience Preferred: Strong proficiency and hands-on experience in both Python(Must-have) and Java(Nice to have). Experience building and maintaining data pipelines (batch or streaming) preferably on Cloud platforms(especially GCP). Experience with at least one major distributed data processing framework (e.g., DBT, DataForm, Apache Spark, Apache Flink, or similar). Experience with workflow orchestration tools (e.g., Apache Airflow, Qlik replicate etc). Experience working with relational databases (SQL) and understanding of data modeling principles. Experience with cloud platforms (Preferably GCP. AWS or Azure will also do) and relevant data services (e.g., BigQuery, GCS, Data Factory, Dataproc, Dataflow, S3, EMR, Glue etc.). Experience with data warehousing concepts and platforms (BigQuery, Snowflake, Redshift etc.). Understanding of concepts related to integrating or deploying machine learning models into production systems. Experience working in an Agile development environment & hands-on in any Agile work management tool(Rally, JIRA etc.). Experience with version control systems, particularly Git. Solid problem-solving, debugging, and analytical skills. Excellent communication and collaboration skills. Experience working in a production support team (L2/L3) for operational support. Preferred Skills and Qualifications (Nice to Have): Familiarity with data quality and data governance concepts. Experience building and consuming APIs (REST, gRPC) related to data or model serving. Bachelor's or Master's degree in Computer Science, Engineering, Data Science, or a related field. Education Required: Bachelor's Degree Education Preferred: Bachelor's Degree TekWissen Group is an equal opportunity employer supporting workforce diversity.

Position Description: Employees in this job function are responsible for designing, building, and maintaining data solutions including data infrastructure, pipelines, etc. for collecting, storing, processing and analyzing large volumes of data efficiently and accurately Key Responsibilities: 1) Collaborate with business and technology stakeholders to understand current and future data requirements 2) Design, build and maintain reliable, efficient and scalable data infrastructure for data collection, storage, transformation, and analysis 3) Plan, design, build and maintain scalable data solutions including data pipelines, data models, and applications for efficient and reliable data workflow 4) Design, implement and maintain existing and future data platforms like data warehouses, data lakes, data lakehouse etc. for structured and unstructured data 5) Design and develop analytical tools, algorithms, and programs to support data engineering activities like writing scripts and automating tasks 6) Ensure optimum performance and identify improvement opportunities

Overview: TekWissen is a global workforce management provider throughout India and many other countries in the world. The below clientis a global company with shared ideals and a deep sense of family. From our earliest days as a pioneer of modern transportation, we have sought to make the world a better place one that benefits lives, communities and the planet Job Title: Specialty Development Practitioner Location: Chennai Work Type: Hybrid Position Description: This role is for a proactive Full Stack Software Engineer responsible for creating products to host Supply Chain Analytics algorithms. You will ensure software engineering excellence while developing web applications and tools, employing practices like pair programming and Test-Driven Development (TDD) within an Agile environment. Key responsibilities include acting as a change agent, mentoring teams on Agile methodologies, and contributing to Client's institutional knowledge. Strong written and oral communication skills are essential for interacting with Client leadership, along with a self-starting approach. Required Skills: Bachelor's or Master's degree in Computer Science, Computer Engineering, or a related technical field. 5-7+ years of software engineering and testing experience, including Agile methodologies and Jira. Technical requirements include 3+ years in Python, Java, and Spring Boot development 3+ years with REST APIs; and 3+ years developing web-based UIs using JavaScript, React, Angular, Vue, or TypeScript, along with Pub Sub, APIGEE, and Cloud Storage. Experience with relational (e.g., PostgreSQL, SQL Server), NoSQL, and columnar databases (e.g., BigQuery) is necessary. At least 1 year of experience developing and deploying to cloud platforms such as Google Cloud Platform, Pivotal Cloud Foundry, Amazon Web Services, and Microsoft Azure is also required. A passion for clean code and a strong desire for continuous learning are key. Desired Skills: Full-stack expertise, automated testing (Unit, Integration, E2E), Cloud Computing/Infrastructure experience (especially Google Cloud Platform, Cloud Run containerization, and Google Cloud Storage), and proficiency with Continuous Integration/Continuous Delivery tools like Jenkins, Tekton, or Gradle. Skills Required: Big Query,, Python, Angular, Relational Databases, Google Cloud Platform, Google Cloud Platform Biq Query, Data Flow, Dataproc, Data Fusion, TERRAFORM, Tekton,Cloud SQL, AIRFLOW, POSTGRES, Airflow PySpark, Python, API Experience Required: 5+ Years Education Required: Bachelor's Degree TekWissen® Group is an equal opportunity employer supporting workforce diversity.

Overview: TekWissen is a global workforce management provider throughout India and many other countries in the world. The below clientis a global company with shared ideals and a deep sense of family. From our earliest days as a pioneer of modern transportation, we have sought to make the world a better place one that benefits lives, communities and the planet Job Title: Software Engineer Practitioner Location: Chennai Work Type: Hybrid Position Description: We're seeking a highly skilled and experienced Full Stack Data Engineer to play a pivotal role in the development and maintenance of our Enterprise Data Platform. In this role, you'll be responsible for designing, building, and optimizing scalable data pipelines within our Google Cloud Platform (GCP) environment. You'll work with GCP Native technologies like BigQuery, Dataform,Dataflow, and Pub/Sub, ensuring data governance, security, and optimal performance. This is a fantastic opportunity to leverage your full-stack expertise, collaborate with talented teams, and establish best practices for data engineering at client. Basic Qualifications: Bachelors or Masters degree in a Computer Science, Engineering or a related or related field of study 5+ Years - Strong understating of Database concepts and experience with multiple database technologies optimizing query and data processing performance. 5+ Years - Full Stack Data Engineering Competency in a public cloud Google Critical thinking skills to propose data solutions, test, and make them a reality. 5+ Years - Highly Proficient in SQL, Python, Java- Experience programming engineering transformation in Python or a similar language. 5+ Years - Ability to work effectively across organizations, product teams and business partners. 5+ Years - Knowledge Agile (Scrum) Methodology, experience in writing user stories Deep understanding of data service ecosystems including data warehousing, lakes and Marts User experience advocacy through empathetic stakeholder relationship. Effective Communication both internally (with team members) and externally (with stakeholders) Knowledge of Data Warehouse concepts experience with Data Warehouse/ ETL processes Strong process discipline and thorough understating of IT processes (ISP, Data Security). Skills Required: Data Architecture, Data Warehousing, DataForm, Google Cloud Platform - Biq Query, Data Flow, Dataproc, Data Fusion, TERRAFORM, Tekton,Cloud SQL, AIRFLOW, POSTGRES, Airflow PySpark, Python, API Experience Required: Excellent communication, collaboration and influence skills; ability to energize a team. Knowledge of data, software and architecture operations, data engineering and data management standards, governance and quality Hands on experience in Python using libraries like NumPy, Pandas, etc. Extensive knowledge and understanding of GCP offerings, bundled services, especially those associated with data operations Cloud Console, BigQuery, DataFlow, Dataform, PubSub Experience with recoding, re-developing and optimizing data operations, data science and analytical workflows and products. Experience Required: 5+ Years Education Required: Bachelor's Degree TekWissen Group is an equal opportunity employer supporting workforce diversity.

Role & responsibilities Bachelors Degree 2+Years in GCP Services - Biq Query, Data Flow, Dataproc, DataPlex,DataFusion, Terraform, Tekton, Cloud SQL, Redis Memory, Airflow, Cloud Storage 2+ Years inData Transfer Utilities 2+ Years in Git / any other version control tool 2+ Years in Confluent Kafka 1+ Years of Experience in API Development 2+ Years in Agile Framework 4+ years of strongexperience in python, Pyspark development. 4+ years of shell scripting to develop the adhoc jobsfor data importing/exporting Google Cloud Platform - Biq Query, Data Flow, Dataproc, Data Fusion, TERRAFORM, Tekton,Cloud SQL, AIRFLOW, POSTGRES, Airflow PySpark, Python, API

Role & responsibilities Python, Dataproc, Airflow PySpark, Cloud Storage, DBT, DataForm, NAS, Pubsub, TERRAFORM, API, Big Query, Data Fusion, GCP, Tekton Preferred candidate profile Data Engineer in Python - GCP Location Chennai Only 4+ Years of Experience

Job Title: Software Engineer III - Core Engineer III Location: Chennai Work Type: Hybrid Overview: TekWissen is a global workforce management provider throughout India and many other countries in the world. The below clientis a global company with shared ideals and a deep sense of family. From our earliest days as a pioneer of modern transportation, we have sought to make the world a better place one that benefits lives, communities and the planet Position Description: MUST HAVE: Performs DevSecOps and SRE adoption for existing/new SpringBoot/Angular/REACT based applications to deploy in CaaS/GCP CloudRun. 5+ years of experience in Java/SpringBoot/Angular based development projects Hands-on experience in implementing Security tools like Cycode (SAST),42 crunch(Open API standard), Sonarqube, FOSSA(SCA), Kyverno(Policy as code) for existing/new applications Experience on programming languages such as Python or R, with experience in AI/ML libraries 2+ years of experience working in Agile projects Experience on Development, Operational, Monitoring tools with knowledge on SRE will be advantageous Research new tools and technologies, influences and implements enterprise technology shifts and new trends impacting Client Dev Ops Strategy Consults to resolve complex application DevSecOps challenges/issues. Consults with teams on applying relevant DevSecOps Integration tools suitable for the application. Experience in handling Source control Tool like Git Strong Experience in developing and troubleshooting Build scripts like GRADLE Experience in Developing and maintaining pipeline tools like Cloud build/Tekton Strong Experience in handling and troubleshooting Dev Sec Ops tools like Cycode, FOSSA, 42 crunch, SonarQube Strong Experience in integrating Dev Ops Testing tools like Selenium, Playwright NICE TO HAVE Experience in handling Monitoring & Logging tools like Dynatrace, Splunk and Grafana • Experience in deploying applications in OpenShift CaaS, GCP Cloud Run Experience and understanding of various LLMs and GenAI capabilities to bring ideas to production for enablers, automation etc. Experience with DevSecOps practices and integrating security into CI/CD pipelines. (including Jenkins/Tekton, GitLab CI, GitHub Actions), infrastructure-as-code (Terraform, Ansible, CloudFormation), and automation. Skills Required: Full Stack Java Developer, DevOps, Google Cloud Platform - Biq Query, Data Flow, Dataproc, Data Fusion, TERRAFORM, Tekton,Cloud SQL, AIRFLOW, POSTGRES, Airflow PySpark, Python, API Skills Preferred: Dynatrace, AI/ML, AIPGEE Experience Required: Engineer 3 Exp: Prac. In 2 coding lang. or adv. Prac. in 1 lang. 6+ years in IT; 4+ years in development Education Required: Bachelor's Degree TekWissen Group is an equal opportunity employer supporting workforce diversity.