5 Spark Optimization Jobs

To transform data into a format that can be easily analyzed, you will be responsible for developing, maintaining, and testing infrastructures for data generation. Working closely with data scientists, you will play a key role in architecting solutions that enable data scientists to perform their tasks efficiently. Your main responsibilities will include: - Developing and maintaining scalable data pipelines and building new API integrations to support increasing data volume and complexity - Connecting offline and online data to enhance customer behavior understanding for personalization - Cleansing data, improving data quality, and preparing data for analysis - Processing data through cleansing and transforming processes for querying and analysis - Ensuring data accessibility and usability across the enterprise for enhanced operational understanding We are looking for candidates with: - A Master's or Bachelor's degree in Engineering, Computer Science, Math, Statistics, or equivalent - Strong programming skills in Python, Pyspark, and SAS - Experience with large data sets and technologies such as Hadoop, Hive, Spark optimization - Knowledge of cloud platforms, preferably Azure, and services like Azure Data Factory, ADLS Storage, Azure DevOps, Databricks, Delta Lake, and Workflows - Familiarity with DevOps processes and tools like Docker, CI/CD, Kubernetes, Terraform, and Octopus - Hands-on experience with SQL, data modeling, and BI tools like Power BI - Cloud migration and Data Engineering certification are considered a plus - Experience working in an Agile environment FedEx is committed to building a diverse, equitable, and inclusive workforce that values fair treatment and growth opportunities for all. As an equal opportunity/affirmative action employer, we welcome qualified applicants from various backgrounds. About FedEx: FedEx is a leading express transportation company, consistently recognized as one of the top 10 Worlds Most Admired Companies by "Fortune" magazine. With a global network serving over 220 countries and territories, FedEx relies on a dedicated team of employees committed to delivering exceptional service every day. Our Philosophy: At FedEx, the People-Service-Profit philosophy guides our decisions and actions. We prioritize our people, who in turn provide exceptional service to our customers, leading to profitability and sustainability. By reinvesting profits back into the business and our employees, we foster a culture of innovation and quality service delivery. Our Culture: FedEx's culture and values have been fundamental to our success since our inception in the 1970s. Our unique culture sets us apart in the global marketplace, emphasizing behaviors, actions, and activities that reflect our commitment to excellence and growth. Through our P-S-P philosophy, we create an environment where team members are encouraged to innovate and provide top-tier service while prioritizing their well-being and contributions to the company.,

We are seeking a highly skilled PySpark Developer with expertise in Distributed data processing to optimize Spark Jobs and ensure efficient data processing in a Big Data platform. This role demands a deep understanding of Spark performance tuning, distributed computing, and Big data architecture. Responsibilities include creating data tools for analytics and data scientist team members, collaborating with data and analytics experts to enhance functionality in the data system, and having the ability to adjust priorities quickly based on circumstances. The ideal candidate should have 8+ years of relevant experience in Apps Development or systems analysis. Key Responsibilities: - Analyze and comprehend existing data ingestion and reconciliation frameworks - Develop PySpark programs to process large datasets in Hive tables and Big data platforms - Perform complex transformations and advanced data manipulations - Fine-tune Spark jobs for performance optimization at scale - Work closely with Data Engineers, Architects, and Analysts to understand data reconciliation requirements - Collaborate with cross-functional teams to enhance data ingestion, transformation, and validation workflows Required Skills and Qualifications: - Extensive hands-on experience with Python, PySpark, and PyMongo for efficient data processing across distributed and columnar databases - Expertise in Spark Optimization techniques, debugging Spark performance issues, and optimizing resource utilization - Proficiency in Python and Spark DataFrame API, with experience in complex data transformations using PySpark - Experience with large-scale distributed data processing, and understanding of Big Data architecture and distributed computing frameworks - Strong problem-solving and analytical skills - Experience with CI/CD for data pipelines - Experience with SnowFlake for data processing and integration Education: - Bachelors degree/University degree in Computer Science or equivalent experience - Masters degree preferred Citi is an equal opportunity and affirmative action employer, encouraging all qualified interested applicants to apply for career opportunities. For applicants with disabilities requiring accommodation, review Accessibility at Citi.,

You will be responsible for developing, optimizing, and maintaining business intelligence and data warehouse systems. Your main accountabilities include developing scalable data pipelines, integrating offline and online data for improved customer understanding, managing data quality and governance strategies, transforming data for analysis, and ensuring data accessibility for the enterprise. To qualify for this role, you should have a Masters/Bachelors degree in Engineering/Computer Science/Math/Statistics or equivalent, strong programming skills in Python/Pyspark/SAS, experience with large datasets and technologies like Hadoop, Hive, and Spark optimization. Additionally, experience with cloud platforms (preferably Azure), DevOps processes and tools, SQL, and data modeling is required. Familiarity with BI tools like Power BI, cloud migration, cloud and data engineering certification, and Agile environment would be advantageous. The ideal candidate should possess 4-6 years of relevant work experience, stakeholder management skills, and proficiency in English, analytical, numerical, and organizational skills, data modeling, ETL, programming, and presentation skills. A Bachelor's degree in Computer Science, MIS, Mathematics, Statistics, or related field is necessary, with a Master's degree or PhD preferred. FedEx is an equal opportunity/affirmative action employer committed to diversity and inclusion in the workforce. Regardless of age, race, gender, disability, or any other characteristic protected by law, all qualified applicants will be considered for employment. As one of the world's largest express transportation companies, FedEx values its team members and upholds the People-Service-Profit philosophy. This philosophy emphasizes taking care of employees to deliver exceptional service to customers and reinvesting profits back into the business and its people. FedEx's culture, built on values and behaviors, has been integral to its success and growth since its inception in the 1970s, setting it apart in the global marketplace.,

Hiring, Data Engineer 4+ years of hands-on experience Experience in building and optimizing Big Data data pipelines, architectures, and data sets. Experience with big data tools: Hadoop, Spark Optimization, Spark-Streaming, Hive, Kafka, Hbase, Airflow, etc. Hands-on in SQL

About Impetus: Impetus Technologies is a digital engineering company dedicated to offering expert services and products to support enterprises in accomplishing their transformation objectives. Specializing in solving the analytics, AI, and cloud challenges, we empower businesses to foster unparalleled innovation and expansion. Established in 1991, we stand out as leaders in cloud and data engineering, catering cutting-edge solutions to Fortune 100 corporations. Our headquarters are located in Los Gatos, California, while our development centers span across NOIDA, Indore, Gurugram, Bengaluru, Pune, and Hyderabad, boasting a global team of over 3000 professionals. Additionally, we have operational offices in Canada and Australia and maintain collaborative relationships with renowned organizations such as American Express, Bank of America, Capital One, Toyota, United Airlines, and Verizon. Skills Required: - Bigdata - Pyspark - Hive - Spark Optimization Good to have: - GCP,