Specialist Data/AI Engineering

Job Description:

Key Responsibilities:

Data Engineering & Architecture:

• Design, develop, and maintain high-performance data pipelines for structured and unstructured data using Azure Data Bricks and Apache Spark.
• Build and manage scalable data ingestion frameworks for batch and real-time data processing.
• Implement and optimize data lake architecture in Azure Data Lake to support analytics and reporting workloads.
• Develop and optimize data models and queries in Azure Synapse Analytics to power BI and analytics use cases.

Cloud-Based Data Solutions:

• Architect and implement modern data lakehouses combining the best of data lakes and data warehouses.
• Leverage Azure services like Data Factory, Event Hub, and Blob Storage for end-to-end data workflows.
• Ensure security, compliance, and governance of data through Azure Role-Based Access Control (RBAC) and Data Lake ACLs.

ETL/ELT Development:

• Develop robust ETL/ELT pipelines using Azure Data Factory, Data Bricks notebooks, and PySpark.
• Perform data transformations, cleansing, and validation to prepare datasets for analysis.
• Manage and monitor job orchestration, ensuring pipelines run efficiently and reliably.

Performance Optimization:

• Optimize Spark jobs and SQL queries for large-scale data processing.
• Implement partitioning, caching, and indexing strategies to improve performance and scalability of big data workloads.
• Conduct capacity planning and recommend infrastructure optimizations for cost-effectiveness.

Collaboration & Stakeholder Management:

• Work closely with business analysts, data scientists, and product teams to understand data requirements and deliver solutions.
• Participate in cross-functional design sessions to translate business needs into technical specifications.
• Provide thought leadership on best practices in data engineering and cloud computing.

Documentation & Knowledge Sharing:

• Create detailed documentation for data workflows, pipelines, and architectural decisions.
• Mentor junior team members and promote a culture of learning and innovation.

Required Qualifications:

• Experience:
  • 7+ years of experience in data engineering, big data, or cloud-based data solutions.
  • Proven expertise with Azure Data Bricks, Azure Data Lake, and Azure Synapse Analytics.
• Technical Skills:
  • Strong hands-on experience with Apache Spark and distributed data processing frameworks.
  • Advanced proficiency in Python and SQL for data manipulation and pipeline development.
  • Deep understanding of data modeling for OLAP, OLTP, and dimensional data models.
  • Experience with ETL/ELT tools like Azure Data Factory or Informatica.
  • Familiarity with Azure DevOps for CI/CD pipelines and version control.
• Big Data Ecosystem:
  • Familiarity with Delta Lake for managing big data in Azure.
  • Experience with streaming data frameworks like Kafka, Event Hub, or Spark Streaming.
• Cloud Expertise:
  • Strong understanding of Azure cloud architecture, including storage, compute, and networking.
  • Knowledge of Azure security best practices, such as encryption and key management.

Preferred Skills (Nice to Have):

• Experience with machine learning pipelines and frameworks like MLFlow or Azure Machine Learning.
• Knowledge of data visualization tools such as Power BI for creating dashboards and reports.
• Familiarity with Terraform or ARM templates for infrastructure as code (IaC).
• Exposure to NoSQL databases like Cosmos DB or MongoDB.

Experience with data governance to

Weekly Hours:

Time Type:

Location:

It is the policy of AT&T to provide equal employment opportunity (EEO) to all persons regardless of age, color, national origin, citizenship status, physical or mental disability, race, religion, creed, gender, sex, sexual orientation, gender identity and/or expression, genetic information, marital status, status with regard to public assistance, veteran status, or any other characteristic protected by federal, state or local law. In addition, AT&T will provide reasonable accommodations for qualified individuals with disabilities. AT&T is a fair chance employer and does not initiate a background check until an offer is made.