Description
Data Engineering with Azure PySpark Python SQL
- Role focus: Data engineering on Azure with PySpark centers on building scalable ETL/ELT pipelines, preparing reliable datasets for analytics and ML.
- Core platform: Commonly implemented on Azure Databricks or Synapse Spark pools to run PySpark workloads with managed clusters.
- Primary APIs: Use PySpark DataFrame and SQL APIs for expressive, distributed transformations and aggregations.
- Storage patterns: Implement lakehouse patterns (bronze/silver/gold medallion layers) using Delta Lake or parquet on ADLS for reliable versioning and ACID semantics.
- Ingestion: Support batch and streaming ingestion from sources like Event Hubs, Kafka, blob storage, and relational databases with connectors and structured streaming.
- Transformations: Combine SQL, PySpark transformations, and UDFs to implement joins, windowing, aggregations, and complex business logic at scale.
- Performance tuning: Optimize with partitioning, predicate pushdown, broadcast joins, caching, and choosing appropriate cluster sizing and instance types.
- Incremental processing: Use watermarking, CDC patterns, and incremental pipelines to minimize recomputation and support near‑real‑time updates.
- Testing and CI/CD: Integrate notebooks and jobs with Git, unit tests, and Azure DevOps or GitHub Actions to automate deployments and promote artifacts across environments.
- Observability: Implement logging, job metrics, and lineage tracking to monitor job health, troubleshoot failures, and measure SLAs.
- Security and governance: Enforce RBAC, workspace isolation, managed identities, and data encryption to meet enterprise compliance and access controls.
- Feature engineering: Produce ML‑ready feature tables using PySpark pipelines and register or serve features for model training and scoring.
- Advanced patterns: Architect medallion lakehouse, implement multi‑tenant workspaces, and design cost‑aware autoscaling and spot instance strategies.
- Streaming analytics: Build low‑latency pipelines with structured streaming, stateful processing, and windowed aggregations for event‑driven use cases.
- Interoperability: Combine PySpark with native SQL, Python libraries, and REST APIs to integrate with Azure services (Data Factory, Synapse, ML services).
- Skill expectations (3–7 years): Deliver robust ETL jobs, write PySpark and SQL transformations, tune jobs, and operate Databricks/Spark clusters.
- Skill expectations (8–20 years): Lead architecture for lakehouse design, CI/CD, governance, cost optimization, cross‑team MLOps, and platform reliability.
