In many enterprise environments, critical business data is often stored in SharePoint Online, whether in document libraries, structured lists, or exported Excel files. To enable analytics, reporting, or AI use cases, this data must be reliably ingested into a centralized Azure Data Platform.

This pattern demonstrates how to securely integrate and ingest data from SharePoint into Azure Data Lake Gen2, using Azure Databricks as the processing engine. The approach follows best practices in security, scalability, and observability, while promoting reusability across multiple projects

You can choose this pattern if:

• Your source data resides in SharePoint Online (documents, lists, Excel, CSV…).
• You need to automate ingestion using Databricks notebooks or metadata-driven pipelines.
• The target is a structured Azure Data Lake (Bronze/Silver/Gold layers).
• You require a secure, compliant, and reusable ingestion approach.
• Ingestion is batch-based or scheduled, not real-time.
• You want to parameterize ingestion per SharePoint site/library, without changing the core logic.
• Your organization enforces DevOps, IAM, and data encryption standards.
• You need integration with a VNet-isolated environment for compliance and performance.

This ingestion pattern is built around a VNet-injected Azure Databricks workspace. The workspace is hosted in a dedicated subnet, which allows us to tightly control outbound and inbound traffic.

From this subnet, Databricks accesses Azure Key Vault and Azure Data Lake Gen2 using Service Endpoints. This ensures that all traffic stays within the Azure backbone network, avoiding exposure to the public internet.

Secrets (client ID and secret for SharePoint access) are stored securely in Key Vault and retrieved during notebook execution. The actual data ingestion logic is implemented in Databricks notebooks, which interact with Microsoft Graph API to extract data from SharePoint and write it into the lakehouse storage layer.

If you’re interested in going deeper into the network isolation and configuration strategies (e.g. private endpoints, firewalls, route tables), feel free to drop a comment at the end of the article — I’ll be happy to explore that in a follow-up post 😀

Components and Roles

Component	Role
SharePoint Online	Acts as the business-facing data source (files, lists, folders).
Azure Key Vault	Stores secrets (SPN credentials) and encryption keys securely.
Azure Databricks	Hosts the ingestion logic as notebooks; orchestrates the process.
Azure Storage (Data Lake Gen2)	Target destination; stores structured ingested data for analytics.

Security

In any data integration flow—especially those dealing with external APIs, secrets, and sensitive business content—security posture is a core pillar. In this pattern, the focus has been on minimizing exposure, enforcing least-privilege access, and protecting data both in transit and at rest.

Identity & Access Management

To begin with, we use an Azure App Registration (SPN) to authenticate against Microsoft Graph API. Its credentials are never hardcoded—they are stored in Azure Key Vault, which is accessed securely from within the Databricks notebook using Managed Identity.

All access to Azure services like Storage and Key Vault is governed via RBAC (Role-Based Access Control). This ensures that each component can only perform the operations it strictly needs.

Scope	Resource	Role/Permission
Application	SharePoint	Sites.Selected (via SPN)
Application	Storage Account	Contributor (via MSI)
Application	Key Vault	GET access policy (via MSI)

Encryption

Finally, encryption is enforced across all channels and storage layers, using Azure-native encryptionmechanisms (including optional Customer Managed Keys when needed). Secrets in Key Vault are always accessed via secure APIs, and never exposed in plaintext.

• In Transit: All communication is secured over HTTPS.
• At Rest: Data in Data Lake is encrypted using Microsoft-managed keys or CMK if required. Secrets in Key Vault are encrypted with HSM and never exposed in plain text.

Availability

The availability strategy for this pattern depends heavily on how critical the data is to the business. In some cases, ingestion may be a background job that tolerates delay. But in the context of business-critical analytics—where dashboards or AI pipelines depend on timely data—availability becomes a must-have.

In this example, I assume the ingestion pipeline supports business-critical workloads. Therefore, I’ve selected Azure services with strong SLAs and high availability features:

• SharePoint Online: Backed by a 99.9% SLA by Microsoft 365.
• Azure Databricks (Premium Tier): 99.95% SLA when deployed in a VNet-injected workspace.
• Azure Storage Account (RA-GRS): 99.9% for read/write operations with geo-replication.
• Azure Key Vault: 99.9% SLA for secrets access and durability.

These guarantees ensure that the ingestion can run reliably, resiliently, and can recover quickly in the event of partial outages.

These guarantees ensure that ingestion can run reliably and recover gracefully in case of failures.

Scalability

A good ingestion pattern should not break when the business scales—or when the number of SharePoint sites and lists starts growing. That’s why this architecture supports multiple scaling strategies, both technical and functional.

The use of Databricks clusters with autoscaling allows the compute environment to dynamically adjust to workload volume. More data? More parallel notebooks? No problem.

On top of that, the ingestion logic is designed to be metadata-driven. That means new SharePoint sources can be onboarded by simply modifying a configuration file, without changing any code. It also supports parallel execution of ingestion jobs from different sites or lists.

Here’s a summary of how this pattern scales:

•  Parallel Execution across multiple SharePoint sites/libraries.
•  Cluster Autoscaling in Databricks for compute flexibility.
•  Metadata-driven logic to ingest new sources easily.
•  Scheduled or triggered execution via Databricks Jobs, ADF, or Logic Apps.
•  Reusable framework across multiple teams or projects.

Network

Network architecture can take many different forms depending on the context—sometimes requiring private connectivity, DNS resolution, firewalls, and complex routing. In this pattern, the focus has been on a secure but simple approach using Service Endpoints.

Databricks is deployed in a VNet-injected subnet. This subnet is authorized at the Storage Accountand Key Vault level to allow internal Azure traffic via Service Endpoints—effectively routing all traffic over the Azure backbone.

Access to SharePoint is public by nature (it’s SaaS), but all communication happens over HTTPS with strong token-based authentication. In other use cases, private connections via Azure Private Link or S2S VPN could be considered—but that’s a topic I’ll cover in another article.

In summary:

•  Service Endpoints are used for secure, internal Azure traffic.
•  SharePoint is accessed publicly, but securely.
•  VNet injection is used to isolate Databricks and enforce outbound control.

==> No component is exposed to the public internet beyond SharePoint’s API itself.

Observability

Observability is key to ensure ingestion pipelines don’t fail silently. In this pattern, telemetry is collected at multiple layers to offer a 360° view of pipeline health and performance:

• Azure Monitor and Log Analytics track resource-level metrics and logs.
• Application Insights can be integrated into notebooks for custom metrics and exception tracking.
• Databricks logging provides job-level visibility and history.
• Grafana dashboards can aggregate metrics across runs and sites.
• Alerts can be configured on failure, delay, or data completeness thresholds.

With this level of observability, operations teams and data engineers are empowered to react quickly, understand root causes, and improve the reliability of the ingestion process over time.