1041: Split Learning

1. Overview

Split Learning is a distributed machine learning technique that addresses the challenge of training models on decentralized data without compromising privacy. The core problem it solves is enabling multiple parties to collaboratively build a robust machine learning model without sharing their raw data, which is often sensitive or proprietary. This is particularly crucial in fields like healthcare, finance, and any domain where data privacy is a primary concern. By splitting a neural network model into multiple parts—typically a client-side portion and a server-side portion—Split Learning allows for the training to occur across different devices or institutions. The client devices perform the initial computations on their local data, sending only the intermediate results (activations from the ‘cut layer’) to a central server. The server then completes the forward propagation, computes the loss, and initiates the backpropagation, sending gradients back to the clients. This process ensures that the raw data never leaves the client’s premises, thus preserving data privacy.

The concept of Split Learning was introduced by researchers at the MIT Media Lab’s Camera Culture group. It emerged from the growing need for privacy-preserving AI, especially in the context of increasingly stringent data protection regulations like GDPR and HIPAA. Traditional centralized machine learning approaches require amassing large datasets in a single location, creating significant privacy risks and logistical hurdles. Split Learning, along with its counterpart Federated Learning, offers a paradigm shift towards decentralized AI. For organizations, this pattern is highly valuable as it unlocks the potential of collaborative data analysis that was previously impossible due to privacy constraints. It allows for the creation of more accurate and generalized models by leveraging diverse datasets from multiple sources. For a commons-based approach, Split Learning fosters a collaborative ecosystem where organizations can pool their computational resources and model insights without sacrificing control over their data, leading to shared advancements and collective benefit.

2. Core Principles

1. Data Minimization and Privacy: Only intermediate outputs and gradients from the split-point of the model are exchanged, not the raw data itself. This is the foundational principle that ensures the privacy of the data subjects.
2. Model Splitting: The neural network is partitioned into a client-side model and a server-side model. The client-side model is typically smaller and resides on the device where the data is located, while the server-side model is hosted on a more powerful machine.
3. Collaborative Training: Multiple clients can participate in the training process, contributing their data to a shared model without directly exposing it. This collaborative nature allows for the creation of more robust and generalized models.
4. Decentralized Computation: A significant portion of the computation, especially the initial layers of the neural network, is performed on the client devices. This distributes the computational load and can reduce the burden on the central server.
5. Secure Gradient Exchange: The gradients are sent back from the server to the client to update the client-side model. This process is designed to be secure, often incorporating additional privacy-enhancing techniques to prevent information leakage.

3. Key Practices

1. Define the Cut Layer: Carefully select the layer at which the model is split. This decision impacts privacy, communication overhead, and computational load on both the client and server.
2. Manage Communication Overhead: While Split Learning can be more communication-efficient than Federated Learning in some scenarios, it’s crucial to manage the size of the transferred tensors (activations and gradients) to avoid bottlenecks.
3. Handle Data Heterogeneity: In real-world scenarios, data distribution across clients can be non-IID (not independent and identically distributed). Employ strategies to mitigate the impact of data heterogeneity on model performance.
4. Secure the Server: The central server is a critical component of the architecture. Ensure its security to prevent attacks that could compromise the model or infer information about the client data.
5. Consider Hybrid Approaches: Combine Split Learning with other privacy-preserving techniques like Federated Learning (SplitFed) or differential privacy to achieve stronger privacy guarantees.
6. Monitor for Information Leakage: Although Split Learning is designed to be private, it’s not immune to sophisticated attacks. Monitor for potential information leakage from the exchanged activations and gradients.

4. Implementation

Implementing Split Learning involves a series of steps, starting with the design of the neural network architecture and the decision of where to split it. A typical implementation would follow these steps: 1) The model is divided into a client-side part and a server-side part. 2) The client loads its local data and performs a forward pass through its part of the model. 3) The resulting activations from the cut layer are sent to the server. 4) The server takes these activations and completes the forward pass through its part of the model, calculates the loss, and starts the backpropagation. 5) The server sends the gradients back to the client. 6) The client uses these gradients to update its part of the model. This process is repeated for multiple epochs and batches of data until the model converges.

Key considerations during implementation include the choice of the cut layer, which is a trade-off between privacy and utility, and the management of the communication between the clients and the server. Frameworks like PySyft, part of the OpenMined ecosystem, provide tools and abstractions to facilitate the implementation of Split Learning. TensorFlow and PyTorch can also be used to build Split Learning systems from scratch. Success metrics for a Split Learning implementation include not only the final model accuracy but also the privacy guarantees it provides, the communication and computation costs, and the scalability of the system to a large number of clients.

5. 7 Pillars Assessment

6. When to Use

• When multiple parties want to train a model on their combined data, but are unable to share the raw data due to privacy concerns or regulations.
• In healthcare, for training models on patient data from different hospitals without violating patient privacy.
• In finance, for training fraud detection models on transactional data from different banks.
• When the client devices have limited computational resources, as the majority of the computation can be offloaded to the server.
• In edge computing scenarios, where data is generated at the edge and needs to be processed locally for privacy and efficiency.

7. Anti-Patterns & Gotchas

• Ignoring the threat of a malicious server: A malicious server can still try to infer information about the client data from the received activations. Additional privacy-enhancing techniques may be needed to mitigate this risk.
• Choosing a shallow cut layer: A cut layer that is too close to the input layer can leak more information about the raw data.
• Underestimating communication costs: While Split Learning can be communication-efficient, the transfer of large activation tensors can still be a bottleneck, especially in low-bandwidth environments.
• Neglecting the impact of data heterogeneity: Non-IID data can lead to biased models. It’s important to use techniques to address this issue.
• Failing to establish clear governance: The lack of clear rules and agreements between the participants can lead to conflicts and undermine the collaboration.

8. References

1. Vepakomma, P., Gupta, O., Swedish, T., & Raskar, R. (2018). Split learning for health: Distributed deep learning without sharing raw patient data. ArXiv. https://arxiv.org/abs/1812.00564
2. Gupta, O., & Raskar, R. (2018). Distributed learning of deep neural network over multiple agents. Journal of Network and Computer Applications, 116, 1-8. https://www.sciencedirect.com/science/article/abs/pii/S108480451830093X
3. Nguyen, D. M. A. (2023, April 11). A Gentle Introduction to Split Learning. Medium. https://medium.com/@minhanh.dongnguyen/a-gentle-introduction-on-split-learning-959cfe513903
4. Thapa, C., et al. (2020). SplitFed: When Federated Learning Meets Split Learning. ArXiv. https://arxiv.org/abs/2004.12088
5. Hao, K. (2019, October 24). The Algorithm: The privacy-preserving AI technique that will transform healthcare. MIT Technology Review. https://www.technologyreview.com/2019/10/24/132105/the-algorithm-the-privacy-preserving-ai-technique-that-will-transform-healthcare/