Learn How to Query from Unlabeled Data Streams in Federated Learning.

1. The Problem The authors start from a realistic problem. Most Federated Learning research assumes that clients already have a labeled dataset. But what happens when data is continuously arriving at the client without labels? For example: Hospital ↓ Patient image arrives ↓ No label yet ↓ Doctor / expert must annotate it ↓ Then it can be used for training Imagine a client continuously receiving new data. The problem is that getting an expert to label every single sample is expensive. So you can't simply label everything. This creates the question: Which incoming samples should the client spend its labeling budget on? This is the data-querying problem that the paper is trying to solve. Now, a deeper problem arises. Suppose Client 1 receives: A, B, C, D, E But it only has enough budget to label 2 samples. The client might think: "A and B are the most useful for my local model." That sounds reasonable. But there is a problem. Federated Learning is not trying to optimize only Client 1's model. It is trying to optimize the global model across all clients. So the samples that are most useful from Client 1's local perspective might not necessarily be the samples that are most useful for the global training objective. The authors describe this conflict between the local data-access perspective and the global training objective as a fundamental challenge. And this is the key problem that LeaDQ tries to attack. SERVER ↓ Global Model ↓ ┌───────────┴───────────┐ ↓ ↓ Hospital 1 Hospital K ↓ ↓ Unlabeled images Unlabeled images ↓ ↓ Query policy Query policy ↓ ↓ Select samples Select samples ↓ ↓ Doctor / expert Doctor / expert labels labels ↓ ↓ Local dataset Local dataset ↓ ↓ Local training Local training └───────────┬───────────┘ ↓ SERVER ↓ Global Model The overall process has two main phases. Unlabeled data ↓ Selected Samples ↓ Obtain labels Clients receive incoming unlabeled data. The querying policy decides which samples are worth sending to an oracle for labeling. So the overall idea is basically: Unlabeled data ↓ Selected Sample ↓ Get labels ↓ FedAvg ↓ New Global model ↓ Repeat The selected samples are labeled and then used for federated training. After training, more unlabeled data arrives and the process repeats. This is where LeaDQ becomes a reinforcement learning problem. For each incoming sample, the policy makes a binary decision. 1 = query this sample 0 = don't query this sample The decision is basically: Select the sample? Yes → query the oracle for its label No → don't query it So if 5 samples arrive: x1 x2 x3 x4 x5 The policy might make decisions like: 1 0 1 0 0 and then: x1 and x3 Are then sent to the oracle for labeling. But there is an important constraint. The client only has a limited querying budget. So: Number of selected samples = Nq The policy therefore has to decide which samples are worth spending that limited budget on. The policy needs some information before deciding whether to select an incoming sample. The observation it receives includes the model's predictive logits. Compute Predicitive logits ↓ local Observation ↓ Query Policy ↓ Q-Values ↓ Choose Samples So the policy gets information about: "How does the current model respond to this incoming data?" For example, if the model is very uncertain about a sample, that information can potentially influence the querying decision. The important thing here is that the policy isn't simply looking at the raw sample and randomly deciding whether to select it. It is using information from the current model's response to the incoming data. This is probably the part that made the most sense to me once I understood the problem. The client doesn't immediately know: "Was choosing this sample a good decision?" For example: Client Selects:[]( ) Sample X ↓ Get labels ↓ Trains ↓ Global model Change ↓ Global Accuracy Improves Suppose the client chooses sample A. At that exact moment, it doesn't know whether A will actually help the global model. The consequence of that decision might only become visible after the sample is labeled and used during federated training. So there is a delay between: Decision → Training → Global model performance That is exactly the kind of problem reinforcement learning can model. The querying policy takes an action, and later receives information about the consequences of that action. The authors therefore formulate the querying problem as a: Decentralized Partially Observable Markov Decision Process (Dec-POMDP). This part is important. Each client only sees its own information. Client 1 doesn't know exactly what data Client 2 has. Client 2 doesn't know exactly what Client 3 has. And so on. But the objective they're ultimately trying to optimize depends on the global federated learning system. So each client only has a partial view of the overall environment. Hence: Decentralized + Partially Observable → Dec-POMDP Decentralized + Partially Observable + Multiple Decision-Makers Now we have another problem. There isn't just one client making decisions. There are multiple clients making querying decisions. So the authors use: Multi-Agent Reinforcement Learning (MARL). Each client can be treated as an agent. Client 1 → Policy y1 Client 2 → Policy y2 Client K → Policy yk The policies make their decisions locally. So Client 1 has its own policy. Client 2 has its own policy. Client 3 has its own policy. And they are all making decisions based on their own local observations. This is where Centralized Training with Decentralized Execution (CTDE) comes in. The basic idea is: The system can use global information to help train the policies. Each client has to make decisions using only its local information. So: Training can be centralized, but execution remains decentralized. This makes sense for Federated Learning because we don't want every client to require access to all the other clients' private data just to make a querying decision. The authors adapt QMIX for the multi-agent policy training. Each client has its own local Q-function: Q₁, Q₂, Q₃, ..., Qₖ These local Q-functions represent the value of actions from the perspective of each individual agent. But the overall goal is still the global objective. QMIX is used to combine these individual value functions into a global value function. Individual client decisions ↓ Individual Q-functions ↓ QMIX ↓ Global Q-value The important idea is that the clients can learn policies that make local decisions while still being trained toward a shared global objective. What I found interesting is that the paper doesn't treat data selection as simply: "Which samples are the most useful to this client?" Instead, it asks a much harder question: "Which samples should this client query so that they ultimately help the global Federated Learning objective?" That difference is important. A sample can be very useful for one client's local model but not necessarily useful for the overall federated system. LeaDQ tries to bridge that gap by treating the querying process as a multi-agent decision-making problem. The approach is interesting, but it also introduces quite a lot of complexity. Instead of simply running Federated Learning, we now have: Data querying → Reinforcement Learning → Multi-Agent Reinforcement Learning → Dec-POMDP → CTDE → QMIX → Federated Training That is a lot of machinery around the original problem of deciding which data to label. There is also the practical question of how well these policies can be trained and how much additional computation and coordination the approach requires. So while the idea is interesting, implementing something like LeaDQ is definitely more complicated than simply running a standard FedAvg system. The biggest thing I took away from this paper is that data selection in Federated Learning is not just a local optimization problem. When data is unlabeled and labeling is expensive, the client has to decide where to spend its limited labeling budget. But the client's decision should ultimately help the global model, not just its own local model. LeaDQ approaches this by turning the data-querying problem into a multi-agent reinforcement learning problem, where clients make decisions locally while being trained toward a global objective. And that was my main takeaway from Paper 5. There is a lot more technical detail in the paper, especially around the QMIX formulation and how the querying policy is trained, but this was my attempt to understand the core idea behind the work before diving deeper into the implementation. Paper: Learn How to Query from Unlabeled Data Streams in Federated Learning Link: https://arxiv.org/html/2412.08138v2