Learn a better way to adapt.

Optimize what happens after practice

Ordinary supervised learning optimizes performance on supplied examples. Meta-learning trains something about the adaptation process across tasks: an initialization, representation, update rule, or strategy for using a small amount of new evidence.

Each training episode has support examples used for adaptation and query examples used to evaluate the adapted model. Query feedback can train the shared meta-parameters through the outer objective; it must not be quietly included in the task’s inner adaptation updates.

$\min_\theta\ \mathbb E_{T}\left[L_{T,\mathrm{query}}\left(\mathrm{Adapt}(\theta,D_{T,\mathrm{support}})\right)\right]$

Read this as: choose shared parameters that produce low query loss after a limited amount of task-specific practice. Evaluating a genuinely new task requires reserving that task from the outer training loop as well.

Follow a small MAML calculation

Model-Agnostic Meta-Learning (MAML) learns an initialization through the performance of adapted copies. Our one-weight neuron predicts y = wx. Meta-training tasks have target slopes t = 2 and t = 8. Support inputs are −1 and 1; their mean squared input is one.

The support mean squared loss is therefore (w − t)². One inner step with rate α gives:

$w'_t=w-2\alpha(w-t)$

We measure each adapted copy on query inputs −2, −0.5, 0.5, and 2. Their mean squared input is 2.125, so query loss is 2.125(w′ − t)². The outer objective averages that loss over the two tasks. An outer update changes the shared starting weight, not either task’s final copy.

The symmetric objective favors a starting weight of five for suitable rates. Five does not solve either task by itself; it puts both within a useful adaptation distance. At the special rate α = 0.5, one inner step reaches either target exactly regardless of the initialization. The objective is flat and supplies no preference for five. This exception explains why assumptions about the adaptation rule matter.

Optional derivative: differentiate through adaptation

The sensitivity of w′ to w is 1 − 2α. Query loss contributes another factor of w′ − t = (1 − 2α)(w − t). The task’s outer gradient is therefore 2 × 2.125 × (1 − 2α)² × (w − t). Average over tasks, then apply the outer rate 0.1. The implementation checks this gradient numerically.

A reserved task tests the adaptation claim

Task C is excluded from meta-training. Its support examples reveal a slope of 6.5. The lab adapts a zero initialization and the meta-trained initialization with the same step budget, then measures both on different query inputs. The shared saved initialization stays unchanged during this check.

C belongs to the same simple task family. Better results here do not establish adaptation to unrelated domains, noisy observations, different representations, or much larger systems. Repeatedly choosing settings based on this same task also turns it into familiar practice. A larger study would reserve multiple task families and keep its final tests outside selection.

Other adaptation mechanisms

Metric-based meta-learning can learn an embedding space where a few examples define a prototype for a new class. Classification compares a new item with those prototypes. That differs from taking gradient steps on the new task, even if both use a small support set.

In-context learning changes behavior by conditioning on examples in the current input context. Ordinary inference does not update the model’s stored weights. Removing the context removes that particular source of evidence unless some separate memory or learning process retains it.

The source exercise contrasts adding three demonstrations to a prompt with changing model weights. A saved parameter update persists in the saved model; temporary demonstrations do not automatically become weights. A system can combine context, external memory, and parameter learning, so identify what is actually retained.

Learning how to spend learning effort

A meta-learning controller could choose which specialist to adapt, which examples to request, or when retrieval is enough. Such a strategy must be trained and evaluated across relevant tasks. Overfitting the meta-training suite can produce a clever-looking adaptation rule that transfers poorly. The objective is reusable learning behavior, not merely repeated training on one familiar problem.