SOCiaL — Beyond Disjoint Tasks: Natural Continual Learning for VLMs

Abstract

Continual learning for vision-language models is usually studied on benchmarks where each task introduces entirely new domain knowledge. Real task streams are more natural: they routinely share visual concepts, language patterns, and even individual training samples.

Under such overlap, mixture-of-expert methods that assign one expert per task with fixed routing split similar inputs across experts and lose accuracy. We introduce SOCiaL (Semantic Overlap-aware Continual Learning), a framework built around a single lightweight Gaussian mixture model (GMM) per task that serves triple duty: it measures directional semantic overlap to decide when to merge experts, generates synthetic replay to train the router without storing raw data, and merges in closed form as experts are consolidated. Overlapping LoRA experts are merged in full weight space to avoid cross-term artifacts, and an elastic contextual-bandit router adapts as the expert pool grows and shrinks. To study this setting we release UCIT-O, a 10-task benchmark with three protocols of increasing overlap.

What's new

Natural continual learning

Formalizes the realistic setting where sequential tasks share content instead of being artificially disjoint.

One GMM, three roles

A per-task Gaussian mixture detects overlap, drives closed-form expert merging, and generates replay for the router.

Interference-free merging

Merging LoRA experts in full weight space removes the cross-terms that low-rank factor averaging introduces.

Elastic bandit router

A contextual bandit selects experts and adapts as the pool expands with new tasks or contracts after merges.

UCIT-O benchmark

Ten tasks across three controlled overlap protocols, released openly for the community.

Fewer adapters

Consolidating overlapping tasks reduces deployed adapters from 10 to 3 with no loss in accuracy.

Method

The SOCiaL framework

Each task gets a dedicated LoRA expert. A single GMM per task, fit on frozen vision+text embeddings, then drives three decisions, organized around when, how, and where:

When to merge?

GMM overlap detection

Asymmetric density coverage scores between task GMMs measure directional semantic overlap. Mutual coverage above a threshold ρ triggers a merge; GMMs combine in closed form.

How to merge?

Interference-free LoRA merge

Overlapping experts are averaged in full weight space (ΔW = BA), avoiding cross-terms, compressed back via rank-r SVD, then briefly fine-tuned anchored to the merge.

Where to route?

Elastic contextual bandit

A bandit policy selects experts, trained on real embeddings plus GMM replay with entropy regularization and a learned baseline. Its action space grows and shrinks with the pool.

Coverage — fraction of task t's embeddings that exceed task k's GMM density floor, giving a directional overlap score.

Δ-space merge — summing per-layer ΔW_i = B_iA_i is free of the quadratic cross-terms that corrupt factor-level averaging.

Generative replay — GMMs synthesize embeddings for all past tasks, so the router never stores raw data.

UCIT-O

UCIT-O arranges six vision-language datasets (ImageNet-R, ArxivQA, VizWiz, CLEVR-Math, IconQA, Flickr30k) into ten sequential tasks under three protocols of increasing semantic overlap.

Protocol	Name	Overlap	Description
P1	Selective Overlap	Partial	Only some tasks share content through partial sample reuse.
P2	Universal Overlap	Systematic	Every task shares data partitions with others; no identical samples.
P3	Sample-Level Overlap	Duplication	Identical samples appear across tasks under different labels.

10 sequential tasks2 classification3 captioning5 VQAfixed seed = 42

Protocols and generation code live in /data. Instruction JSONs reference images by relative path; source images keep their original dataset licenses and are not redistributed.

State of the art on disjoint and overlapping streams

Last accuracy (%) after learning all tasks sequentially with LLaVA-1.5-7B. SOCiaL improves on the disjoint benchmarks and on every UCIT-O protocol, with the gap widening as overlap increases.

Method	CoIN	UCIT	UCIT-O P1	UCIT-O P2	UCIT-O P3
Best prior baseline	62.82	64.19	61.43	65.44	59.53
SOCiaL (ours)	70.17+7.35	72.23+8.04	64.50+3.07	70.18+4.74	69.30+9.77

Best prior baseline = HiDe-LLaVA on CoIN/UCIT, EWC on UCIT-O. Over three seeds, SOCiaL is stable: CoIN 70.05±0.13, UCIT 71.53±0.61, UCIT-O P1 65.11±0.97. Merging reduces deployed adapters from 10 to 3.

BibTeX

@inproceedings{xu2026beyond,
  title     = {Beyond Disjoint Tasks: Towards More Natural Continual Learning
               for Vision-Language Models},
  author    = {Xu, Xiang and Su, Yiyang and Zhao, Tianchen and Zhang, Zheng
               and Tu, Zhuowen and Jain, Anil and Wu, Jonathan},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year      = {2026}
}