What Matters for Cross-Modal Alignment

Yipeng Sun1, Jiajun Wang1, Siyuan Mei1, Yan Xia2, Siming Bayer1, Vincent Christlein1, Andreas Maier1

1Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg

2Department of Orthodontics and Orofacial Orthopedics, Friedrich-Alexander-Universität Erlangen-Nürnberg

Friedrich-Alexander-Universität Erlangen-Nürnberg Pattern Recognition Lab

Median distance in the plot

The encoders do not change. Only the map W changes.

CT scan Report Report states the finding ·Hover: findings

Zero pairs leave the scans at random. 200 pairs move them toward their own reports. 600 CT-RATE validation scans. Each point is one feature vector. The two axes are fixed: the report-feature directions that the scans predict best. One random seed.

Abstract

The Platonic Representation Hypothesis posits that models trained on different modalities converge toward a shared representation. Related work aligns frozen image and text encoders on natural images without paired data. We test zero-pair alignment on five datasets across four modalities: natural images, environmental audio, chest radiographs, and 3D computed tomography (CT). It succeeds only on COCO and fails on the other four. We analyze the failures and find that alignment must first locate the subspace where the modalities co-vary and then orient the mapping within it. Consistent with previous studies, medical imaging is the hardest case: its shared subspace carries the smallest fraction of the variance. When this fraction is small, alignment needs image–text pairs, and random-matrix theory predicts how many pairs are needed. These findings motivate LightAlign: a few pairs let a ridge regression between two frozen encoders locate and orient the mapping simultaneously, and a classifier fitted only on report text then predicts findings from the mapped images. On 3D CT, LightAlign surpasses models trained end to end on image–report pairs without finding-level labels: with only 200 pairs, LightAlign outperforms CT-CLIP on CT-RATE (0.754 vs. 0.733 AUROC), and with only 100 CT-RATE pairs, it does so on RAD-ChestCT, an out-of-distribution dataset from a different hospital. On CheXpert, it reaches 0.862 mean AUROC with only about 5 % as many pairs as models trained on MIMIC-CXR.

Two Conditions for Zero-Pair Alignment

1

Locate

Find the subspace where image and text features change together. Without pairs, this works only when the subspace has a large part of the variance. On CT, it has 0.5–10.5 %.

2

Orient

Select the sign of each shared direction. Without pairs, skewed features let the Gromov–Wasserstein loss select the correct sign.

On CT, condition 1 fails without pairs. A random-matrix threshold predicts how many pairs it needs. LightAlign fits a ridge regression on these pairs, which finds and orients the map in one step. A classifier, fitted on report text only, then predicts each finding. The fit takes approximately 1 minute on 4 CPU cores.

Four panels: shared directions on CT-RATE, detection of shared directions against the number of pairs, a direction and its mirror, and the LightAlign pipeline.
The method in four panels. (a) Image and report features agree on the first two shared directions. (b) The shared directions have 8 % of the image variance and 7 % of the report variance. Each direction becomes detectable above its predicted number of pairs. (c) A direction and its mirror. The mirror has the higher GW loss. (d) Ridge regression maps a new scan into the text space. A report classifier scores each finding.

Results

Left: zero-pair alignment error against the share of image variance in the shared directions for five datasets; scaling the shared directions aligns every dataset. Right: mean AUROC on CT-RATE against the number of pairs; LightAlign passes CT-CLIP at 200 pairs.
Zero pairs need a large shared variance. 200 pairs are better than an end-to-end model. (a) Each marker is one dataset, before (open) and after (filled) we multiply the shared directions by 8. Lower FOSCTTM is better. (b) Mean AUROC for 18 findings on 3,002 CT-RATE scans. Dashed line: CT-CLIP, trained on approximately 20,000 patients.

Frozen encoders. CT: VoxelFM and Qwen3.5-9B. Chest radiographs: RAD-DINO and Qwen3.5-4B. Scores: mean AUROC.

0.754vs. 0.733 CT-CLIP

CT-RATE, 200 pairs

0.824vs. 0.816 COLIPRI

CT-RATE, 5,994 pairs

0.636vs. 0.629 CT-CLIP

RAD-ChestCT, 100 CT-RATE pairs

0.862CheXpert

5 % of the MIMIC-CXR pairs

Mean AUROC against the number of pairs on CT-RATE and on CheXpert, with end-to-end models as reference points.
A few pairs are sufficient. (a) CT-RATE, 3,002 scans. Red: LightAlign. Gray: the same encoders with a zero-shot prompt. Diamonds: end-to-end models. (b) CheXpert test set, against models trained on MIMIC-CXR.

Analysis

Zero-pair alignment error against the variance share of the shared directions for each encoder and text model, and the effect of shrinking the shared directions on COCO and Clotho.
Without pairs, alignment works only when the shared subspace has a large part of the variance. (a) Each marker is one encoder–text combination, before and after we multiply the shared directions by 8. (b) On COCO, a decrease of the shared directions to one quarter stops the alignment.
Mean AUROC per seed before and after sign correction, and the Gromov-Wasserstein loss against AUROC for correct and mirrored maps.
Found subspaces are frequently mirrored. The GW loss finds the mirror. (a) Mean AUROC for each seed, before and after sign selection. (b) Mirrored maps have a higher GW loss.
Predicted against observed squared canonical correlations, and a 200-pair pilot score against AUROC for 15 CT encoders.
The threshold predicts the number of pairs. A 200-pair pilot ranks the encoders. (a) Predicted against measured canonical correlations. (b) Pilot score against AUROC for 15 CT encoders.

Examples

Four CT-RATE validation patients with report impressions, the rank of each patient's own report, and the report-classifier scores of 18 findings.
Four CT-RATE validation patients. Left: the report impression and the rank of the own report among 2,025 reports. Right: classifier scores for 18 findings. Orange: findings that the report states.

Use the Released Weights

The weights contain the ridge map from VoxelFM to Qwen3.5-9B features and 18 report classifiers, fitted on 5,994 CT-RATE patients. To score a scan, you need VoxelFM and a CUDA GPU.

Quick start
git clone https://github.com/sypsyp97/lightalign
cd lightalign
pip install -e .
lightalign-predict --ct volume.nii.gz --weights weights/ --voxelfm-code /path/to/VoxelFM

The repository also contains the experiment code and all result files. For research only. The scores are not calibrated probabilities. Do not use them for clinical decisions.

Citation

BibTeX
@article{sun2026whatmatters,
  title   = {What Matters for Cross-Modal Alignment},
  author  = {Sun, Yipeng and Wang, Jiajun and Mei, Siyuan and Xia, Yan and Bayer, Siming and Christlein, Vincent and Maier, Andreas},
  journal = {arXiv preprint arXiv:2611.20000},
  year    = {2026}
}