Where Not to Learn: Prior-Aligned Training with Subset-based Attribution Constraints

Preprint arXiv:2602.07008 2026

Where Not to Learn

Prior-Aligned Training with Subset-based Attribution Constraints

Correct predictions are not enough. We train models to shift their attributed evidence toward specified human priors—without suppressing every piece of surrounding context.

Ruoyu Chen¹, Shangquan Sun², Xiaoqing Guo³, Kangwei Liu¹, Sanyi Zhang⁴, Zhangcheng Wang⁵, Shiming Liu⁶, Qunli Zhang⁶, Wei Wang¹, Hua Zhang¹, Xiaochun Cao¹

¹University of Chinese Academy of Sciences ²Nanyang Technological University ³Hong Kong Baptist University ⁴Communication University of China ⁵ByteDance ⁶Imperial College London

Read the paperPDF ↗ ClassificationCode ↗ GUI AgentCode ↗

01 / Motivation

A model can arrive at the right answer while attending to the wrong evidence. Human priors turn this hidden mismatch into a trainable signal.

The problem

Right answer.
Wrong reason.

Ordinary supervision constrains the output. It rarely specifies which evidence is acceptable for producing it.

Abstract

Reliable models should not only predict correctly, but also base their decisions on acceptable evidence. Yet class-level supervision can reward shortcuts that never touch the intended object or interface element.

We periodically expose compact decision-supporting regions with subset-selection attribution, compare them with human-provided masks or boxes, and penalize influential evidence that falls off-prior.

Across image classification and MLLM-based GUI clicking, the resulting models improve observed task performance and attribution–prior alignment.

Outputs under-specify evidence

Two models may make the same prediction while relying on very different regions.

Human priors are weak signals

Masks and target boxes guide acceptable evidence, but are not treated as causal ground truth.

Alignment should be selective

We target influential off-prior regions rather than erasing all contextual information.

The method

Constrain the evidence,
not the architecture.

One alignment principle spans discriminative classifiers and autoregressive GUI agents: identify compact supporting subsets, then act only on evidence that violates the supplied prior.

Input

Predict normally

Optimize the standard task objective on every step.

Rank

Expose evidence

LIMA or EAGLE ranks compact input regions by decision influence.

Gate

Compare with prior

Overlap gates isolate top-ranked and higher-order off-prior regions.

Align

Shift attribution

Selective losses discourage off-prior utility and marginal gains.

Top-ranked evidence

Deviation loss

When the most influential region lies outside the human prior, suppress its singleton utility. If it is prior-consistent, this term stays off.

L_dev = F({v_π1}) · 𝟙[off-prior]

Higher-order evidence

Redundancy loss

For every later off-prior region, penalize only its positive marginal gain. This gate operates independently of the top-ranked region.

L_red = Σ ReLU(Δ_r) · 𝟙[off-prior]

One principle, two prediction settings

Classification

LIMA

Target-class confidence under retained and complementary image subsets.

Code ↗

GUI agent

EAGLE

Token-level insight and necessity across the thinking–action sequence.

Code ↗

Results at a glance

Better decisions,
better-aligned attribution.

6 / 6classification settings improve clean Top-1 accuracy over fine-tuning

68.64%average clean accuracy across six matched-supervision comparisons

89.23%GUI click success with attribution-constrained SFT

0.8615GUI Point Game, the best result among evaluated methods

Classification values are reported over three independent runs unless noted. The 936-task GUI study uses one training run per method and should be read as a preliminary proof of concept.

Track 01

Image classification

Attribution moves toward the object.

Across Saliency-Bench and ImageNet-S, our method improves clean accuracy for CLIP, ViT, and ResNet-101 while increasing LIMA-based Point Game in all reported comparisons.

Classification repository ↗

Classification attribution comparison across fine-tuning, RRR, XIL, MEGL, and our method — Figure 02 LIMA attribution maps on Saliency-Bench and ImageNet-S. White contours mark the supplied target-object priors.

Clean performance

Fine-tuning → Ours

Accuracy and attribution–prior Point Game

Dataset	Backbone	Top-1 accuracy	Point Game
Saliency-Bench	CLIP	61.61 → 65.43	0.4567 → 0.4733
	ViT (base)	53.82 → 58.06	0.4933 → 0.5031
	ResNet-101	41.16 → 55.90	0.4533 → 0.5600
ImageNet-S	CLIP	79.05 → 79.84	0.7633 → 0.8233
	ViT (base)	79.05 → 79.44	0.8700 → 0.8767
	ResNet-101	70.30 → 73.14	0.8633 → 0.8833

Values shown as percentages for Top-1 accuracy. Full mean ± standard deviation results and all baselines are in the paper.

5 / 6

independent evaluator combinations improve over fine-tuning using RISE or gradient-based attribution.

+11.01

point gain for ResNet Grad-CAM: 37.90% → 48.91%.

+5.67

point gain for ResNet RISE: 40.33% → 46.00%.

Track 02

MLLM-based GUI agent

The click and its evidence move together.

On 936 single-step Android tasks, attribution-constrained SFT produces the highest click success and the highest target-box Point Game among the evaluated methods.

GUI agent repository ↗

AgentCPM-GUI · single run

Task performance & attribution alignment

Method	Click success ↑	Distance error ↓	Point Game ↑
SFT (LoRA)	84.61%	94.71	0.8153
Box-Guided Aug. SFT	86.15%	85.26	0.8462
Task-Reward GRPO	87.69%	78.42	0.7692
Ours (LoRA)	89.23%	78.64	0.8615

GRPO has a marginally lower distance error by 0.22; our method leads click success and Point Game. These descriptive single-run results do not establish deployment reliability.

GUI task examples comparing SFT and our attribution-constrained method — Figure 03 Representative clicks, attribution maps, and target UI boxes.

Correct GUI clicks with different attribution patterns under SFT and our method — Figure 04 Both agents click correctly; the attributed evidence is not the same.

Failure analysisOpen case study +

Better localization can still end in the wrong action.

Neither model outputs a POINT in this example. Our method emphasizes the intended follow control more clearly, but still returns a STATUS. Attribution–prior overlap is evidence of localization consistency—not proof of causal grounding or safety.

Training behavior

A selective constraint needs a measured dose.

Applying attribution alignment too frequently can disrupt the main task, while an overly large deviation weight can dominate standard supervision. The method therefore uses intermittent, confidence-gated updates.

Classification alignment: every 10 steps
GUI alignment: every 5 steps
Applied only to confidently correct predictions

Sensitivity of accuracy to attribution interval and deviation loss weight — Figure 05 Accuracy sensitivity to alignment interval and deviation-loss weight.

Resources

Paper, code,
and citation.

Preprint

arXiv paper

Abstract, PDF, source, and version history.

↗ 02

Repository

Classification code

Prior-aligned image classification experiments.

↗ 03