Source-free domain adaptation (SFDA) adapts a source-trained model to unlabeled target domains without access to the source data. Two important failure modes remain: attention drift, where learned attention shifts toward spurious or background cues under distribution shift, and domain-gap-agnostic initialization, where the same source model is used irrespective of the target-domain shift.
We propose DGA-CAFC, which combines Counterfactual Attention-based Feature Consistency (CAFC) and Domain-Gap Aware Source Model Selection (DGASS). CAFC stabilizes attention-conditioned features using counterfactual-inspired normalized attention perturbations, while DGASS uses prediction entropy on unlabeled target data to select an appropriate source-model variant before adaptation.
CAFC is also applicable as a lightweight regularizer for domain generalization (DG), where target-domain data are entirely unavailable during training.