Finding Spurious Correlations with Function-Semantic Contrast Analysis
In the field of Computer Vision (CV), the degree to which two objects, e.g. two classes, share a common conceptual meaning, known as semantic similarity, is closely linked to the visual resemblance of their physical appearances in the data: entities with higher semantic similarity, typically exhibit greater visual resemblance than entities with lower semantic similarity. Deep Neural Networks (DNNs) employed for classification exploit this visual similarity, incorporating it into the network’s representations (e.g., neurons), resulting in the functional similarity between the learned representations of visually akin classes, often manifesting in correlated activation patterns. However, such functional similarities can also emerge from spurious correlations — undesired auxiliary features that are shared between classes, such as backgrounds or specific artifacts. In this work, we present the Function-Semantic Contrast Analysis (FSCA) method, which identifies potential unintended correlations between network representations by examining the contrast between the functional distance of representations and the knowledge-based semantic distance between the concepts these representations were trained to recognize. While natural discrepancy is expected, our results indicate that these differences often originate from harmful spurious correlations in the data. We validate our approach by examining the presence of spurious correlations in widely-used CV architectures, demonstrating that FSCA offers a scalable solution for discovering previously undiscovered biases, that reduces the need for human supervision and is applicable across various Image Classification problems.
