One Swallow Does Not Make a Summer: Understanding Semantic Structures in Embedding Spaces

To address these challenges, we introduce the Semantic Field Subspaces (SFS), a low-dimensional subspace spanned by semantically related vectors. This geometric representation naturally absorbs expression variants via linear dependence and provides a compact, geometry-preserving abstraction of semantic content.

We compute the basis of ${\mathbb{S}}_{{\mathcal{F}}}$ via SVD on matrix ${\bm{M}}_{{\mathcal{F}}}=[{\bm{v}}_{1},\cdots,{\bm{v}}_{n}]$, i.e., ${\bm{M}}_{{\mathcal{F}}}={\bm{U}}{\bm{\Sigma}}{\bm{V}}^{\top}$. This yields a continuous, low-rank representation that captures semantic structure while remaining invariant to redundancy.

To delineate boundaries between Semantic Fields, we leverage the hierarchical nature of language semantics, where expanding fields lead to broader, more abstract concepts.

As a Semantic Field expands, a significant shift in meaning often indicates an evolution to a new field. We quantify this evolution via the Semantic Shift between two subspaces ${\mathbb{S}}_{{\mathcal{F}}_{x}}$ and ${\mathbb{S}}_{{\mathcal{F}}_{new}}$:

Semantic Shift acts as a boundary criterion: A large shift suggests that the new subspace ${\mathbb{S}}_{{\mathcal{F}}_{new}}$ represents a more abstract concept that subsumes ${\mathbb{S}}_{{\mathcal{F}}_{x}}$, whereas small values reflect refinements within the same semantic field.

Building on Definitions 4.1 and 4.2, we propose SAFARI, an algorithm to uncover SFSes by monitoring Semantic Shifts during iterative clustering. At each step, SAFARI merges the nearest clusters, resulting in a new subspace. The algorithm then evaluates the Semantic Shift between the new subspace and the previous subspace and checks whether such a shift is significant. If so, the new subspace is identified as a new SFS that subsumes the previous subspaces.

The pseudo-code is provided in Algorithm 1. SAFARI initializes by assigning each vector as a singleton cluster in a set $\Omega$, and maintains a set $\Phi$ to store the discovered SFSes. It proceeds iteratively with the steps below until only one cluster remains ($|\Omega|\leq 1$):

• •

  Step 1: Cluster Merging. The two nearest clusters ${\mathcal{C}}_{x}$ and ${\mathcal{C}}_{y}$ are identified using Semantic Distance $d_{sem}({\mathcal{C}}_{x},{\mathcal{C}}_{y})$, with centroids representing each cluster. They are then merged into a new cluster ${\mathcal{C}}_{new}$, after which ${\mathcal{C}}_{x}$ and ${\mathcal{C}}_{y}$ are removed, and ${\mathcal{C}}_{new}$ is added to $\Omega$.

• •

  Step 2: SFS Delineation. SAFARI constructs the subspaces ${\mathbb{S}}_{new}$ and ${\mathbb{S}}_{x}$ for ${\mathcal{C}}_{new}$ and the larger cluster ${\mathcal{C}}_{x}$, and computes Semantic Shift $\Delta F_{sem}({\mathbb{S}}_{x},{\mathbb{S}}_{new})$. A sliding window of size $w$ tracks the recent $w$ values, computing mean $\mu$ and standard deviation $\tau$. If the current $\Delta F_{sem}({\mathbb{S}}_{x},{\mathbb{S}}_{new})$ exceeds the dynamic threshold ($\mu+3\tau$), ${\mathbb{S}}_{new}$ is added to $\Phi$ as a new SFS.

The use of a sliding window ($w$) and the dynamic threshold ($\mu+3\tau$), which is obtained via parameter study, is essential to accurately identify SFSes because we empirically observe that the baseline values of Semantic Shifts grow gradually as the algorithm proceeds. The dynamic threshold allows SAFARI to adapt to such gradual change and effectively detect local spikes of real significance. Moreover, SAFARI adopts the hierarchical process to reflect the layered nature of semantic relationships: (1) It dynamically determines SFSes based on semantic structures, avoiding relying on pre-defined cluster counts; (2) It reveals natural semantic hierarchies through the resulting dendrogram.

Let ${\bm{A}}_{new}=[{\bm{A}}_{x}|{\bm{A}}_{y}]$ be the matrix representing the newly merged cluster ${\mathcal{C}}_{new}$. We now justify the approximation by establishing two key theoretical results:

• •

  An upper bound on the dimensional importance shift in singular values;

• •

  A connection between directional change and the largest singular value of ${\bm{A}}_{x}$.

We begin with the following result:

This result implies that the change in any singular value is at most the spectral norm of the perturbed matrix, regardless of its dimension (stewart1998perturbation). To apply this, we rewrite ${\bm{A}}_{new}=[{\bm{A}}_{x}|{\bm{A}}_{y}]=[{\bm{A}}_{x}|{\bm{O}}]+[{\bm{O}}|{\bm{A}}_{y}]$, where ${\bm{O}}$ is a zero matrix. According to Theorem 4.2, we have:

Since $\sigma_{i}([{\bm{A}}_{x}|{\bm{O}}])=\sigma_{i}({\bm{A}}_{x})$, Theorem 4.1 is proved. ∎

Intuitively, $\sum_{i}d_{sem}({\bm{v}}_{i},{\tilde{\bm{v}}}_{i}^{*})$ captures the perturbation in basis directions during cluster merging. Following (meyer2000matrix; belsley2005regression), it is expected that a higher sensitivity in ${\bm{A}}_{x}$, typically characterized by its condition number $\kappa({\bm{A}}_{x})={\sigma_{max}({\bm{A}}_{x})}/{\sigma_{min}({\bm{A}}_{x})}$, will result in larger directional changes when ${\bm{A}}_{x}$ and ${\bm{A}}_{y}$ are merged, i.e., $\sum_{i}d_{sem}({\bm{v}}_{i},{\tilde{\bm{v}}}_{i}^{*})=\mathcal{O}(\kappa({\bm{A}}_{x}))$.

Since ${\bm{A}}_{x}$ and ${\bm{A}}_{y}$ represent embedding vectors sampled similarly from the embedding space, the impact of noise on these vectors are similar. Therefore, their minimum singular values can be assumed comparable, i.e, $\sigma_{min}({\bm{A}}_{x})\approx\sigma_{min}({\bm{A}}_{y})\approx\sigma_{min}({\bm{A}}_{new})$. This allows us to discard the effects of minimum singular values, and the directional change can be approximated by $\sigma_{max}({\bm{A}}_{x})$, i.e., $\sigma_{max}({\bm{A}}_{x})\approx\mathcal{O}(\kappa({\bm{A}}_{x}))\approx\sum_{i}d_{sem}({\bm{v}}_{i},{\tilde{\bm{v}}}_{i}^{*})$.

While this approximation is necessarily coarse, it offers an efficient alternative for real-time clustering. Empirical results (Section 5.5) confirm its efficacy with minimal impact on performance.

We evaluate SAFARI and SFSes on six public datasets across text and image modalities. For the text modality, we use five diverse datasets: AG-News (zhang2015character) (with 4 topic classes: Business, Sci/Tech, Sports, and World), AAPD (yang2018sgm), IMDB (maas2011learning), Yelp,¹¹1https://www.yelp.com/dataset and NewsSpectrum (sun2024diversinews). For the image modality, we employ MIT-States (isola2015discovering), which enables the evaluation of object-attribute composition in visual embeddings. Further dataset details are available in Appendix A.1.

All methods were implemented in Python 3.8 and evaluated on a Ubuntu 20.04 machine with Intel^® Xeon^® Platinum 8480C and an NVIDIA H100 GPU.

We evaluate SAFARI and the resulting SFSes on two modality-diverse datasets: AG-News for text and MIT-States for images. Embedding spaces are constructed using BLINK (wu2020scalable) for AG-News and CLIP (radford2021learning) for MIT-States. To benchmark hierarchical discovery, we generate 4-level semantic label hierarchies (Lv0 to Lv3) for both datasets using Claude 3.7 Sonnet (anthropic2025claude), with details provided in Appendix A.2. We assess how well the SFSes align with the reference hierarchy using the impurity metric:

where $L_{i}$ is label class $i$; $C_{j}$ is cluster $j$ used to construct SFSes; $l$ and $c$ are the number of label classes and clusters. Lower impurity indicates greater semantic coherence within clusters, with 0 denoting perfect label concentration within clusters. As SAFARI progresses, it is expected that impurity grows due to the merging of semantically broader categories, with a consistent ordering across levels: Lv0 (most specific) $>$ Lv1 $>$ Lv2 $>$ Lv3 (most abstract).

Figure 4 shows a consistent decrease in impurity from Lv0 to Lv3 across iterations for both modalities. This trend reflects a hierarchical shift from specific to abstract semantics, confirming that SFSes capture coherent semantic groupings at multiple granularities.

The consistent patterns across text and image confirm the modality-agnostic nature of SAFARI. Without supervision, it uncovers increasingly abstract semantic relationships by tracking Semantic Shifts. By revealing how local neighborhoods compose global hierarchies, SAFARI offers a robust and generalizable framework for identifying hierarchical structures in embedding spaces, advancing our understanding of their inherent semantic organization.

We measure whether SFSes preserve meaningful semantics by testing their performance on classification tasks across text and image modalities. For text classification, we use four datasets: AG-News (4 topics), AAPD, IMDB, and Yelp, covering seven distinct classes. For image classification, we use MIT-States, filtering to 97 object classes with at least 240 samples each. We compare SAFARI against four standard classifiers: Support Vector Machine (SVM) (platt1999probabilistic; chang2011libsvm), K-Nearest Neighbors (KNN) (cover1967nearest; fix1985discriminatory), Random Forest (RF) (breiman2001random), and Multi-Layer Perceptron (MLP) (he2015delving; hinton1990connectionist). For classification using SFSes, we compute the distance between each test embedding and all identified SFSes, assigning the label to the nearest one. Results are presented in Table 1.

For text classification, SAFARI outperforms all baselines, surpassing SVM, the second-best, while using only around 50% of the computation time. In image classification, it ranks second to SVM in accuracy but runs 3.7$\times$ faster, demonstrating a strong accuracy-efficiency trade-off. Overall, SAFARI delivers competitive performance across modalities with notable computational savings, confirming that SFSes offer effective and efficient semantic representations.

We further evaluate political bias detection on the NewsSpectrum dataset (details in Appendix B). Results show that SAFARI successfully captures nuanced ideological distinctions, where standard classifiers often fail, particularly on underrepresented political leanings, highlighting its robustness in modeling subtle, real-world semantics beyond surface-level topics.

To evaluate the efficiency of SAFARI’s approximate Semantic Shift computation (Equation 2), we compare it against the full SVD-based method (Equation 1), in terms of runtime. Following the setup in Section 5.3, we sample the top 2,000 entities from each dataset and perform hierarchical clustering. Figure 5 depicts the results, with runtime averaged over 10 independent runs.

As shown in Figure 5, the approximate method achieves a 15$\sim$30$\times$ speedup over full SVD across all classes, with consistently low variance (as indicated by the error bars). Despite the substantial acceleration, the average error between exact and approximate Semantic Shifts remains below 0.01, within the $10^{-3}$ scale, ensuring strong accuracy-efficiency trade-offs. These results confirm that our approximation is a fast, stable, and reliable alternative to full SVD, making SAFARI scalable for large datasets. A detailed analysis of the two Semantic Shift components, i.e., Dimensional Importance Shift and Directional Change, is provided in Appendix C.

To illustrate SAFARI’s ability to uncover hierarchical semantics, we conduct a case study on the Sports category of the AG-News dataset, chosen for its structured, event-driven content. We apply SAFARI to the top 2,000 entity embeddings, computing Semantic Shifts at each iteration using both the exact (Equation 1) and approximate (Equation 2) methods. Figure 6 plots Semantic Shift curves between iterations 11,000 and 16,000, with notable evolutions at iterations 11,352 and 15,856. Figures 7 and 8 visualizes the corresponding hierarchical groupings.

Figure 6 shows that the approximate Semantic Shift curve closely mirrors the exact one, achieving a Pearson correlation of 0.92. This strong correlation confirms the reliability of our efficient approximate method introduced in Section 4.3. Moreover, SAFARI’s sliding-window-based dynamic thresholding effectively adapts to the non-uniform fluctuation of shifts across iterations, enable the discovery of subtle yet meaningful SFSes that would be missed by static thresholds. Further parameter study about the dynamic threshold mechanism in SAFARI is detailed in Appendix D.

SAFARI also reveals hierarchical semantic relationships with fine-to-coarse granularity. In Figure 7, early clusters capture specific U.S. university basketball and football teams, which gradually merge into broader categories like university sports teams. Figure 8 illustrates cross-national grouping: teams initially cluster by country, then merge into regional groupings. Notably, European teams (blue) consolidate more tightly than non-European teams (red), indicating structure-aware semantic abstraction. These results showcase SAFARI’s ability to track evolving semantic organization, understanding latent hierarchies in embedding spaces. Additional analysis is provided in Appendix E.