Ordination¶
A survey illuminates one chart at a time. Synthesis puts the elite trees of every chart in one picture, which needs distances between trees that no single chart holds.
The tree metric is squared Euclidean by construction, so a tree can be written
as a point whose ordinary distance to another point is the tree distance.
split_coordinates is that construction, and it makes distances exact rather
than an approximation. The whole cloud is therefore handled without subsampling:
the coordinate width follows the number of distinct splits, which the taxon set
bounds, not the number of trees.
faithfulness carries the scientific claim. A cloud of elite trees occupies
more than two dimensions, so no plane keeps both its distances and its
neighborhoods, and the measures state which was kept. neighbor_rank answers
where a map is untrustworthy rather than only how much.
This module returns arrays and dataclasses. It imports no plotting library, and
ordinate is a convenience whose t-SNE import is lazy. A caller may embed with
any tool and hand the result to faithfulness, which scores every method on the
same terms.
GOWER (1966), venna2001neighborhood, facco2017twonn. Each construction is
verified in docs/theory/algebra/.
split_coordinates
¶
Write each tree as a point whose ordinary distance is the tree distance.
The metric is
d^2 = (1 - w) * CBS^2 / c_ref + w * RF / r_ref
and both terms are squared Euclidean. The clade branch score is a squared Euclidean distance on the vector of split lengths, and the Robinson-Foulds count is the Hamming distance on the vector of split indicators, which is squared Euclidean because a binary difference squares to itself. A non-negative combination of squared Euclidean distances is squared Euclidean with the coordinates written side by side, each scaled by the square root of its weight:
x = concat( sqrt((1 - w) / c_ref) * lengths,
sqrt(w / r_ref) * indicators )
The columns run over the union of the splits of every tree, in ascending key order. A split absent from a tree contributes a length of zero and an indicator of zero, which is what makes the branch score of two trees with different split sets come out right.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
trees
|
sequence of HifukuTree
|
Trees on one namespace, each with |
required |
metric
|
CBSMetric
|
The metric whose |
required |
Returns:
| Type | Description |
|---|---|
ndarray
|
|
Notes
The array is dense in the splits, so its width grows with the diversity of the cloud rather than with the number of trees. A cloud whose trees share most of their splits stays narrow.
Source code in src/hifuku/ordination.py
distances
¶
The full matrix of tree distances, computed through the coordinates.
The result is exact. It is the same number
:class:~hifuku.metric.branch_score.CBSMetric returns for a pair, computed
for every pair at once.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
source
|
Survey or sequence of HifukuTree
|
A survey uses its cloud, in the order :meth: |
required |
metric
|
CBSMetric
|
Required when |
None
|
Returns:
| Type | Description |
|---|---|
ndarray
|
|
Source code in src/hifuku/ordination.py
mds
¶
Classical multidimensional scaling, also called principal coordinates.
Gower's section 3 gives the construction. Set a_ij = -d_ij^2 / 2, center
the matrix by rows and columns, and take the latent roots and vectors. A
vector scaled so that its sum of squares equals its root gives one column of
the configuration, and the distances between the rows are the distances the
input asked for.
A real configuration exists exactly when the centered matrix is positive semi-definite. The tree metric is squared Euclidean for any set of trees, so that condition always holds here and a negative root above rounding error means a defect in the implementation.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
D
|
ndarray
|
|
required |
dim
|
int
|
Coordinates to keep. Fewer than the positive roots gives the best fit in that many dimensions, in the least-squares sense of Gower step (v). |
2
|
report
|
bool
|
Return a :class: |
False
|
Returns:
| Type | Description |
|---|---|
ndarray or tuple
|
|
Source code in src/hifuku/ordination.py
ScalingReport
dataclass
¶
What classical scaling found, beyond the configuration itself.
Attributes:
| Name | Type | Description |
|---|---|---|
eigenvalues |
ndarray
|
The latent roots of the centered matrix, largest first. |
positive_mass |
float
|
Sum of the positive roots. The variance a real configuration holds. |
negative_mass |
float
|
Absolute sum of the negative roots. Zero for a squared Euclidean input. A value above rounding error means the distances are not of negative type, which for this metric means a defect in the implementation and not a property of the data. |
residual |
float
|
Gower step (v): the trace of the centered matrix less the roots kept. The sum of squares of the perpendiculars onto the reduced space. |
Source code in src/hifuku/ordination.py
faithfulness
¶
Score an embedding against the distances it came from.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
D
|
ndarray
|
|
required |
X
|
ndarray
|
|
required |
k
|
int
|
Neighborhood size. Venna and Kaski state the scaling for |
15
|
Returns:
| Type | Description |
|---|---|
Faithfulness
|
|
Source code in src/hifuku/ordination.py
Faithfulness
dataclass
¶
How much of a cloud's neighborhood structure a map keeps.
A cloud of elite trees occupies more than two dimensions, so no plane keeps both its distances and its neighborhoods. These measures state which was kept and by how much.
trustworthiness and continuity are the two measures of Venna and
Kaski, equations 1 and 2. The penalty is weighted by rank rather than
counted, so a point drawn onto the map from true rank 1000 costs far more
than one from rank 16. A map that moves nothing scores 1 and the worst
possible map scores 0; the scaling is derived in
docs/theory/algebra/venna2001neighborhood-trustworthiness.py.
neighbor_rank is not from the paper. It answers where a map is
untrustworthy rather than how much: for each point, the mean true rank of
the neighbors the map puts around it.
Attributes:
| Name | Type | Description |
|---|---|---|
k |
int
|
Neighborhood size the measures used. |
trustworthiness |
float
|
Penalizes points the map drew in from far away. A reader has no way to tell an introduced proximity from a real one, which is why Venna and Kaski call this the more damaging error. |
continuity |
float
|
Penalizes true neighbors the map pushed away. The paper calls this preservation of the original neighborhoods. |
neighbor_rank |
ndarray
|
|
ideal_rank |
float
|
|
false_neighbor_rate |
float
|
Share of map neighbors whose true rank is above |
Source code in src/hifuku/ordination.py
intrinsic_dimension
¶
Estimate the intrinsic dimension from first and second neighbors.
The TWO-NN estimator of Facco et al. For each point take the distances to
its first and second nearest neighbors and form mu = r2 / r1. For a
locally uniform density the distribution of mu depends on the intrinsic
dimension and not on the density, which cancels, so
-log(1 - F(mu)) / log(mu) = d.
The points (log mu, -log(1 - F)) therefore lie on a line through the
origin whose slope is the dimension. F comes from the sample: sort
mu upward and set F(mu_i) = i / n.
Using only the two nearest neighbors is what makes the estimate usable on a cloud of elite trees, which is not uniformly distributed. Local uniformity is needed only at the scale of the second neighbor.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
D
|
ndarray
|
|
required |
discard
|
float
|
Fraction of the largest |
0.1
|
Returns:
| Type | Description |
|---|---|
float
|
The estimated dimension. |
Notes
Report this alongside the classical scaling spectrum rather than instead of it. This measures the local dimension of the manifold, and the spectrum measures how many linear axes the distances occupy. A disagreement between them says something about the cloud.
Source code in src/hifuku/ordination.py
project_landmarks
¶
Place points that are not in the cloud into an existing embedding.
An anchor tree is a coordinate reference, not a member of the cloud, so it has no row in the embedding. This puts one on the map by averaging the embedded positions of the cloud, weighted by closeness in the original space.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
coords
|
ndarray
|
|
required |
X
|
ndarray
|
|
required |
landmarks
|
ndarray
|
|
required |
sigma
|
float
|
Width of the weighting kernel. Defaults to the median distance from a landmark to its nearest cloud point, which follows the scale of the data. A small width places a landmark on its nearest cloud point, and a large one pulls every landmark to the centroid. |
None
|
Returns:
| Type | Description |
|---|---|
ndarray
|
|
Source code in src/hifuku/ordination.py
ordinate
¶
Embed a distance matrix with an outside tool.
Barnes-Hut t-SNE is the one step this package does not own, so this is a
convenience behind the analysis extra with a lazy import, in the same
way :meth:Survey.frame reaches for pandas. A caller may skip it, embed
with any tool, and hand the result to :func:faithfulness, which scores
every method on the same terms.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
D
|
ndarray
|
|
required |
method
|
str
|
|
'tsne'
|
**kw
|
Passed to the underlying estimator. |
{}
|
Returns:
| Type | Description |
|---|---|
ndarray
|
|