A disclaimer: I still don’t have full command over the concepts of the papers, and this post may be subject to a future update, so feel free to mention if I got something wrong.

So, what if we have a very dense graph maybe and we want to beat the Dijkstra algorithm, then how can we do it? At least in the planar case. The answer is to sparsify the initial object a bit, and instead decompose the graph into local pieces in which you have a rough information of, you use that to traverse the global structure in a way, and when needed you use that to compute the true shortest path, by the local graph where using Dijkstra or Bellman–Ford is efficient.

So in this post I am going to assume the existence of \(r\)-separators.

Ok, let’s review the definition of \(r\)-separators:

Let \(\Sigma\) be a simple planar map with weighted darts; for now we’ll assume that all edge weights are non-negative. If necessary, add infinite-weight edges so that \(\Sigma\) is a simple triangulation.

Recall that a good \(r\)-division of \(\Sigma\) is a subdivision of \(\Sigma\) into \(O(n/r)\) pieces \(R_1, R_2, \ldots\) satisfying three conditions:

• Each piece has \(O(r)\) vertices.
• Each piece has \(O(\sqrt{r})\) boundary vertices (that is, vertices that are shared with other pieces).
• Each piece has \(O(1)\) holes (faces of the piece that are not faces of \(\Sigma\)).

Recall. An MSSP (Multiple-Source Shortest Paths) data structure for a planar graph with \(n\) vertices can be constructed in \(O(n\log n)\) preprocessing time and space, and supports distance queries in \(O(\log n)\) time. The queries are restricted to sources lying on a single face.

For a piece \(R_i\) of an \(r\)-division, we have \(O(r)\) vertices and \(O(\sqrt{r})\) boundary vertices. Since the boundary vertices lie on the constant number of holes of the piece, we can construct an MSSP data structure for \(R_i\) and use it to compute all boundary-to-boundary distances.

The preprocessing time for one piece is

There are \(O(\sqrt{r})\) boundary vertices, so there are

boundary-to-boundary distance queries. Each query takes \(O(\log r)\) time, giving another

time per piece. Hence, the total time for one piece is

Since the \(r\)-division contains \(O(n/r)\) pieces, the total time to construct the dense distance graph is

Dense Distance Graph. For each piece \(R_i\), we construct a weighted clique on its boundary vertices. For every pair of boundary vertices \(u,v\in\partial R_i\), the corresponding dart has weight

where \(d_{R_i}(u,v)\) denotes the shortest-path distance from \(u\) to \(v\) using only vertices and edges of \(R_i\).

The dense distance graph is the union of these \(O(n/r)\) weighted cliques.

Each piece has \(O(\sqrt{r})\) boundary vertices, so the total number of distinct boundary vertices is

Furthermore, each piece contributes

weighted darts. Therefore, over all \(O(n/r)\) pieces,

Thus the dense distance graph has

and can be constructed in

time.

So, how can we use this rough structure to beat Dijkstra in an efficient manner? Let’s do what one naturally may do:

Theorem. Given any planar map \(\Sigma\) with non-negative lengths on its edges, we can compute the shortest paths from any vertex \(s\) to every other vertex of \(\Sigma\) in \(O(n\log\log n)\) time.

Proof. We begin by triangulating \(\Sigma\) in \(O(n)\) time, building a good \(r\)-division for the resulting triangulation in \(O(n)\) time, and building the dense distance graph for the \(r\)-division in \(O(n\log r)\) time, for some parameter \(r\) to be determined. In the top-level recursive call to build the good \(r\)-division, we artificially declare \(s\) to be a boundary vertex, so that it survives as a vertex in the dense-distance graph.

Next, we compute the shortest-path distance from \(s\) to every boundary vertex of the \(r\)-division by running Dijkstra’s algorithm on the dense distance graph. If we implement Dijkstra’s algorithm using Fibonacci heaps, this step takes

time, since

And actually this is possible because we have the information of paths and distances implicitly stored in the distance subgraph, so for each boundary, to go into another boundary we must traverse different pieces from boundary to boundary, so running Dijkstra actually gives us the desired distances.

Finally, for each piece \(P\), we attach an artificial source \(s'\) to each boundary vertex \(u\) with an edge of length

where \(d(s,u)\) is the distance from \(s\) to \(u\) computed in the dense distance graph. We then compute a shortest-path tree in \(P\) rooted at \(s'\) using Dijkstra’s algorithm. Since each piece contains \(O(r)\) vertices, this takes

time per piece, or

time overall.

Thus, the overall running time is

In particular, setting

gives

and

Therefore, the total running time is

So as you can see, we incorporated terse sparsified local structures and their information to traverse the graph in a more efficient manner, then when we come sufficiently close, we can consider a piece or region with all of its nodes.

So now here come the ideas of the paper actually. We want to use Dijkstra and Bellman–Ford in case we have negative cases eventually, but we want to do it in a more efficient manner. And one of the more useful techniques of doing so is to bypass the \(O(m)\) complexity of the relaxation phase for each vertex. And to do so, we must use some characteristics of planar graphs. And we will see how we can do it, do kind of bring it down to something like \(O(log(m))\). It’s a rough idea, and so in order to do that we must use the idea of a Monge array, that lets us to do a search like binary search.

A matrix \(M=(M_{ij})\) is totally monotone if for every \(i,i',j,j'\) such that \(i<i'\) and \(j<j'\), if

then

A matrix \(M=(M_{ij})\) is convex Monge (respectively, concave Monge) if for every \(i,i',j,j'\) such that \(i<i'\) and \(j<j'\), we have

(respectively,

\(M_{ij}+M_{i'j'} \le M_{ij'}+M_{i'j}.\))

It is immediate that if \(M\) is convex Monge, then it is totally monotone. It is also easy to see that the matrix obtained by transposing \(M\) is also totally monotone.

Remark. In 1987, Alok Aggarwal, Maria Klawe, Shlomo Moran, Peter Shor, and Robert Wilber described an elegant recursive algorithm that finds the minimum element in every row of an \(n\times n\) Monge array in \(O(n)\) time, now usually called the SMAWK algorithm.

Theorem. For any four indices \(k,k',\ell,\ell'\) such that either \(A_{k\ell}\), \(A_{k\ell'}\), \(A_{k'\ell}\), and \(A_{k'\ell'}\) are all in \(A\)’s upper triangle, or are all in \(A\)’s lower triangle (i.e., either

or

the convex Monge property holds:

Proof. It is simply a case of using the Jordan curve theorem, finding a middle node or intersection node and uncrossing the nodes to derive the inequality.

The Monge property also implies another important property that we use. That is, for \(y\in B\), if \(u\in A\) is the node that minimizes \(d(u,y)\), then for any \(v\neq u\in A\) and \(x\neq y\in B\),

Given \((A,B,d)\) with the Monge property, the Monge matching problem is to find a parent \(p(v)\in A\) for all \(v\in B\) such that

The set of pairs

forms the minimum Monge matching.

The non-crossing property states that it is enough to look for a function \(p(\cdot)\) that has no “crossing.” Therefore, we can use standard divide-and-conquer techniques to derive an

algorithm for the problem, where

Or constructively we can start from a row to find the min element in the column and use a binary search to add a row, and compute the local minima, which takes the same complexity and is basically the same algorithm.

so ok.

General complete graphs are not Monge. We can preserve the information that the dense distance graph gives us using \(O(\log n)\) Monge arrays, by halving the cycle separator in half and considering only the distances between the two halves and continuing to do this recursively. This way we have at most \(O(\log n)\) levels of Monge matrix, which we can find the possible relaxation candidates simultaneously in \(O(k\log k)\) for which \(k\) is the number of rows and columns in one of the Monge arrays.

Citing the paper.

We accomplish this by maintaining the edges of each subpiece of \(P\) in \(O(\log n)\) levels of Monge arrays. After the definition of the Monge arrays, we describe how all edges in each Monge array with \(k\) nodes can be relaxed in \(O(k\log k)\) time, where \(k\) is the number of nodes in the data structure.

The first Monge array that we define is formed as follows. Divide the border nodes in some subpiece into two halves, the first (or left) half in the circular order (with an arbitrary starting point) and the second (or right) half. Consider the set of edges in the dense distance graph that go from the left border nodes to the right ones. The edges obey the Monge property, since there is an underlying shortest path tree in which the corresponding paths do not cross.

Using the same left–right partitioning, we can define another Monge array with the direction of edges reversed (i.e., edges in the array go from the right border nodes to the left border nodes).

So, when we want to simulate Dijkstra’s algorithm or Bellman–Ford, we only perform the relaxation operations within these related \(O(\log n)\) data structures.

Therefore, relaxing the edges can be done in

since the total number of nodes in all the data structures is

for each subpiece of \(P\) in the decomposition, and relaxing the edges within a Monge array with \(k\) nodes takes \(O(k\log k)\) time.

In this way, we are handling the information in a very clever manner in order to obtain a better query or time complexity.

Suppose that we have no negative edge; for the other case we have to talk about Bellman–Ford too, which I probably won’t in this chapter or will later update this post.

So how does FR-Dijkstra work?

First of all we must know that things are computed in a clever data structure:

On-line Bipartite Monge Search

It is online in the sense that Dijkstra’s algorithm is online, because in each loop, after relaxation we can find a new node and set it to shortest distance instead of infinity.

An important subroutine in the efficient implementation of Dijkstra’s algorithm running on dense distance graphs is the on-line bipartite Monge search problem.

The algorithm computes shortest paths by combining several instances of the following problem.

Actually the way that we can recursively partition the graph has been improved since the time of the paper publication; they would halve the graph horizontally following the Tarjan–Lipton paper, but we can decompose the planar map cyclically in each step.

So for each step we can compute the global to local structure of that piece, and we may at most do it \(\log n\) times, so the time complexity would be \(\log n\) times whatever complexity that we get.

And we recursively perform the algorithm that beat Dijkstra recursively to get from one layer to another layer of \(s\)–\(v\) shortest path.

On-line bipartite Monge search. Given \(d(x,y)\), representing a dense distance graph for \(X\times Y\), maintain a parent \(x\in X\) for every \(y\in Y\), while the initialization value \(D(\cdot)\) for each \(x\in X\) is revealed on-line, one node at a time.

The matching can be computed on-line in overall time

which is the same asymptotic time as the standard divide-and-conquer algorithm.

Maintaining a matching means, in this context, managing:

• a set of active nodes \(A\subseteq X\); and
• a growing set of matched nodes \(M\subseteq Y\), or, more importantly, a shrinking set of yet-unmatched nodes \(Y\setminus M\).

The data structure supports the following three operations, which are used by Dijkstra’s algorithm:

• FindMin(): returns the minimum unmatched node \(y\in Y\setminus M\). This functions as a priority queue for the right-hand side.
• ExtractMin(): adds the current minimum node to \(M\).
• ActivateLeft\((x,\delta)\): reveals the initialization value \(D(x)=\delta\) for some \(x\in X\) and updates the preliminary matches.

Efficient Implementation.

We use a heap and intervals in an ordered set \(Y\). Assume that, for each \(x\in X\), we have precomputed, together with the dense distance graph, a data structure supporting queries of the form

for any \(i^-,i^+\). Note that this data structure is independent of \(D(x)\), and the query can be answered using an LCA data structure in \(O(1)\) time.

Maintain a binary search tree for the active nodes \(x\in A\).

For every active \(x\in A\), maintain an interval

of children \(y\in Y\). Here, \(x\) is the current parent of \(y\), although this parent may change over time.

Maintain a priority queue (heap) containing, for every active \(x\in A\), its shortest edge to an unmatched node \(y\in Y\setminus M\).

• FindMin() returns the minimum element of the heap in \(O(1)\) time.

• ExtractMin() adds the current minimum to \(M\). Suppose the minimum is \(y_j\in Y\setminus M\) with parent \(x\in A\). We need to insert the second-shortest edge from \(x\) into the heap.

  Instead, create two dummy nodes \(x'\) and \(x''\) spanning the intervals

  \[[i^-(x),j) \qquad\text{and}\qquad [j+1,i^+(x)),\]

  respectively. Find the minimum in each of these intervals using the above LCA data structure for \(x\), and insert the resulting edges into the heap. This takes

  \[O(\log |Y|)\]

  time.

  There are at most two new dummy nodes for each extracted node of \(Y\). Hence, there are only

  \[O(|X|+|Y|)\]

  dummy nodes in total.

• ActivateLeft\((x,\delta)\) activates a new node \(x\in X\) with initialization value

  \[D(x)=\delta.\]

  First, compute its interval

  \[[i^-(x),i^+(x)).\]

  If \(x\) is the first active node, the interval contains all of \(Y\). Otherwise, we proceed as follows.

  1. Walk upward through the active nodes with non-empty intervals until reaching the first node \(x'\) such that

     \[d(x',y_{i^-(x')}) < d(x,y_{i^-(x')}).\]

     Then perform a binary search for \(i^-(x)\) in the range

     \[[i^-(x'),i^+(x')).\]

     This takes

     \[O(\log |Y|)\]

     time.

  2. Analogously, walk downward through the active nodes with non-empty intervals until reaching the first node \(x''\) such that

     \[d(x'',y_{i^+(x'')-1}) < d(x,y_{i^+(x'')-1}).\]

     Then perform a binary search for \(i^+(x)\) in the range

     \[[i^-(x''),i^+(x'')).\]

     This also takes

     \[O(\log |Y|)\]

     time.

  Finally, update the other intervals and the heap.

  For nodes in \(A\) between \(x\) and \(x'\) (and between \(x\) and \(x''\)), the corresponding interval becomes empty. These nodes are therefore deactivated, and their corresponding minimum unmatched nodes in \(Y\setminus M\) are removed from the heap.

  Since the sequential search (the “walk up/down”) visits each such node at most once, its total cost is amortized

  \[O(\log |Y|)\]

  per \(x\in X\).

  For \(x'\) and \(x''\), their intervals may shrink to

  \[[i^-(x'),i^-(x)) \qquad\text{and}\qquad [i^+(x)+1,i^+(x'')),\]

  respectively. Either interval may also become empty. The corresponding minimum unmatched node in \(Y\setminus M\) is then updated in the heap, taking

  \[O(\log |Y|)\]

  time.

So really we are keeping a global heap, local heaps for each half of the complete dense graph pieces, making them Monge. And the magic really happens when we are maintaining the parents for the right side of bipartite graphs, which takes \(\log Y\) times amortized. And the algorithm is true because just like the usual Dijkstra it is picking the edges greedily, so only we are using the structure to relax more efficiently.

And the real magic happens when we are handling the parents as I said, Reveal\((j,x)\) is implemented in \(O(\log k)\) amortized time as follows:

• Find the live intervals

  \[I^-=(j^-,i^-_{\min},i^-_{\min}) \qquad\text{and}\qquad I^+=(j^+,i^+_{\min},i^+_{\min})\]

  immediately before and after \((j,\cdot,\cdot)\) in lexicographic order, in \(O(\log k)\) time, by querying the balanced binary search tree.

• While

  \[M[i^-_{\min},j] < M[i^-_{\min},j^-],\]

  replace \(I^-\) with its predecessor in lexicographic order. Then binary search for the smallest index \(i_{\min}\) such that

  \[M[i_{\min},j] < M[i_{\min},j^-].\]

• While

  \[M[i^+_{\min},j] < M[i^+_{\min},j^+],\]

  replace \(I^+\) with its successor in lexicographic order. Then binary search for the smallest index \(i_{\max}\) such that

  \[M[i_{\max},j] < M[i_{\min},j^+].\]

• Delete any live intervals \((j^\pm,\cdot,\cdot)\) that overlap

  \[(j,i_{\min},i_{\max})\]

  from the priority queue.

• Insert the new live intervals

  \[(j,i_{\min},i_{\max}), \qquad (j^-,\cdot,i_{\min}-1), \qquad (j^+,i_{\max}+1,\cdot)\]

  into the priority queue.

To obtain the claimed \(O(\log k)\) amortized time bound, we charge the time required to delete any interval from the priority queue to its earlier insertion. The requirement that we only hide rows when their minimum elements are visible implies that there is exactly one live interval in the revealed column.

FR-Dijkstra\((G,s)\)

for all v in V(G):
    d(v) ← ∞
d(s) ← 0
S ← ∅

Convert each DDG into bipartite DDGs (X_i, Y_i).

for each (X, Y) such that s ∈ Y:
    (X, Y).Insert(s, 0)
    Global.DecreaseKey((X, Y), 0)

while S ≠ V(G):
    (X, Y) ← Global.ExtractMin()
    u ← (X, Y).ExtractMin(Y \ M)

    if u ∉ S:
        for each (X', Y') such that u ∈ X':
            (X', Y').ActivateLeft(u, d(u))
            v ← (X', Y').FindMin(Y' \ M')
            Global.DecreaseKey((X', Y'), d(v))
        S ← S ∪ {u}

    v ← (X, Y).FindMin(Y \ M)
    Global.DecreaseKey((X, Y), d(v))

Sources

• Jittat Fakcharoenphol and Satish Rao, Planar graphs, negative weight edges, shortest paths, and near linear time. Journal of Computer and System Sciences (the 2006 paper this post is about).
• Erik Demaine, MIT 6.889, Lecture 12: Multiple-Source Shortest Paths.
• Philip Klein, Shortest paths in planar graphs with negative lengths.
• Jeff Erickson, computational topology notes: 15 — Shortest paths and 17 — Faster minimum cut.

I am going to study and explain three papers which have led to a dynamic solution for the facility location problem.

Ok, I think in the series of the several papers that I am going to review, the data structure designed by James Lee and his co-author is a revealing piece.

Nearest-neighbor search (NNS) is the problem of preprocessing a set $S$ of $n$ points lying in a huge (possibly infinite) metric space $(M,d)$ so that, given a query $q \in M$, one can efficiently locate the nearest point to $q$ among the points in $S$.

Let $q$ be the query and $a \in S$ the closest point to $q$ in $S$. Then a $(1+\varepsilon)$-NNS algorithm is one which, given $q$, returns some $s \in S$ such that

So while it is NP-hard to return the nearest neighbor, it is worthwhile to examine whether we can approximate the answer in an efficient manner. Lee and his co-author used a very interesting classic idea: move from coarse to fine approximation. In a sense, their data structure is analogous to the binary search algorithm, with an extra twist and turn. Moreover, they employ the idea of doubling dimension as a means to count the complexity of the algorithm that they propose.

The doubling dimension of a metric space $(X,d)$, denoted in this post by $\dim(X)$, is the minimum value $\rho$ such that every set in $X$ can be covered by $2^{\rho}$ sets of half the diameter. (The diameter of a set $S \subseteq X$ is

)

A metric space is called doubling if its doubling dimension is $O(1)$. This notion of dimension measures the “volume growth” of $X$.

So, following the paper, we want to prove the following result:

Theorem. Let $(S,d)$ be a finite metric space with doubling dimension $\dim(S)$ and aspect ratio $\Delta=\Delta_S$, defined as

Then there exists a dynamic data structure for approximate nearest-neighbor search with the following properties:

1. The data structure uses $O(n)$ space if $S$ is doubling.
2. It answers $(1+\varepsilon)$-nearest-neighbor queries in

time.

1. It supports insertions and deletions in

time.

Moreover, in most applications,

and therefore

The running time depends exponentially on $\dim(S)$. Furthermore, in the distance-oracle model, where access to the ambient metric space is restricted to distance queries, this exponential dependence on the doubling dimension is necessary.

So let us first follow a simplified version of the data structure.

Let $(S,d)$ be the metric space against which queries are performed. For the sake of this informal discussion, assume that the minimum interpoint distance in $S$ satisfies

Under this assumption, the aspect ratio $\Delta$ is simply the diameter of $S$.

In what follows, for a subset $R\subseteq S$ and a point $x\in M$, we define

Let $k=\log \Delta$, and for each $i=0,1,\ldots,k$, let $Y_i$ be a $2^i$-net of $S$. For every point $y\in Y_i$, define the set

where $\gamma$ is a constant to be specified later.

So what are we going to do here? We are providing details from coarse to fine for the underlying data structure! And $L_{y,i}$ gives us a means to search through the metric space. I would rather call them approximate $2^{i-1}$ neighbors of the given point. So we have a means of sparsifying the metric space, and we build a tree based on the sparsification. It is obvious that we can build the nets in a greedy manner.

Since $L_{y,i}\subseteq Y_{i-1}$, and $Y_{i-1}$ is a $2^{i-1}$-net, any two distinct points of $L_{y,i}$ satisfy

Hence, the minimum interpoint distance in $L_{y,i}$ is $2^{i-1}$.

Moreover, by the definition

every point of $L_{y,i}$ lies within distance $\gamma 2^i$ of the center $y$. Therefore, for any $z_1,z_2\in L_{y,i}$, the triangle inequality gives

Thus,

Consequently, the aspect ratio of $L_{y,i}$ is bounded by

which is a constant.

So, in order to count the number of children of any node in the tree, we use the doubling property and use that to count the number of children.

Lemma. Let $(S,d)$ be a metric space, and let $Y\subseteq S$. If the aspect ratio of the metric induced on $Y$ is at most $\alpha$, where $\alpha\ge 2$, then

Proof. Let

and

be the minimum and maximum interpoint distances in $Y$, respectively. Assume that

Notice that $Y$ is contained in a ball of radius

centered at any point of $Y$.

Applying the definition of doubling dimension iteratively, this ball (and hence $Y$) can be covered by

balls of radius

Each such ball can contain at most one point of $Y$. Indeed, if two distinct points of $Y$ were contained in the same ball, then by the triangle inequality their distance would be at most

contradicting the definition of $d_{\min}$.

Therefore,

When $S$ is a doubling metric, the Lemma implies that

where the hidden constant depends only on the choice of $\gamma$ (and the doubling dimension of $S$).

Furthermore, the top-level net $Y_k$ contains exactly one (arbitrary) point, which we denote by

Given a query point $q\in M$, we first set

Then, iteratively for

we find the point in

that is closest to $q$, and update $y$ to be this point for the next iteration.

If, at some stage, we reach a point such that

then we stop the procedure and output the current value of $y$. Otherwise, after completing all iterations, we output the final value

First, notice that the running time of this algorithm is at most

since finding the closest point to $q$ among a list $L_{y,i}$ takes constant time. This follows from the fact that every list has size

It remains only to prove that the output point $y$ is a $3$-approximation to the nearest neighbor. Let $a\in S$ be the true nearest neighbor of $q$. We need to show that

To this end, let $j$ be such that

but

i.e., the step in which the distance does not decrease by a factor of $2$. Since the closest $2^{j-1}$-net point to $a$ satisfies

the closest $2^{j-1}$-net point to $a$ is contained in $L_{y,j}$. Let $y^*\in Y_{j-1}$ be such that

We need to show that

In this case, we have $y^*\in L_{y,j}$. Using the triangle inequality,

Therefore, choosing

suffices.

This shows that the descent process “tracks” the closest net point to $a$. Now, we have

Hence,

Since

we conclude that

Therefore, $y$ is a $3$-approximate nearest neighbor of $q$.

Similar arguments show that if the algorithm terminates with

then $y$ is actually the closest point to $q$ in $S$, because we have a $1$-net and we have the closest point to $a$ in the set $L_{y,i}$.

For a technical reason, the above data structure does not support efficient deletions and might require space

The main remedy for these latter problems is to choose $Y_i$ to be a $2^i$-net of $Y_{i-1}$ rather than a $2^i$-net of $S$.

Let us get into the main algorithm.

Let

and call every value $r\in\Gamma$ a scale. Only $O(\log\Delta)$ of these scales are relevant; the remaining scales are trivial.

For every $r\in\Gamma$, let $Y_r$ be an $r$-net of $Y_{r/2}$. We define

for all scales

For every $r\in\Gamma$ and every $y\in Y_r$, …

so we are making the details more coarse by only having the nets in $Y_i$ instead of $S$.

For every $r\in\Gamma$ and every $y\in Y_r$, the data structure stores a list of the nearby points to $y$ among the $(r/2)$-net $Y_{r/2}$. This scale-$r$ navigation list of $y$ is defined as

where $\gamma>0$ is a universal constant.

So we are roughly halving the distance at each step.

Lemma. For every scale $r$, the following properties hold:

1. Covering: \(d(z,Y_r)<2r\) for every $z\in S$.

2. Packing: \(d(x,y)\geq r\) for every $x,y\in Y_r$ with $x\neq y$.

The proofs are easy, and the first one follows by induction. So, it adheres to our intuition of having a rougher estimation while also helping us to preserve space in our algorithm.

We also have the following lemma.

Lemma. The size of every navigation list is at most

which has the same proof as before.

The data structure maintains this point $y_{\mathrm{top}}$ and the cutoff scale

for the sake of bootstrapping most of the operations.

Third, for all scales

the net $Y_r$ is equal to $S$. Therefore, for scales

every point $x\in S$ has a trivial list

These trivial lists can be represented succinctly by storing, for every $x\in S$, a scale $r_x\in\Gamma$ below which all lists of $x$ are trivial. For the sake of analysis, define

Lemma. The cutoff scales satisfy

and

Consequently, every point has

non-trivial navigation lists.

The proof is easy.

Combining the above results, we obtain an upper bound on the total space required by the data structure. Specifically, the space complexity is

For each point $x\in S$, the non-trivial navigation lists of $x$ are stored using, for example, a balanced search tree. This requires linear space and supports the operations find, insert, and delete in logarithmic time.

It then follows from Lemma 2.3 that inserting a new navigation list for a point can be performed in time

Theorem. The size of the data structure is

words.

Proof. The proof bounds the size using a charging argument.

Definition. A subset $Z_r\subseteq Y_r$ is called non-proper if it contains only one point $x$ and

Otherwise, $Z_r$ is called proper.

Algorithm: Approx-NNS

Input: $q\in X$ and $\varepsilon>0$.

1. Set

and

1. While

and $Z_r$ is proper, do:

a. Set

b. Set

1. Return the point $z\in Z_r$ for which

is minimal.

Lemma. Let $a$ be a closest point to $q$ among $S$. Then every set $Z_r$ computed by Procedure Approx-NNS contains a point $z_r$ such that

Lemma. Procedure Approx-NNS outputs a point whose distance from $q$ is at most

Proof. Let $r^*$ be the value of $r$ at the end of the procedure. It suffices to prove that

because then the proof follows from the choice made in line 5 of the procedure.

There are two conditions in line 2 that may cause the while-loop to terminate. First, consider the case where

By the previous lemma, we know that

Combining the above inequalities, we obtain

Substituting this bound back into the previous inequality gives

as desired.

Consider now the case where $Z_{r^*}$ is non-proper. We may assume that

otherwise the claim is immediate.

For analysis, suppose we continue the procedure without the condition that $Z_r$ is proper. This process must terminate, since

while

Moreover, the returned point remains unchanged, because all further sets $Z_r$ for $r<r^$ contain the same single point as $Z_{r^}$. It is easy to see that the above analysis (including Lemma 2.4) also applies to the modified procedure. Therefore, both the modified and the actual procedure satisfy

In fact, if the procedure stops with a non-proper set $Z_r$, then it returns the unique closest point to $q$ in $S$. This follows by applying the same modified-procedure argument with an arbitrarily small $\varepsilon>0$.

Lemma. Procedure Approx-NNS runs in time

Proof. Since the distance between any two points of

is at least $r/2$, Lemma 1.2 implies that

A simpler bound is obtained by separately bounding the number of iterations of the procedure and the running time of a single iteration. For the first part, consider the number of iterations performed by the procedure.

Theorem. The data structure can be updated after inserting a point into $S$ in time

This update includes

distance computations.

Proof sketch. The main idea is that, regardless of how the nets were constructed, each $r$-net $Y_r$ can be updated after an insertion of a new point $q$ by either adding $q$ to $Y_r$ or leaving it unchanged.

This follows by induction on the scale $r$. For the base case, consider a sufficiently small scale $r$. By definition, $Y_r$ contains all points of $S$, and therefore $q$ must be added to this net.

For the inductive step, first assume that the update to $Y_{r/2}$ leaves it unchanged. In this case, $Y_r$ also remains unchanged. Otherwise, suppose that $q$ is added to $Y_{r/2}$. We then update $Y_r$ by adding $q$ to it if and only if

This guarantees that $Y_r$ remains an $r$-net of $Y_{r/2}$ (although other valid updates are also possible).

To decide whether

we use a set $Z_r$ that contains all points of $Y_r$ that are sufficiently close to $q$.

Recall that the net $Y_r$ is maintained only through its navigation lists. Therefore, adding $q$ to $Y_r$ requires constructing the scale-$r$ navigation list of $q$ and updating the scale-$2r$ navigation lists of nearby points. Both tasks are performed using the sets $Z_r$.

The proof uses induction on the scale $r$ to decide whether a new point $q$ should be added to the net $Y_r$.

Base case (smallest scale). At the finest scale, $Y_r$ contains all points of $S$ by definition. Therefore, the new point $q$ must be added to $Y_r$.

Inductive step. Assume that the update rule is correct for the smaller scale $r/2$. We consider two cases.

1. $q$ is not added to $Y_{r/2}$.

In this case, $q$ is already within distance less than $r/2$ of some point in $Y_{r/2}$. Hence, it is also covered at scale $r$, and there is no need to add $q$ to $Y_r$.

1. $q$ is added to $Y_{r/2}$.

Now we decide whether $q$ should also be added to $Y_r$. The condition is whether $q$ is sufficiently far from all existing points of $Y_r$:

If this condition holds, we add $q$ to $Y_r$. Otherwise, $Y_r$ remains unchanged.

This update rule preserves the invariant that $Y_r$ is an $r$-net of $Y_{r/2}$.

Step 3: Using $Z_r$ for efficient distance checks.

A naive check of the condition

would require computing the distance from $q$ to every point in $Y_r$, which is too expensive.

Instead, the algorithm maintains a local set $Z_r$ that contains all points of $Y_r$ that are sufficiently close to $q$. Since $S$ has bounded doubling dimension $\dim(S)$, the number of points that can exist in such a local neighborhood while maintaining the packing property of the net is bounded by

Therefore, it is sufficient to check the distance between $q$ and the points in $Z_r$. If no point in $Z_r$ is within distance $r$ of $q$, then

and we add $q$ to $Y_r$. Otherwise, $Y_r$ remains unchanged.

Step 4: Updating the navigation lists.

If $q$ is added to $Y_r$, the update of the data structure is not complete yet. The navigation lists, which connect points between consecutive scales, must also be updated.

First, we construct the scale-$r$ navigation list of $q$, which contains the nearby points of $q$ in the coarser net $Y_{2r}$.

Second, we update the scale-$2r$ navigation lists of all nearby points in $Y_{2r}$ by adding $q$ as one of their neighbors.

The set $Z_r$ is used to identify exactly the points whose navigation lists must be updated. Since $Z_r$ has size bounded by

the update remains local and can be performed efficiently.

The final update time is

The complexity can be understood as follows.

Local work. At each scale, the algorithm only examines the local set $Z_r$. By the doubling dimension bound,

Therefore, the distance computations and navigation-list updates performed at a single scale require

time.

Data structure overhead. The navigation lists are maintained using balanced search trees. Each insertion or update of a navigation list requires an additional

overhead. Therefore, the total update time is

Moreover, at each of the $O(\log\Delta)$ scales, the algorithm computes distances only to the

points in the corresponding local set $Z_r$. Hence, the total number of distance computations is

And the scale is handled automatically by the insertion update.

References

• R. Krauthgamer and J. R. Lee, Navigating Nets: Simple Algorithms for Proximity Search. In Proceedings of the Fifteenth Annual ACM-SIAM Symposium on Discrete Algorithms (SODA 2004), pp. 798–807. PDF

Definition (Dynamic Sum-of-Radii Clustering). They studied the dynamic sum-of-radii clustering problem, defined as follows. The original input consists of a (possibly infinite) set \(V\) of potential clients or points, a finite set \(F \subseteq V\) of facilities with an opening cost \(f_j\) for each facility \(j \in F\), and a metric \(d\) over \(V\).

For the online input, a set \(C\) of live clients evolves over time: at each timestep \(t\), either a new client arrives and is added to \(C\), or a client from \(C\) departs and is removed from \(C\), a query is made for the approximate cost of an optimal solution (cost query), or a query asks for the entire current solution (solution query).

For the output, at each timestep the algorithm maintains a set of open facilities, each open facility \(j\) being associated to a radius \(R_j\), such that every client of \(C\) is covered, i.e., belongs to some open ball \(B(j, R_j)\), and the goal is to minimize the cost, namely, the sum over open facilities \(j\) of \(f_j + R_j\).

The dynamic sum-of-radii clustering problem can actually be interpreted as a special case of dynamic set cover.

The doubling dimension of a metric space \((V, d)\) is said to be bounded by \(\kappa\) if any ball \(B(x, r)\) in \((V, d)\) can be covered by \(2^\kappa\) balls of radius \(r/2\), as defined in the previous post based on Lee’s paper.

The algorithm designed by them answers cost queries in constant time and solution queries in linear time in the size of the solution, up to a factor of \(\log(W/f_{\min})\), where \(W\) is the diameter of the metric space and \(f_{\min}\) is the minimum opening cost of any facility.

Theorem. There exists an algorithm for the dynamic sum-of-radii clustering problem, when clients and facilities live in a metric space with doubling dimension \(\kappa\), such that at every timestep the solution has cost at most \(O(2^{2\kappa})\) times the cost of an optimal solution at that time, and such that the update time is \(O(2^{6\kappa} \log(W/f_{\min}))\), where \(W\) is the diameter of the space, \(n\) is the current number of clients, and \(f_{\min}\) is the minimum opening cost. A cost query can be answered in constant time, and a solution query in time \(O(s \log(W/f_{\min}))\), where \(s\) is the size of the output.

There exists a collection \(\Pi\) of pairs \(\langle j, r \rangle\) where \(j \in F\) and \(r\) is a non-negative integer, each with an associated area \(A(j, r)\) of \(V\), and an abstract tree \(T\) over \(\Pi\), with the following properties:

1. \(T\) has height \(O(\log(W/f_{\min}))\) and degree at most \(2^{4\kappa}\).
2. The collection \(\mathcal{A}\) of areas is a laminar family, its laminar structure is given by \(T\), and for each area, \(A(j, r) \subseteq B(j, 7 \cdot 5^r)\).
3. For any subset \(C\) of \(V\), there exists a collection \(S\) of areas covering \(C\) and whose cost,

is \(O(2^{2\kappa})\) times the optimal cost for \(C\).

So if you pay close attention, they are really using the scheme and the paradigm used by Lee and his co-author, really by maintaining a tree with nodes as a set of \(Z_{r/2}\) (namely, the points that are sufficiently close in the \(2^i\) net to the point that is being queried upon). But now the difficulty lies in the fact that we have to keep track of the opening cost, so we have to think of a heuristic there to come up with a good solution.

So the algorithm runs in two phases: one processes the data and finds a solution, and the other maintains the solution by the successive updates that come along the way. They start by discretizing a potential solution to the problem.

Lemma. For all \(C \subseteq V\), there exists a solution such that every ball \(B(j, R)\) has \(f_{\min} \leq f_j \leq R \leq 5 \cdot W\), the radius \(R\) is an integer power of 5, and the cost is \(O(\text{OPT})\).

Proof. Consider the unknown optimal solution. If some ball is such that \(\max(f_j, R) > W\), then replace the entire solution by a ball centered at the facility of cost \(f_{\min}\) and of radius \(W\). Else, for each ball \(B(j, R)\) of the optimal solution:

• if \(f_j > R\) then increase the radius of the ball from \(R\) to \(f_j\).
• Increase \(R\) to the smallest integer power of 5 that is greater than or equal to \(R\).

The new solution satisfies the desired constraints, and the cost has increased by a factor of 10 at most, which is not that hard to actually compute.

A logradius is an integer \(r\) such that \(f_{\min} \leq 5^r \leq 5 \cdot W\). Let \(\rho_{\min} = \lfloor \log_5 f_{\min} \rfloor\) and \(\rho_{\max} = \lceil \log_5 W \rceil\). Then the number of different logradii, \(\rho_{\max} - \rho_{\min} + 1\), is \(O(\log(W/f_{\min}))\) (they are the relevant scales that had been used in Lee’s paper, but the scales are maintained by a power of 5).

We construct a set \(\Pi\) of pairs \(\langle j, r \rangle\) where \(j\) is a facility and \(r\) is a logradius. For each logradius \(r \in [\rho_{\min}, \rho_{\max}]\):

• let \(J'_r = \{j \in F \mid f_j \leq 5^r \}\).
• let \(J_r\) be a maximal subset of \(J'_r\) such that any two facilities in \(J_r\) are at distance greater than \(5^{r+1}\).

Note that for \(r = \rho_{\max}\), the set \(J_r\) contains just one facility.

So after we have constructed the coarse leveled information, we are going to construct a tree here:

Hierarchical decomposition of \(\Pi\)

Construct an abstract tree \(T\) over \(\Pi\) as follows (with ties broken arbitrarily):

• the root of \(T\) is the unique pair \(\langle j, \rho_{\max} \rangle\).
• for all \(r < \rho_{\max}\) and \(j \in J_r\):
  • let \(j'\) be the facility of \(J_{r+1}\) closest to \(j\)
  • \[\text{parent}(j, r) \gets \langle j', r + 1 \rangle\]

Lemma (Nesting of balls). If \(\text{parent}(j, r) = \langle j', r + 1 \rangle\), then \(B(j, 7 \cdot 5^r) \subseteq B(j', 7 \cdot 5^{r+1})\).

Proof. We have \(\langle j, r \rangle \in \Pi\), so \(j \in J'_r \subseteq J'_{r+1}\). By the Covering property of \(\Pi\), the maximum distance from any point in \(B(j, 7 \cdot 5^r)\) to \(j'\) is \(d(j, j') + 7 \cdot 5^r \leq 5^{r+2} + 7 \cdot 5^r \leq 7 \cdot 5^{r+1}\).

Lemma. For any point \(p\) and radius \(r\), the set of pairs

has at most \(2^{(\alpha+1)\kappa}\) elements, where \(\kappa\) is the doubling dimension of the metric space.

Proof. By definition of doubling dimension, \(B(p, 2^\alpha \cdot 5^{r+1})\) can be covered by a set of at most \((2^\kappa)^{\alpha+1}\) balls of radius \((1/2) \cdot 5^{r+1}\). By the Separating property of \(\Pi\), any two pairs \(\langle j, r \rangle\) of \(\Pi(p, r)\) are at distance greater than \(5^{r+1}\) from each other, hence must belong to different balls of the set, and so \(\Pi(p, r)\) has cardinality at most \((2^\kappa)^{\alpha+1}\).

Lemma. A node \(\langle j, r \rangle\) of \(T\) has at most \(2^{4\kappa}\) children.

Proof. Children of \(\langle j, r \rangle\) have logradius \(r - 1\), so by the Covering property of \(\Pi\) their distance to \(j\) is at most \(5^r\). Thus, they belong to \(\Pi(j, r - 1)\) for \(\alpha = 3\), and so Lemma 4 applies.

So, now just like the sets \(Z\) that have been maintained in the data structure, they now construct a set of related areas.

Recall that a collection \(\mathcal{A}\) of sets is laminar if for any two \(A, B \in \mathcal{A}\), either \(A \cap B = \emptyset\) or \(A \subseteq B\) or \(B \subseteq A\). We partition \(V\) into a laminar family of areas, denoted by \(\mathcal{A}\), such that no two same-logradius areas overlap.

Algorithm: Constructing Laminar Areas

1. For all \(\langle j, r \rangle \in \Pi\): \(A(j, r) \gets \emptyset\).
2. For all points \(p \in V\): a. \(r^* \gets \min \{ r \mid \exists \langle j, r \rangle \in \Pi \text{ such that } p \in B(j, 7 \cdot 5^r) \}\). b. \(\langle j^*, r^* \rangle \gets \arg\min_{\langle j, r^* \rangle} d(p, j)\) among all valid pairs with logradius \(r^*\). c. Add \(p\) to \(A(j^*, r^*)\). d. For all \(\langle j', r' \rangle\) that are ancestors of \(\langle j^*, r^* \rangle\) in \(T\):
   • Add \(p\) to \(A(j', r')\).

Lemma. For every \(\langle j, r \rangle \in \Pi\), \(A(j, r) \subseteq B(j, 7 \cdot 5^r)\).

Proof. Let \(p \in A(j, r)\). Either it has been added directly, in which case it belongs to \(B(j, 7 \cdot 5^r)\), or it has been inherited, in which case it also belongs to it by Lemma 3.

Lemma. For every subset \(C \subseteq V\) of clients there exists \(S \subseteq \Pi\) such that \(C\) is covered by \(\cup \{A(j, r) : \langle j, r \rangle \in S\}\) and

or:

Proof. Let \(S^*\) be a solution of cost \(O(\text{OPT})\) satisfying the properties of Lemma 2. For each ball \(B(j, 5^r)\) of \(S^*\), put in \(S\) all the pairs \(\langle j', r \rangle \in \Pi\) such that \(d(j, j') \leq 8 \cdot 5^r\).

We claim that \(C\) is covered by \(\cup \{A(j', r) : \langle j', r \rangle \in S\}\). Indeed, consider a client \(p \in C\) and a ball \(B(j, 5^r)\) of \(S^*\) containing \(p\). By the Covering property of \(\Pi\), there exists \(\langle j', r \rangle \in \Pi\) with \(d(j, j') \leq 5^{r+1}\). Then \(d(p, j') \leq 5^{r+1} + 5^r < 7 \cdot 5^r\), and so in the definition of areas covering \(p\) we must have \(r^* \leq r\). Along the path from \(\langle j^*, r^* \rangle\) to the root of \(T\), there exists a pair for logradius \(r\), \(\langle j'', r \rangle\). By definition of areas and by Lemma 6, \(p \in A(j'', r) \subseteq B(j'', 7 \cdot 5^r)\), so \(d(j, j'') \leq d(j, p) + d(p, j'') \leq 8 \cdot 5^r\), and therefore \(\langle j'', r \rangle \in S\) and \(p\) is covered.

In terms of costs, since all these areas are associated to pairs within distance \(8 \cdot 5^r < 2 \cdot 5^{r+1}\) from \(j\), by Lemma 4 for \(\alpha = 1\), there are at most \(2^{2\kappa}\) of them.

For one optimal ball \(B(j, 5^r)\), the proof constructs at most \(2^{O(\kappa)}\) nearby pairs. Each pair contributes

so the total charge is

Using the equivalence above,

which is exactly a constant-factor multiple of the cost of the original optimal ball. Therefore, summing over all balls in \(S^*\),

Now, a simple dynamic bottom-up DP approach is used to compute the best possible costs within our coarsened information tree.

Computing the optimal solution to the restricted problem in an offline manner is straightforward, thanks to the laminar structure of the candidate areas. We first compute, for each node \(\langle j, r \rangle\) of \(T\), the cost \(c_{j,r} = f_j + c_2 \cdot 5^r\) of area \(A(j, r)\), as well as the number \(n_{j,r}\) of clients that are in area \(A(j, r)\) but not in any of the areas of children nodes: since areas \(A(j', r-1)\) are all disjoint by laminarity, we have

We then compute the optimal cost \(x_{j,r}\) of covering the clients of \(C \cap A(j, r)\) using only areas of the subtree of \(T\) rooted at \(\langle j, r \rangle\), using the following bottom-up recurrence:

For \(\langle j, r \rangle \in \Pi\) in bottom-up order in \(T\):

The cost of the optimal restricted solution is then \(x_{j, \rho_{\max}}\) for the root \(\langle j, \rho_{\max} \rangle\) of \(T\). Given \(c_{j,r}\) and \(x_{j,r}\), computing the optimal restricted solution, a collection \(S\) of areas, is done recursively:

Thus the algorithm to compute the optimal set \(S\) of areas covering \(C\) in the restricted problem, given the values of \(c_{j,r}\) and \(x_{j,r}\), explores a tree \(T'\) that, as it is a partial subtree of \(T\), also has height at most \(O(\log(W/f_{\min}))\) and degree at most \(2^{4\kappa}\); moreover its internal nodes are all ancestors of areas added to the solution \(S\), so the running time to compute \(S\) itself is \(O(2^{4\kappa} \log(W/f_{\min}) \lvert S \rvert)\).

The algorithm will maintain two dynamic data structures:

1. a list of the currently existing clients \(C \subseteq V\), with, for each client \(p\), the \(\langle j, r \rangle \in \Pi\) such that \(p \in A(j, r)\) and \(r\) is minimum; and
2. an annotated dependency tree \(T_A\), keeping for each node \(v = \langle j, r \rangle\) the following additional information: a. its cost \(c_v = f_j + 7 \cdot 5^r\), b. the number \(n_v\) of currently existing clients that belong to \(A(j, r)\) but not to any descendant area, c. the value \(x_v\), which is the minimum cost needed to cover all clients belonging to \(A(j, r)\) using only areas \(A(j', r')\) for \(\langle j', r' \rangle \in \Pi\), and d. the value \(y_v = \sum_{u \text{ child of } v} x_u\).

When a client \(p\) is inserted, we first find \(\langle j, r \rangle\) in \(T_A\), such that \(p \in A(j, r)\) and \(r\) is minimum, in a way to be described shortly; we increment \(n_v\), and then we traverse the path from \(\langle j, r \rangle\) up to the root of \(T_A\), similarly updating \(x_v\) and \(y_{\text{parent}(v)}\).

Thus, it only remains to determine the pair \(\langle j^*, r^* \rangle\) with smallest logradius such that \(p \in A(j^*, r^*)\). By Lemma 6, \(p \in B(j^*, 7 \cdot 5^{r^*})\). Thus we will first find all pairs \(\langle j, r \rangle\) such that \(p \in B(j, 7 \cdot 5^r)\), based on them determine \(r^*\), and then look for \(\langle j^*, r^* \rangle\) in that set of balls. Thanks to Lemma 4, the first part can be done using a simple recursive algorithm starting from the root of \(T_A\) (see below). The second part simply uses the definition of areas, i.e., it finds the pair \(\langle j^*, r^* \rangle\) where \(j^*\) has minimum distance to \(p\) out of all pairs \(\langle j, r^* \rangle\) with \(p \in B(j, 7 \cdot 5^{r^*})\).

Let \(r^*\) be minimum such that there exists pairs \(\langle j, r^* \rangle\) in the set \(\text{Pairs}(p, j_{\text{root}}, \rho_{\max})\). Among all such pairs, output the pair \(\langle j^*, r^* \rangle\) minimizing \(d(p, j^*)\).

The running time is dominated by the first part, which is \(O(2^{4\kappa})\) times the number of pairs \(\langle j, r \rangle\) such that \(p \in B(j, 7 \cdot 5^r)\). There are \(\log(W/f_{\min})\) possible values of \(r\). For each \(r\), by Lemma 4 there are at most \(2^{2\kappa}\) pairs \(\langle j, r \rangle \in \Pi\) such that \(p \in B(j, c_2 \cdot 5^r)\), and the algorithm has to test the \(O(2^{4\kappa})\) children of each of them. Thus the running time to do an insertion is \(O(2^{6\kappa} \log(W/f_{\min}))\).

References

• M. Henzinger, D. Leniowski, and C. Mathieu, Dynamic Clustering to Minimize the Sum of Radii. ESA 2017. arXiv:1707.02577

Dynamic and Massively Parallel, and Streaming Facility Location via Geometric Hashing

So this is going to be a template for a future post: I have been reading the papers Tree Embedding in High Dimensions: Dynamic and Massively Parallel and Streaming Facility Location in High Dimension via Geometric Hashing. So, I am going to make some sense out of these papers, and hopefully, I will be able to use the geometric hashing method to come up with ideas and problems on how to tackle them and how to work through a problem-solving session. Well, both problems at hand seem very geometrical, and the existence of the hash functions really makes life easier, so if we have hash functions based on the ideas of these papers, we can do two things: firstly, we can run a sort of importance sampling, as in discretizing settings, where the balance of the density of the points may not be equal, so in the case that uniform sampling does not work. Secondly, we can find the related points in a dynamic setting and do some calculations thereafter, so in a geometric problem, for example, such as k-means or metric embeddings, we can use the inherent nature of the hash functions to run our algorithm in a dynamic setting, albeit in a probabilistic setting. Let’s first mention some theorems and contemplate them.

Definition ($(\Gamma,\Lambda)$-hash function). A (randomized) hash function

is called a $\Gamma$-gap $\Lambda$-consistent hash with diameter bound $\tau>0$, or simply a $(\Gamma,\Lambda)$-hash function, if it satisfies the following properties:

1. (Diameter) For every image $z \in \phi(\mathbb{R}^d)$, \(\operatorname{diam}\!\left(\phi^{-1}(z)\right) \le \tau.\)

2. (Consistency) For every set $S \subseteq \mathbb{R}^d$ with \(\operatorname{diam}(S) \le \frac{\tau}{\Gamma},\) we have \(\mathbb{E}\!\left[\,|\phi(S)|\,\right] \le \Lambda.\)

Theorem 4.2.

Assume there exists a \((\Gamma,\Lambda)\)-hash \(\phi : \mathbb{R}^d \to \mathbb{R}^d\) with diameter bound \(\tau\) such that for \(p \in \mathbb{R}^d\), the hash value \(\phi(p)\) and the set of hash values \(\phi(B(p,\tau/\Gamma))\) can be evaluated in \(O(\mathrm{poly}(d))\) and \(O(|\phi(B(p,\tau/\Gamma))|\cdot \mathrm{poly}(d))\) time, respectively.

Then there exists a dynamic algorithm which, for a dynamic set of points \(P \subseteq \mathbb{R}^d\) with \(|P|\le n\) undergoing point insertions and deletions, maintains a tree embedding of \(P\) with \(O(\Gamma \log \Gamma \log n)\) distortion in \(\widetilde{O}(d+\Lambda)\) expected amortized update time.

The underlying tree embedding is rebuilt by the algorithm after every $n$ updates. An update to the input points $P$ results in \(\widetilde{O}(1)\) expected updates to the tree embedding of the following types:

1. Type 1: A leaf of the embedding becomes inactive in the sense that no point in $P$ corresponds to it.
2. Type 2: A new leaf and a path connecting the leaf to an existing node in the tree embedding are inserted into the embedding.

Theorem 3.1. There is a one-pass randomized algorithm that, given $P \subseteq [\Delta]^d$ presented as a dynamic geometric stream, samples a random point $p^* \in P \cup {\bot}$ such that

and also reports a 2-approximation $\widehat{\Pr}[p^*]$ for the probability of sampling this point, i.e.,

This algorithm uses $\mathrm{poly}(d \cdot \log \Delta)$ bits of space, and fails with probability at most $1/\mathrm{poly}(\Delta^d)$.

Symmetrization, Generic Chaining, and Concentration · Chapter 3

How to compute the things?

So in this post I am going to recite some of the results of the paper in some generality. As I was going through it, there are still some concepts I have difficulty grasping — like the clustering in generic chaining, or the concentration of measure, and the convex body inequalities that are existing within the paper. More specifically, I think I should study Milman’s paper for a more fundamental understanding; and besides their own paper, I wonder how best to study and deepen my understanding further.

Well, let’s go through the inner dynamics of the paper, to learn something out of it. We must be aware that some classes of norms are studied because they can guarantee some nice analytic results, and in a way they give us control over the problem of sparsification — otherwise we would be hopeless.

In the paper, lots of concentration results and convex body inequalities are used. Most importantly, I would say that this paper is a nice application of the concentration of measure phenomena. Everything else is built around it. I will use one early theorem in Talagrand’s book Probability in Banach Spaces to showcase the first principles concerning some inequalities used. The base observation is that, in really high dimensions, behavior is in a sense deterministically random — the law of large numbers is one such example.

Ok, let’s go through the paper:

The base observation is that we want to sparsify the norms based on importance sampling (which is a general scheme in solving several sparsification problems; for example you can study the paper Streaming Facility Location in High Dimension via Geometric Hashing to see another nice application of this scheme).

More formally:

Consider a probability distribution $\rho = (\rho_1, \ldots, \rho_m) \in (0,1]^m$ on ${1,\ldots,m}$, and sample $M$ indices $i_1,\ldots,i_M$ independently from $\rho$. Define

We have

and therefore

for any fixed $x$.

So the whole paper is now about concentration, and controlling the behavior of probabilities in some way.

The authors use a nice application of symmetrization (which is heavily used in Talagrand’s book) to show that, basically, the sparsification of norms can be reduced to a stochastic process of Bernoulli variables, which is sub-Gaussian with respect to the norm — hence you can use several theorems of generic chaining to provide some sort of concentration results.

We want to control the following:

More formally, consider this. Fix a metric space $(T,d)$. A random process ${V_x : x \in T}$ is said to be subgaussian with respect to $d$ if there exists a constant $\alpha > 0$ such that for all $x,y \in T$ and all $t > 0$,

We say that ${V_x : x \in T}$ is centered if

Given a metric space $(T,d)$, define the ball

For a subset $\Omega \subseteq \mathbb{R}^n$, we use the notation

The authors really consider the following set, which is ok because everything else works up to scaling:

So how does symmetrization work?

Note that

Thus, for any convex function $\psi : \mathbb{R}+ \to \mathbb{R}+$,

where $\tilde{\nu}$ is an independent copy of $\nu$.

The argument inside $\psi$ on the right-hand side can be written as

Since the distribution of

is symmetric, we have

for any choice of signs $\varepsilon_1,\ldots,\varepsilon_M \in {-1,1}$.

This yields the stochastic domination

So, basically, we can now study the sub-Gaussian stochastic process as we try to randomly sparsify the norms and then find some sort of result to bound the concentration.

And they go on to prove the following lemma:

Lemma 2.6

Let $M \ge 1$, let $\Omega \subseteq \mathbb{R}^n$, and let $\rho \in \mathbb{R}_+^m$ be a probability vector. Assume that

Suppose further that for some $0 < \delta \le 1$, and every $\nu \in [m]^M$, it holds that

If $\nu_1,\ldots,\nu_M$ are sampled independently from $\rho$, then

If it also holds that, for all $\nu \in [m]^M$,

then there exists a universal constant $K>0$ such that for all $0 \le t \le \frac{1}{2K\hat{\delta}}$,

let me first examine a nice application of the concentration of measure, which roughly suggests that a function behaves like a constant on almost all of the space.

Theorem 1.1

If $A$ is a Borel set in $S^{N-1}$ and $H$ is a cap (i.e. a ball for the geodesic distance $p$) with the same measure

then for any $r > 0$,

where we recall that

is the $r$-neighborhood of $A$ for the geodesic distance.

The Isoperimetric Problem. Given $a \in (0,1)$ and $\varepsilon > 0$, the isoperimetric problem asks to determine the Borel subsets $A \subseteq X$ satisfying

for which the measure of the $\varepsilon$-neighborhood

is as small as possible, where

Equivalently, one seeks to solve

So the theorem that is mentioned in the start of the book of the Talagrand’s mentions a rather nice theorem about the set that has the lowest expansion, among other sets. But what is the significance of studying the isoperimetric problem in the $S^{n-1}$? it is useful because by using Poincaré limit, or rather by a nice application of the law of large numbers, we can relate studying the inequalities in the Sphere down to the $\mathbb{R}^d$ using projection, and excessive larger Dimension of the sphere.

what I mean by that is if we consider:

For every $N$, let $U^{N-1}$ denote the normalized uniform probability measure on the sphere

Let $\Pi_{N,d}$, with $N\ge d$, denote the canonical projection from $\sqrt{N}\,S^{N-1}$ onto $\mathbb{R}^d$, namely,

Then the sequence of probability measures

on $\mathbb{R}^d$ converges weakly, as $N\to\infty$, to the canonical Gaussian measure on $\mathbb{R}^d$.

Let

where $G_1,\ldots,G_N$ are independent and identically distributed random variables with

Define

Then $X$ is uniformly distributed on the sphere $\sqrt{N}\,S^{N-1}$.

Now write

By the Law of Large Numbers,

Hence,

Since $G_i$ is asymptotically independent of the denominator (or, equivalently, by Slutsky’s theorem),

where $I_d$ denotes the $d\times d$ identity matrix. furthermore The concentration of measure phenomenon is thus usually derived from an isoperimetric inequalities. to illustrate we use an example from Assaf Naor lecture notes about concentration of the measure.

Lemma

Let $(X,d)$ be a metric space equipped with a Borel probability measure $\mu$. Let $f:X\to\mathbb{R}$ be an $L$-Lipschitz function. Then there exists $M\in\mathbb{R}$ such that, for every $\varepsilon>0$,

Proof. Let $M$ be a median of $f$, namely,

Set

If $x\in A_{\varepsilon/L}$, then there exists $y\in A$ such that

Since $f$ is $L$-Lipschitz,

As $y\in A$, we have $f(y)>M$, and therefore

Hence

Since $\mu(A)\ge\frac12$, the definition of the isoperimetric profile $\phi_{1/2}$ yields

Similarly, let

Then $\mu(B)\ge\frac12$, and the same argument gives

Finally,

where the two sets are disjoint. Therefore,

which completes the proof. $\square$

so lets get back to an application of the first theorem, on how to make sense of these things.

Denote by $\gamma_N$ the canonical Gaussian probability measure on $\mathbb{R}^N$, Denote further by $\Phi$ the distribution function of $\gamma_1$, i.e.,

Theorem 1.2

If $A$ is a Borel set in $\mathbb{R}^N$ and if

is a half-space, where $u\in\mathbb{R}^N$, $|u|=1$, and $\lambda\in(-\infty,+\infty)$, with the same Gaussian measure

then, for every $r>0$,

where

is the Euclidean neighborhood of order $r$ of $A$.

Equivalently,

In particular, if

then

where

is the Gaussian tail function.

so a half space has the least enlargement over the other sets with equal measure, we can suppose that enlargement works in a single dimension, because everything is rotationally invariant. So we can suppose that the half plane is orthogonal to an axis.

Now,

Therefore,

Since

we obtain

Since

this becomes

Now combine this with

Since $\Phi^{-1}$ is increasing, we obtain

define

Since

they may assume that $a > -\infty$.

As we have shown, Poincaré’s argument says that as $k \to \infty$, the first coordinates become approximately standard Gaussian:

Because $b < a$, as $k \to \infty$,

while

Hence for $k$ large enough,

It is easy to see that

where the neighborhood of order $r$ on the right is understood with respect to the geodesic distance on $\sqrt{k}\,S^{k-1}$.

Since $\Pi_{k,1}^{-1}((-\infty,b])$ is a cap on $\sqrt{k}\,S^{k-1}$, by the isoperimetric inequality on spheres, we obtain

Now,

for some $r(k) \ge 0$ satisfying

Therefore, in the Poincaré limit, we get

and hence the result since

is arbitrary.

so really it is a variant of limiting argument and an application of law of large numbers, plus the geometry of sphere. So use the rotational invariance of Gaussian product measures, and basically go to the sphere to prove the inequality for large enough K, as you get the same measure for really high dimension, so it pinpoints that you may use a limiting argument.

so lets back to Lee’s paper.

Let $(V,\mathrm{dist})$ be a metric space. Define

For a subset $P\subseteq V$, define

Let $\pi:P\to [0,1]$ be a random map, and let $r\sim \mathrm{Unif}!\left[\frac{w}{4},\frac{w}{2}\right]$.

For each $p\in P$, define

A $\tau$-bounded decomposition ($\tau$-BD) is a partition of $V$ such that every part has diameter at most $\tau$.

Let $\phi:V\to U$ be a hash function. For $x\in V$, define its bucket

and for any set $S\subseteq V$,

Define the bucketed neighborhood

Finally, define

Assumptions and Parameters

For the sake of presentation( as in the paper ), we assume, without loss of generality, that the dataset $P\subseteq V$ has the smallest inter-point distance greater than $1$, and that its diameter $\Delta := \operatorname{diam}(P)$ satisfies

for some $m\in\mathbb{N}$. For every integer $i$, let $w_i := 2^{m-i}$.

Well recently I have come across a really cool paper, Tree Embedding in High Dimensions: Dynamic and Massively Parallel, which is a nice and cool utilization of geometric hashing which in turn is itself a sparse partition of the space and metric space at hand. Basically how the algorithm at the basic level works is the same as its predecessors like the CKR process: put a randomization forward, break down the metric space in a recursive ( in predecessors) manner, and the structure and analysis help you get an ultrametric with distortion of $O(\log n)$. But the novelty of their paper lies in the fact that they propose a cool dynamic algorithm, work in a more general setting, and make novel use of the dynamic paradigm, which also works extremely well in the MPC agenda, a distributed version of streaming where parallelism replaces sequential access. Without consistent hashing or geometric hashing such a thing may not have been possible.

So the intuition is really there: find a good neighborhood of points, put a random priority map, decompose the labels in the appropriate buckets according to distance at each level. But it takes a more sophisticated approach which I am going to explain. Instead of a top-down recursive approach a rather greedy paradigm which in this setting may force more computational power we build it independently at each level. This helps in the design of the dynamic algorithm.

So, the base intuition is there, but I may start with some technical analysis like the authors did or just outright explain the algorithm. Explaining the analysis is more revealing, so we start with that.

It is a standard fact that the distance between points $p,q$ in the tree $T$ is determined by their LCA (least common ancestor) in the tree. The following notation defines the level of the LCA (minus $1$) with respect to the labels $\ell$.

Algorithm 1: Tree embedding on input $P \subseteq V$ with access to metric hashes ${\phi_i}_{i\in[m]}$

Note: $\phi_i$ is a metric hashing with diameter bound $\tau_i := w_i/2$

1. Sample $\beta \in \left[\tfrac{1}{4}, \tfrac{1}{2}\right]$ uniformly at random
2. Let $\pi$ be a uniform random map from $P$ to $[0,1]$
3. For $i \leftarrow 1$ to $m$:
   • $r_i \leftarrow \frac{\beta}{\Gamma} \cdot w_i$
   • For each \(p \in P\):
     • \[\ell^{(i)}_p \leftarrow \pi_{\min}\!\big(\widetilde{B}^{P}_i(p,r_i)\big)\]
4. Return \(\{\ell^{(i)}_p\}_{i\in[m],\,p\in P}\)

Definition (Tree level). For $p,q\in P$, define \(\operatorname{lv}(p,q) := \min\left\{\, i\in [m] : \ell^{(i)}_{p}\neq \ell^{(i)}_{q} \,\right\}.\)

Since the embedding tree is a $2$-HST, it is immediate that

The embedding tree $T$ has height $m$, and the edge weight from every level-$i$ node to every level-$(i+1)$ node is $w_i$. Here, level $1$ is the root (which consists of a singleton partition), and level $m$ consists of the leaves, each of which contains at most a single data point.

Fix $p,q\in P$, and let

so that we are one level higher than the separator in the tree. Since the embedding tree is a $2$-HST, we have

Moreover, by the definition of $\operatorname{lv}(p,q)$, we have $\ell^{(i)}_p=\ell^{(i)}_q$. Let

the image belongs to CSC2414 - Metric Embeddings Lecture 3: Embedding to Random Trees Notes taken by Nilesh Bansal and Ilya Sutskever Revised by Hamed Hatami.

Then we also have $p’=\pi^{-1}!\left(\ell^{(i)}_q\right)$, because this follows directly from the definition. In other words, $p$ and $q$ are placed in the same bucket at the upper levels; namely, the hash functions classify them identically.

Therefore,

What we are really doing is taking the neighborhood of a point and using the sparsification palette (or agenda) induced by the hashing functions. We then label the point according to the highest-priority point that lies in the corresponding buckets, according to the hash function, and that also belongs to $P$.

Hence, by the definition of $\widetilde{B}_i^P$, we can see that

so the claim is proved. We use this fact to obtain an upper bound on $\operatorname{dist}(p,q)$.

Now, let $\hat{p}$ be an arbitrary point in $\operatorname{buks}_i(p’) \cap B(p,r_i)$, and let $\hat{q}$ be an arbitrary point in $\operatorname{buks}_i(p’) \cap B(q,r_i)$.

Since $\hat{p}$ and $\hat{q}$ belong to the same bucket $\operatorname{buks}_i(p’)$, i.e., $\phi_i(\hat{p})=\phi_i(\hat{q})$, by the diameter bound of $\phi_i$ we have

Also observe that $\operatorname{dist}(p,\hat{p})\le r_i$ and $\operatorname{dist}(q,\hat{q})\le r_i$. Hence,

The proof of

is interesting in its own right, but it is too technical for a general reader. Nevertheless, it can be studied to prove similar results and is quite helpful in those settings. Therefore, I omit it from this post and may discuss it in a future post.

However, we now turn to Lemma 3.3, which is used for the distortion result.

Lemma 3.3. For every $p,q\in P$, $i\in[m]$, and $0 \le r’ \le \tfrac{1}{2} r^{\max}_i$ such that $B(p,r’) \subseteq B!\left(q,\tfrac{1}{2} r^{\max}_i\right)$, it holds that

\[\Pr\!\big[\ell^{(i)}_p \ne \ell^{(i)}_q\big] \le 8 \cdot \frac{\operatorname{dist}(p,q)}{r^{\max}_i} \left( H_{|\widetilde{B}^{P}_i(p,r^{\max}_i)|} - H_{|\widetilde{B}^{P}_i(p,r')|} \right).\]

The dynamics of the proof of this lemma are really interesting and revealing, so I will dedicate a separate post to it I find it the most sophisticated part of the paper.

This lemma is particularly revealing and intuitive in parts, as it captures how the separation probability depends on the local growth of the bucketed neighborhood structure. But the neighborhood definition which the authors defined was so hard to work with in analysis so they came up with a rather nice object to solve the problem.

Their solution was to come up with the representative set, which is a way to keep the geometric structure nice and steady ball-like even while at the same time maintaining crucial information necessary for the analysis of the algorithm, there have been another problem in finding the upperbounds for probabilities which they solved by defining axuliry events, these two key observations are rather interesting.

I will try to explain these concepts.

Representative Sets

Their proof strategy is to utilize the structure of $\widetilde{B}^{P}_i(\cdot,\cdot)$, and argue that a carefully chosen set of representatives of $\widetilde{B}^{P}_i(p,r)$ has similar geometric properties kind of like a standard ball.

Definition (Representative sets). A collection of sets $\operatorname{rep}_i(p,r) \subseteq V$ (which may not necessarily be a subset of $P$), defined for every $p\in P$, $i\in[m]$, and $r\ge 0$, is called a family of representative sets if for every $i\in[m]$, $p\in P$, and $r\ge 0$, the following properties hold:

1. (distinct) For all $x \neq y \in \operatorname{rep}_i(p,r)$, \(\operatorname{buks}_i(x) \cap \operatorname{buks}_i(y) = \emptyset.\)

2. (monotone) For all $r’ \in (0,r)$, \(\operatorname{rep}_i(p,r') \subseteq \operatorname{rep}_i(p,r).\)

3. (ball-preserving) \(\operatorname{rep}_i(p,r) \subseteq B(p,r)\cap \operatorname{buks}_i(P),\) and \(\operatorname{buks}^{P}_i\!\big(\operatorname{rep}_i(p,r)\big) = \operatorname{buks}^{P}_i\!\big(B(p,r)\big).\)

Property 1 is useful for several results in the upcoming analysis. Properties 2 and 3 ensure that the representative sets behave similarly to metric balls, so that no essential information is lost in the abstraction.

Lemma. There exists a family ${\operatorname{rep}_i(p,r) : i\in[m],\, p\in P,\, r\ge 0}$ satisfying the definition above.

Proof. We introduce the following notation to define $\operatorname{rep}$.

For each $i\in[m]$, $p\in P$, and $x\in V$, define

This is the nearest point in $\operatorname{buks}_i(x)$ to $p$, which may not necessarily lie in $P$. Then, for each $i\in[m]$, $p\in P$, and $S\subseteq V$, define

Note that for any $x,y\in S$, if $\phi_i(x)=\phi_i(y)$, then $\operatorname{NNbuks}_i(p,x) = \operatorname{NNbuks}_i(p,y)$, so $\operatorname{NNbuks}_i(p,S)$ contains at most one point from each bucket.

Definition (Key representative). For each $i\in[m]$, $p\in P$, and $r>0$, define $\operatorname{rep}^{*}_i(p,r)$ as the minimum point in $\operatorname{rep}_i(p,r)$ with respect to $\pi$:

\[\operatorname{rep}^{*}_i(p,r) := \arg\min \left\{ \pi_{\min}\!\big(\operatorname{buks}^{P}_i(x)\big) : x \in \operatorname{rep}_i(p,r) \right\}.\]

Note that $\operatorname{rep}^{*}_i(p,r)$ is unique, since all values of $\pi$ are distinct with probability $1$.

Moreover, this key representative precisely realizes the minimizer of the $\pi$-value within $\operatorname{rep}_i(p,r)$:

To give a rough sketch, we define the probability of separation based on the key representative, and introduce auxiliary events to analyze the resulting probabilities.

First, notice that the event $\ell^{(i)}_p \neq \ell^{(i)}_q$ is equivalent to either $\ell^{(i)}_p < \ell^{(i)}_q$ or $\ell^{(i)}_p > \ell^{(i)}_q$, and these two events are symmetric. Therefore, it suffices to bound $\Pr\big[\ell^{(i)}_p < \ell^{(i)}_q\big]$.

Specifically, we show that the event $\ell^{(i)}_p < \ell^{(i)}_q$ implies a geometric fact about the key representative; namely,

To see this, by the key representative identity and Algorithm 1 (line 5), we have

This implies that

which leads to $\operatorname{rep}^{*}_i(p,r_i) \notin B(q,r_i)$. Therefore,

Analyzing the probability via an auxiliary event

One difficulty in analyzing $\Pr[\operatorname{rep}^{*}_i(p,r_i) \notin B(q,r_i)]$ is that it depends on two sources of randomness: the hash function $\pi$ and the random radius $r_i$. To separate these sources, we define an auxiliary event $E$, whose randomness depends only on $\pi$.

This event is defined with respect to $i\in[m]$, $p\in P$, and $x\in V$. It captures whether $x$ achieves the minimum $\pi$-value in $\widetilde{B}^{P}_i(p,r)$, where $r=\operatorname{dist}(p,x)$.

Definition (Auxiliary event). For every $i\in[m]$, $p\in P$, and $x\in V$, define the event (with respect to the randomness of $\pi$)

\[E^{(i)}_p(x) := \left\{ \pi_{\min}\!\big(\operatorname{buks}^{P}_i(x)\big) = \pi_{\min}\!\big(\operatorname{buks}^{P}_i(\operatorname{rep}_i(p,r))\big) \right\},\]

where $r = \operatorname{dist}(p,x)$.

Now we turn to the most interesting part of the paper.

Definition ($(\Gamma,\Lambda)$-hash function). A (randomized) hash function $\phi : \mathbb{R}^d \to \mathbb{R}^d$ is called a $\Gamma$-gap $\Lambda$-consistent hash with diameter bound $\tau>0$, or simply a $(\Gamma,\Lambda)$-hash function, if it satisfies:

1. (Diameter) For every image $z \in \phi(\mathbb{R}^d)$, \(\operatorname{diam}(\phi^{-1}(z)) \le \tau.\)

2. (Consistency) For every $S \subseteq \mathbb{R}^d$ with $\operatorname{diam}(S) \le \tau/\Gamma$, \(\mathbb{E}\big[|\phi(S)|\big] \le \Lambda.\)

In words, this definition formalizes a geometric hashing scheme in which small-diameter sets are expected to remain concentrated under the hash, while each bucket itself has bounded diameter.

Without the notion of consistent hashing, it would not be possible to implement the algorithm in an MPC model, where computation may be performed in a streaming fashion as mentioned and implemented in the original papers.

Theorem 4.2. Assume there exists a $(\Gamma,\Lambda)$-hash function $\phi : \mathbb{R}^d \to \mathbb{R}^d$ with diameter bound $\tau$, such that for $p \in \mathbb{R}^d$, the hash value $\phi(p)$ and the set of hash values $\phi(B(p,\tau/\Gamma))$ can be evaluated in $O(\mathrm{poly}(d))$ time and $O(|\phi(B(p,\tau/\Gamma))|\cdot \mathrm{poly}(d))$ time, respectively.

Then there exists a dynamic algorithm which, for a dynamic set of points $P \subseteq \mathbb{R}^d$ with $|P|\le n$ undergoing point insertions and deletions, maintains a tree embedding of $P$ with $O(\Gamma \log \Gamma \log n)$ distortion and $\tilde{O}(d+\Lambda)$ expected amortized update time.

The underlying tree embedding is rebuilt by the algorithm after every $n$ updates. An update to the point set $P$ results in $\tilde{O}(1)$ expected updates to the tree embedding of the following types:

• Type 1: A leaf of the embedding becomes inactive, in the sense that no point in $P$ corresponds to it.
• Type 2: A new leaf and a path connecting the leaf to an existing node in the tree embedding are inserted into the embedding.

Algorithm 2: Insertion-procedure$(p)$

Note: $\phi_i$ is a metric hashing with diameter bound $\tau_i := w_i/2$

1. Draw $\pi(p)$ and compute $\phi_i(p)$ for all $i \in [m]$
2. For $i \leftarrow 1$ to $m$:
   • Compute hash values $X_i \leftarrow {\phi_i(x) \mid x \in B(p,r_i)}$
   • (Computing label $\ell^{(i)}_p = \pi_{\min}(\widetilde{B}^{P}_i(p,r_i))$)
   • Update \(\phi_i^{-1}(x)$ and $\pi_{\min}(\phi_i^{-1}(x)\cap P)\) for all \(x \in X_i\)
   • \[\ell^{(i)}_p \leftarrow \pi_{\min}\!\left(\bigcup_{x\in X_i} \widetilde{B}^{P}_i(p,r_i)\right)\]
   • (Updating labels $\ell^{(i)}_q$ for $q \in P$)
   • If $\pi(p) = \pi_{\min}(\phi_i^{-1}(\phi_i(p)))$:
     • For all $x \in X_i$ and all $p’ \in \phi_i^{-1}(x)\cap P$:
       • If \(\pi(x) < \ell^{(i)}_{p'}\): update \(\ell^{(i)}_{p'} \leftarrow \pi(x)\)

So based on the theorems of consistent hashing we can provide the preferred amortized upper bound on the insertion.

But how does the dynamic algorithm work? It basically follows the intuition of the previous algorithm in the first half, by finding the path and labels of the inserted point, and the second half updates the path for other nodes in $P$, by altering the labels which need to change.

What Is This Paper About?

Randomized sampling is a very powerful tool in the literature. There are many instances where the number of rows vastly exceeds the dimension of interest. The most obvious and easy such problem is Least Squares: if $d \gg n$, we can sample enough rows and keep going with our life while finding a good minimizer that approximates the original one. So how do we approach this algorithmically?

For the case where the norm is $p = 2$, we have the concept of leverage scores, which assign a score to each row based on its importance. We then sample each row a proportional number of times to get an unbiased estimator. The key step is then proving some sort of concentration inequality just like Johnson–Lindenstrauss and showing that with high probability we get a good approximation, to get that we actually need to have all the basises and directions to have a uniform impact on the question at hand, and in this setting a statistical white transform is utilized.

1. Whitening as a Change of Coordinates

In statistics and machine learning, whitening means transforming data so that its covariance becomes the identity. Good expositions are Andy Jones’s notes on statistical whitening and Dustin Stansbury’s post The Statistical Whitening Transform on The Clever Machine.

Suppose $X \in \mathbb{R}^{n \times p}$ has (centered) covariance $\Sigma = \frac{1}{n} X^\top X$. A whitening matrix $W$ satisfies

so the whitened data $Y = XW$ has identity covariance. One standard choice is

where $\Sigma = U \Lambda U^\top$ is the eigendecomposition. Geometrically (as Stansbury explains): first rotate into the eigenbasis of $\Sigma$, then scale each direction by the inverse of its standard deviation. The result is a sphere of unit variance in every coordinate.

The key point for us is not PCA versus ZCA whitening, but the appearance of $\Sigma^{-1/2}$: a symmetric square root of the inverse covariance. That object reweights directions so that none dominates.

What Is a Leverage Score?

A leverage score is a kind of Statistical Whitening Transform.

In many modeling applications, data are transformed to have an identity covariance matrix, a process known as statistical whitening. After whitening, the features are uncorrelated and each dimension has unit variance.

Given a data matrix $X \in \mathbb{R}^{n \times p}$ with covariance matrix

a whitening transformation seeks a matrix $W$ such that the transformed data has identity covariance:

Consequently, the whitening matrix satisfies $WW^\top = \Sigma^{-1}$. After whitening, the variables are uncorrelated and each dimension has unit variance.

By making the data uncorrelated and viewing it in this special way which is the key intuition here we can change the density of rows and then sparsify the matrix.

The statistical leverage score of the $i$th row $a_i$ is defined as

Equivalently, it is the squared Euclidean norm of the $i$th row after applying the statistical whitening transform.

3. From $\ell_2$ Leverage to $\ell_p$ Lewis Weights

For general $p$, the analogue of leverage scores is less obvious because $|\cdot|_p$ is not induced by a single inner product in the same clean way. Lewis’s classical change-of-density construction (Lewis, 1978, Finite dimensional subspaces of $L_p$) produces weights $w_1, \ldots, w_n \ge 0$ such that an $\ell_p$ norm on rows can be studied through an associated $\ell_2$ geometry. Bourgain, Lindenstrauss, and Milman (1989, Approximation of zonoids by zonotopes) and Talagrand (1995, Embedding subspaces of $L_p$ in $\ell_p^N$) developed and used this framework in convex geometry and functional analysis.

On the algorithmic side, Cohen and Peng made Lewis weights into a practical row-sampling tool in $\ell_p$ Row Sampling by Lewis Weights (PDF). Their problem: given $A \in \mathbb{R}^{n \times d}$ with $n \gg d$, build a sparse matrix $A’$ from rescaled rows of $A$ such that

The sampling probabilities come from the $\ell_p$ Lewis weights, defined as follows.

Definition ($\ell_p$ Lewis weights).

Given $A \in \mathbb{R}^{n \times d}$, the Lewis weights $w_1, \ldots, w_n \ge 0$ are the unique weights such that, for $W = \operatorname{diag}(w_1, \ldots, w_n)$ and every row $i$,

\[w_i = \tau_i\!\left(W^{\,\frac12 - \frac1p} A\right).\]

When $p = 2$, this reduces to ordinary leverage scores (up to the normalization built into $W$). For general $p$, the matrix $W^{\frac12 - \frac1p} A$ is a $p$-dependent reweighting of rows before applying the whitening map $(\cdot)^\top A^\top W’ A (\cdot)$ in disguise. One can also characterize Lewis weights variationally: they arise from maximizing $\det(M)$ subject to $\sum_i (a_i^\top M a_i)^{p/2} \le d$, $M \succeq 0$ (see Lee’s Lewis weights notes).

Cohen and Peng show that sampling $\tilde{O}(\varepsilon^{-2} d \log d)$ rows with probabilities proportional to Lewis weights yields a $(1 \pm \varepsilon)$-approximate $\ell_p$ subspace embedding. That is the backbone for many fast regression and optimization algorithms.

Setup and Goal

The $\ell_p$-norm of a vector $x \in \mathbb{R}^d$ is defined as

For a matrix $A \in \mathbb{R}^{n \times d}$, let $a_i$ denote its $i$th row (viewed as a column vector). Then,

Throughout, we assume that $A$ has full column rank; otherwise, one may restrict to its column space or use the Moore–Penrose pseudoinverse.

For an $n \times d$ matrix $A$ with $n \gg d$ and $\varepsilon > 0$, the goal is to select a small set of (rescaled) rows to form $A’$ such that

for all $x \in \mathbb{R}^d$, where $\approx_{1+\varepsilon}$ denotes multiplicative error within a factor of $1 + \varepsilon$.

The main theorem we want to prove is the following but we consider the case $p = 1$ first.

Theorem. Given a matrix $A$ and $\varepsilon > 0$, there exists a distribution over sampling matrices $S$ with $O(d \log d \cdot \varepsilon^{-2})$ rows and one nonzero entry per row such that, with high probability,

\[\|SAx\|_1 \approx_{1+\varepsilon} \|Ax\|_1\]

for all $x \in \mathbb{R}^d$. Moreover, a sample from this distribution can be generated using $O(\log \log n)$ calls to a procedure that computes $2$-approximate statistical leverage scores of matrices of the form $WA$, where $W$ is a nonneg diagonal matrix.

The Key Intuitions from Talagrand

From the literature especially Talagrand’s papers we get two central intuitions:

1. Matrix concentration bound: uniform sampling works (for all $p$) when all leverage scores are close, i.e., $O(d/n)$.

2. $\ell_p$ whitening transformation: split $a_i$ into $w_i$ (fractional) “copies” with the same leverage scores.

The $\ell_p$ spaces are significantly more complex than $\ell_2$ and lack many of its useful properties. The Lewis weight framework can be viewed as a way to recover some of the structure of $\ell_2$ by associating to any matrix $A$ a matrix $B$ such that $|Ax|_p$ is closely related to $|Bx|_2$.

From $\ell_p$ to $\ell_2$: Lewis Weights

How should we relate the norms to the well-studied $\ell_2$ case?

A naive approach is to set $B = A$. However, this fails because $|Bx|_2$ is not invariant under changes of density. For example, splitting a row $a_i$ of $A$ into $k$ copies of $k^{-1/p} a_i$ leaves $|Ax|_p$ unchanged, but $|Ax|_2$ can change arbitrarily depending on how the splitting is done.

To address this, we modify the notion of density. We view each row $a_i$ as representing $w_i$ copies of $w_i^{-1/p} a_i$, where $w_i$ need not be an integer but acts as a weight in a weighted $\ell_p$ norm. Under an $\ell_2$ transformation, this corresponds to a single row $w_i^{1/2 - 1/p} a_i$. Hence, defining $W = \mathrm{diag}(w_i)$, we set

Defining Lewis Weights

Intuitively, we want the split rows $w_i^{-1/p} a_i$ to be normalized in a consistent way. Lewis’s change of density formalizes this: each normalized row should have leverage score $1$ which makes the matrix isotropic under the geometry induced by the weighted covariance.

This is a circular definition $w_i$ depends on the leverage scores of $B$, while $B$ itself depends on $w_i$ but it turns out to have a unique solution. The Gram matrix of $B$ is $A^\top W^{1 - 2/p} A$.

Definition (Lewis Weights). For a matrix $A$ and $p \ge 1$, the $\ell_p$ Lewis weights $w$ are the unique vector satisfying, for each $i$,

\[w_i = \tau_i\!\left(W^{1/2 - 1/p}A\right).\]

Equivalently,

\[a_i^\top \left(A^\top W^{1 - 2/p} A\right)^{-1} a_i = w_i^{2/p}.\]

For $p = 1$, the Lewis quadratic form is $A^\top W^{-1} A$, and the defining condition becomes $w_i^2 = a_i^\top (A^\top W^{-1} A)^{-1} a_i$, which says the leverage score of each weighted row $w_i a_i$ is exactly $1$. We are setting all leverage scores to $1$ precisely because we want a uniform matrix and to do that we use the change-of-density trick where a row is broken into several fractional copies, we must note that what i’m explaining here is extremely simplified, namely we should be exploring what things are meant here in detailed mannar.

The $\ell_1$ Matrix Concentration Bound

Once we have the Lewis weights in hand, the sampling result follows:

Theorem ($\ell_1$ matrix concentration bound). There exists an absolute constant $C_s$ such that the following holds. Let $A$ be a matrix with $\ell_1$ Lewis weights $w_i$, and let $p_i$ be sampling values satisfying $\sum_i p_i = N$ and

\[p_i \ge C_s\, w_i \log N \cdot \varepsilon^{-2}.\]

Construct a sampling matrix $S \in \mathbb{R}^{N \times n}$ by independently choosing each row to be the $i$th standard basis vector $e_i$ scaled by $1/p_i$ with probability $p_i/N$.

Then, with high probability,

\[\|SAx\|_1 \approx_{1+\varepsilon} \|Ax\|_1 \quad \text{for all } x \in \mathbb{R}^d.\]

In particular, using constant-factor approximations to the Lewis weights, $O(d \log(d/\varepsilon) \cdot \varepsilon^{-2})$ row samples suffice.

The big remaining questions how we should set the Lewis weights so that all leverage scores become uniform, and how to compute them efficiently will be the subject of the next post.

References

Main sources for this chapter

• James R. Lee et al., Sparsifying Sums of Norms
• James R. Lee, CSE 599: Sparsification, sampling, and optimization (Autumn 2023; see also Lewis weights notes)
• Michael B. Cohen and Richard Peng, $\ell_p$ Row Sampling by Lewis Weights (PDF)

Whitening and leverage-score intuition

• Andy Jones, Statistical whitening transformations
• Dustin Stansbury, The Statistical Whitening Transform (The Clever Machine, 2013)
• Kessy, Lewin, and Strimmer, Optimal whitening and decorrelation, The American Statistician 72 (2018), 309–314 — cited in Jones’s post for ZCA versus PCA whitening

Classical Lewis-weight literature

• D. R. Lewis, Finite dimensional subspaces of $L_p$, Studia Mathematica 63 (1978), 207–212 — change of density
• J. Bourgain, J. Lindenstrauss, and V. D. Milman, Approximation of zonoids by zonotopes, Acta Mathematica 162 (1989), 73–141
• M. Talagrand, Embedding subspaces of $L_p$ in $\ell_p^N$, in Geometric Aspects of Functional Analysis (Israel Seminar 1992–94), Oper. Theory Adv. Appl. 77, Birkhäuser, 1995, pp. 311–326 (volume)

Previous chapter in this series

• Chapter 1: A General Overview

One of the central themes I am exploring is tree embeddings as an algorithmic tool. A recurring paradigm in approximation algorithms is to replace a complicated metric space with a distribution over tree metrics. Since many optimization problems become significantly easier on trees, one can design an algorithm in the tree setting and then transfer the guarantee back to the original metric, often losing only a logarithmic factor in approximation quality.

A canonical example of this philosophy is the probabilistic partition framework of Yuri Bartal and the subsequent CKR decomposition of Moses Charikar, Chandra Chekuri, and Anupam Gupta. These techniques form a cornerstone of modern metric embedding theory and have influenced approximation algorithms, online algorithms, routing, clustering, and network design.

What makes this area particularly exciting is that even these classical constructions continue to inspire new advances. For example, the recent paper Tree Embedding in High Dimensions: Dynamic and Massively Parallel by Gramoz Goranci, Shaofeng H.-C. Jiang, Peter Kiss, Qihao Kong, Yi Qian, and Eva Szilagyi extends the tree-embedding paradigm to dynamic and massively parallel settings, obtaining efficient algorithms for high-dimensional Euclidean data and applications such as k-median and earth mover distance. The paper first appeared as an arXiv preprint in October 2025 and was later published at the 2026 ACM-SIAM Symposium on Discrete Algorithms (SODA 2026).

Probabilistic Metric Approximation

We now consider the following idea: if we cannot construct an exact object, we instead build a collection of objects that approximate it, and place a probability distribution over them. By sampling from this distribution, we can reason about the expected quality of the constructed object over time.

Let $\mathcal{S}$ be a family of metrics over $V$, and let $\mathcal{D}$ be a distribution over $\mathcal{S}$. We say that $(\mathcal{S}, \mathcal{D})$ $\alpha$-probabilistically approximates a metric $(V, d)$ if every metric in $\mathcal{S}$ dominates $d$, and for every pair of vertices $u, v \in V$, it holds that

Hierarchical Cut Decomposition

Let $(V,d)$ be a finite metric space. For a parameter $r > 0$, an $r$-cut decomposition is a partition of $V$ into clusters such that:

• Each cluster is associated with a center vertex,
• Every vertex in a cluster lies within distance at most $r$ from its center.

Thus, every cluster has diameter at most $2r$.

A hierarchical cut decomposition is a sequence of nested partitions:

such that:

• $D_\delta = {V}$ (the trivial partition),
• $D_i$ is a $2^i$-cut decomposition,
• $D_i$ refines $D_{i+1}$ for all $i$ (i.e., every cluster in $D_i$ is contained in some cluster of $D_{i+1}$).

At the finest level, $D_0$ consists of singleton clusters.

Intuition

This construction organizes the metric space in a top-down manner. At large scale, points are grouped into coarse clusters, and as the scale decreases, these clusters are recursively refined into smaller ones. This yields a multiscale representation of the metric that can be viewed as a tree-like structure.

Example

Consider a metric space $(V,d)$ where:

Suppose the distances are such that:

• ${a,b}$ are very close,
• ${c,d}$ are very close,
• $e$ is far from all others.

We construct a hierarchical cut decomposition:

Level $D_0$ (radius $1$)

Each vertex is a singleton:

Level $D_1$ (radius $2$)

Close points are grouped:

Level $D_2$ (radius $4$)

Further merging:

Notice the refinement property:

Each level coarsens the metric, while preserving local structure at smaller scales.

Laminar Families and the Induced Tree

What we have constructed is a laminar family, as mentioned in the paper. A hierarchical cut decomposition defines a laminar family.

Recall that a family $F \subseteq 2^V$ is called laminar if for any $A, B \in F$, it holds that

Such families are particularly convenient because they admit a natural tree representation: each set corresponds to a node, and inclusion defines the parent-child structure. Thus, laminar families form a relatively simple algebraic structure from which trees can be constructed directly.

So we try to set the metric in a way that gives us a good distance: start from the root, it doubles, and the distance between two leaves is the distance to their least common ancestor in the tree.

Each node in $D_i$ is connected to each of its children in the tree by an edge of length $2^i$, which is an upper bound on the radius of the corresponding cluster $S$. This induces a distance function $d_T(\cdot,\cdot)$ on $V$, where $d_T(u,v)$ is defined as the shortest-path distance between the leaves corresponding to ${u}$ and ${v}$ in $T$.

Given this edge-length assignment, it is easy to see that

The Partition Algorithm

Algorithm 1. Partition(V, d)

1. Choose a random permutation $\pi$ of $V$, denoted $v_1, v_2, \dots, v_n$.
2. Choose $\beta$ uniformly at random from $[1,2]$.
3. Set $D_\delta \leftarrow \{V\}$ and $i \leftarrow \delta - 1$.
4. while $D_{i+1}$ contains at least one non-singleton cluster do
    $\,$ 5. $\beta_i \leftarrow 2^{i-1}\beta$
    $\,$ 6. for $l = 1,2,\dots,n$ do
    $\,$     $\,$ 7. for each cluster $S \in D_{i+1}$ do
    $\,$     $\,$     $\,$ 8. Create a new cluster consisting of all unassigned vertices $u \in S$ such that $d(u,\pi(l)) \le \beta_i$.
    $\,$     $\,$ 9. end for
    $\,$ 10. end for
    $\,$ 11. $i \leftarrow i - 1$
12. end while

What are we doing? Notice that in each cluster in the process, we have a set of refinements. Now what we are going to do is set the new distance, and we want to cluster the vertices close to each other and refine it further. For that we should really choose a center to distinguish distances from, and we do that by randomizing the vertices available and choosing the first vertex and setting it as a means to further refine the cluster. For example, all the vertices which have distance less than $\beta_i$ to a $\pi(l)$ will be in the same cluster.

Analysis

Now we should try to analyze it. The expected value of $d_T(u,v)$ is bounded by $O(\log n)\cdot d(u,v)$.

We make no attempt to optimize constants in this analysis. From the discussion above, it follows that

Note that if $(u,v)$ is at level $i$, then

If vertices $u$ and $v$ are in separate clusters in $D_i$, we say that $D_i$ separates $(u,v)$. Note that $(u,v)$ is at level $i$ if:

• (a) $D_i$ separates $(u,v)$,
• (b) $D_j$ does not separate $(u,v)$ for any $j > i$.

Clearly, if $d(u,v) > 2^{i+2}$, then $u$ and $v$ cannot lie in the same cluster in $D_{i+1}$, i.e., $D_{i+1}$ separates $(u,v)$. From condition (b), it follows that $(u,v)$ cannot be at level $i$.

Let $j^*$ be the smallest index $i$ such that

Thus,

For \(i \ge j^*\), we bound the probability that \((u,v)\) is at level \(i\). From conditions (a) and (b), for any \(i \ge j^*\),

For any \(j^* \le j \le \delta\), let \(K_j^u\) be the set of vertices in \(V\) closer than \(2^j\) to vertex \(u\), and let \(k_j^u = |K_j^u|\). Similarly define \(K_j^v\) and \(k_j^v\). For \(j < j^*\), we define \(k_j^u = 0\).

We say that a center \(w\) cuts \(u\) out of \((u,v)\) at level \(i\) if \(w\) cuts \((u,v)\) at level \(i\) and \(u\) is assigned to \(w\) (while \(v\) is not assigned to \(w\)) at this level.

Let $w_1, w_2, \dots, w_{k_i^u}$ be the centers ordered by increasing distance from $u$ at level $i$. For a center $w_s$ to cut $(u,v)$ in a way that only $u$ is assigned to $w_s$, the following conditions must hold:

• (a) $d(u,w_s) \le \beta_i$,
• (b) $d(v,w_s) > \beta_i$,
• (c) $w_s$ is the first center that settles $u$ at this level.

Thus $\beta_i$ must lie in the interval $[d(u,w_s), d(v,w_s)]$. By the triangle inequality,

and hence the interval $[d(u,w_s), d(v,w_s)]$ has length at most $d(u,v)$.

Since $\beta_i$ is chosen uniformly in $[2^{i-1},2^i]$ (by construction), the probability that $\beta_i$ falls into this “bad” interval is at most

It follows that the probability that $D_i$ separates $(u,v)$ is bounded by

Thus each level $i$ contributes at most $O(\log n)$ to the expected value of $d_T(u,v)$ (see Equation (1)), and hence the expected distance is bounded by

A General Overview · Chapter 1

Ok, I want to summarize the sparsification of sums of norms problem. Unlike some of my longer earlier posts, I will not try a full review at once, on purpose. Each chapter should focus on one concept or technique in detail, or at least that is the plan. Hopefully I can finish this series if nothing gets in the way.

1. From Graphs to Sums of Norms

Sparsification usually means replacing a large object by a much smaller one while approximately preserving the quantities of interest. The classical starting point in computer science to my knowledge emerged in graph algorithms. Early work of Karger and, subsequently, Benczúr and Karger, Approximating s-t Minimum Cuts in $\tilde{O}(n^2)$ Time, showed that every graph admits a sparse weighted subgraph that approximately preserves all cuts. The field later underwent a major shift with the introduction of spectral sparsification by Spielman and Teng, where the goal is to preserve the entire Laplacian quadratic form rather than only cut values(as far as i know). This viewpoint was refined by Spielman and Srivastava through effective-resistance sampling and by Batson, Spielman, and Srivastava through the construction of nearly optimal linear-size sparsifiers. Over time, sparsification evolved from a graph-theoretic technique into a general paradigm for compressing large geometric and algebraic structures while retaining their essential behavior.

More recently, analogous ideas have appeared beyond graphs. In hypergraph settings, one encounters spectral notions of approximation, such as in the works of Tasuku Soma and James R. Lee and collaborators, where quadratic or energy-like functionals associated with convex bodies replace the classical graph Laplacian. The paper discussed here takes an even broader perspective and studies the sparsification of sums of norms, viewing graph and hypergraph sparsification as special cases of a more general geometric problem. A later development in this direction is the work of Jambulapati, Lee, Liu, and Sidford on generalized linear models Sparsifying Generalized Linear Models. Some techniques utilized also found justifications in Chaining, Group Leverage Score Overestimates, and Fast Spectral Hypergraph Sparsification (in my opinion).

2. What They Prove

I will state the main result in the authors’ wording. For any norms $N_1,\dots,N_m$ on $\mathbb{R}^n$, define

They show that there exists a sparsified norm

such that

where $w_1,\dots,w_m$ are nonnegative weights, of which only

are nonzero.

The weights can be found with high probability in time

where $T$ is the time to evaluate one norm $N_i(x)$, assuming $N$ is $\operatorname{poly}(n)$-equivalent to the Euclidean norm.

This abstraction covers graph sparsification and other cases (for example some submodular functions related to cuts). That is why the general statement is worth the setup.

3. The Problem in One Place

Let \(N_1,\ldots,N_m : \mathbb{R}^n \to \mathbb{R}_{+}\) be seminorms and \(N(x) := \sum_{i=1}^{m} N_i(x)\). Given weights \(w_1,\ldots,w_m \ge 0\), the weighted seminorm is

We call $\widetilde{N}$ $s$-sparse if at most $s$ of the $w_i$ are nonzero, and an $\varepsilon$-approximation of $N$ if

The question is whether one can build such a $\widetilde{N}$ using only a small number of nonzero weights.

4. How the Paper Proceeds

At a high level the workflow is:

1. Assign an importance to each term.
2. Sample terms to get an unbiased estimator while controlling variance and deviation.
3. Compute those importances efficiently, via a homotopy method.

I will unpack each step in later chapters; the homotopy method especially seems worth its own post.

5. Lewis Weights and Row Sampling

The specific algorithmic use of Lewis weights, to my knowledge, goes back to Cohen and Peng, $\ell_p$ Row Sampling by Lewis Weights. Randomized row sampling is a nice technique that can be use to design many algorithms. A standard instance: given

and $\varepsilon>0$, find a matrix $A’$ built from few rescaled rows of $A$ such that

where $a \approx_{1+\varepsilon} b$ means $\frac{1}{1+\varepsilon}\,b \le a \le (1+\varepsilon)b$.

Strong $\ell_p$ row-sampling bounds use Lewis’s change-of-density construction (Lewis, 1978), later used by Talagrand and others (Bourgain–Lindenstrauss–Milman, 1989; Talagrand, 1995): assign a weight to each row (like a leverage score), sample by those weights aka the Lewis weights. In short, Lewis weights let us study an $\ell_p$ matrix through an associated $\ell_2$ matrix $B$, where the space is much nicer in general.

Definition (Leverage Score).

Let $A\in\mathbb{R}^{n\times d}$, and let $a_i$ denote its $i$-th row. The leverage score of the $i$-th row is defined by

\[\tau_i(A) := a_i^{\top}(A^{\top}A)^{-1}a_i.\]

Equivalently,

\[\tau_i(A) = \bigl\|(A^{\top}A)^{-1/2}a_i\bigr\|_2^2.\]

Definition ($\ell_p$ Lewis Weights).

Given a matrix $A\in\mathbb{R}^{n\times d}$, the $\ell_p$ Lewis weights are the unique weights

\[w_1,\ldots,w_n \ge 0\]

such that, if $W=\operatorname{diag}(w_1,\ldots,w_n)$, then for every row $i$,

\[w_i = \tau_i\!\left(W^{\,\frac12-\frac1p}A\right),\]

where $\tau_i(B)$ denotes the leverage score of the $i$-th row of the matrix $B$.

Existence and uniqueness in the general setting can be shown via the optimization problem

Used carefully, this theory is one route toward sparsification. As in Lee’s course notes (Sparsification, sampling, and optimization, Autumn 2023), we start from a concrete $\ell_2$ setting and build outward, that is what the next sections do. For a slower treatment of leverage scores, Lewis weights, and the connection to statistical whitening, see Chapter 2.

6. Unbiased Estimators

A standard sparsifier starts from independent random sampling. Let $F(x)=\sum_{i=1}^m f_i(x)$ and let $\rho=(\rho_1,\ldots,\rho_m)$ be a probability distribution on ${1,\ldots,m}$. Sample $i_1,\ldots,i_M$ independently from $\rho$ and set

Rescaling by $\rho_{i_j}$ makes $\widetilde{F}$ unbiased: for every $x$,

so $\mathbb{E}[\widetilde{F}(x)] = F(x)$.

7. Importance Sampling

Unbiasedness holds for any $\rho$, but a good $\varepsilon$-approximation with small $M$ needs $\rho$ chosen to lower variance, importance sampling: sample large contributors more often.

With $\rho \in (0,1]^m$, sample $i_1,\ldots,i_M$ from $\rho$ and define

Since $\widetilde{N}$ is unbiased, we study how far it can deviate from its mean:

8. Symmetrization and Chaining

To control that deviation we symmetrize and obtain a subgaussian process, then bound it by generic chaining.

Introduce independent Rademachers $\varepsilon_1,\ldots,\varepsilon_M \in {-1,+1}$. By symmetrization it suffices to bound

The target is a bound of the form

Define

The family ${V_x : x\in\mathbb{R}^n}$ is subgaussian, with metric

using orthogonality of the Rademacher variables.

9. A Concrete $\ell_2$ Bound (from the course)

For $p=2$, following Lee’s course, fix a sample $\nu=(\nu_1,\ldots,\nu_M)$ and set

One wants

Let $B_F={x: F(x)\le 1}$. Dudley’s entropy bound on the subgaussian process gives

where $e_h(B_F,d_{\nu})$ is the $h$-th entropy number of $B_F$ for $d_{\nu}$.

The remaining work is covering numbers $N(B_2^n,d_U,\varepsilon)$ for rows

With the Lewis-type choice of $\rho$,

Combining entropy estimates for $B_2^n$, Dudley’s bound, and chaining yields

Taking

for large enough $C$ gives the $\varepsilon$-scale bound needed for the approximation. That is the $\ell_2$ backbone; later chapters push the same pattern toward sums of general norms.

(I will try to keep the notes really short, and also try to add some novelty into them hopefully, i perhaps will investigate this paper through the lens of banach space theory, but it will take a lot of time and effort.)

References

Main Sources for this Chapter

• James R. Lee et al., Sparsifying Sums of Norms
• James R. Lee, CSE 599: Sparsification, sampling, and optimization (Autumn 2023; see also Lewis weights notes)
• Michael B. Cohen and Richard Peng, $\ell_p$ Row Sampling by Lewis Weights (PDF)

Related Sparsification

• Jambulapati, Lee, Liu, Sidford, Sparsifying Generalized Linear Models
• Jambulapati, Liu, Sidford, Chaining, Group Leverage Score Overestimates, and Fast Spectral Hypergraph Sparsification
• James R. Lee, Spectral Hypergraph Sparsification via Chaining
• Tasuku Soma, Spectral Sparsification of Hypergraphs
• Benczúr and Karger (1996), Approximating s-t Minimum Cuts in $\tilde{O}(n^2)$ Time

Classical Lewis-Weight Literature

• D. R. Lewis, Finite dimensional subspaces of $L_p$, Studia Mathematica 63 (1978), 207–212 — change of density
• J. Bourgain, J. Lindenstrauss, and V. D. Milman, Approximation of zonoids by zonotopes, Acta Mathematica 162 (1989), 73–141
• M. Talagrand, Embedding subspaces of $L_p$ in $\ell_p^N$, in Geometric Aspects of Functional Analysis (Israel Seminar 1992–94), Oper. Theory Adv. Appl. 77, Birkhäuser, 1995, pp. 311–326 (volume)

So here we are reviewing a paper by Mika Göös. The main subject of study of this paper is finding distinction and separation between different proof systems, namely Resolution proofs and Sherali–Adams (SA) proofs (algebraic type proof systems), and its effect on TFNP problems, total search problems that are NP but we know that a solution for them exists and that we can verify the solution in an efficient manner. i try to cover in the papers manner in parts because it is a long paper, and a bit dense to read too, so hopefully whenever i find enough time again i will come back into it to cover more. Namely, the computational model works as follows: suppose that we have a black box, an oracle that we can make $q$ queries to, and it gives an answer back based on the query instantly. If we had a deterministic algorithm that based on input could verify us the answer, for let’s say in $\text{poly}(\log(n))$, then we can basically simulate this algorithm based on a decision tree with depth $\text{poly}(\log(n))$ where leaves are different portions of input that we requested access to.

A total query search problem is a sequence of relations $R_n \subseteq {0, 1}^n \times O_n$, one for each size $n \in \mathbb{N}$, such that for all inputs $x \in {0, 1}^n$ there is an output $o \in O_n$ such that $(x, o) \in R_n$. Here $O_n$ is a finite set of outputs and we say that $o$ is a solution to instance $x$, when $(x, o) \in R_n$.

We think of an instance $x \in {0, 1}^n$ as a very long bitstring that can only be accessed through queries to individual bits. In this context, an efficient algorithm is a deterministic algorithm that, for any $x \in {0, 1}^n$, finds a solution $o$ to $x$ by performing a small number of queries to $x$, namely at most $\text{poly}(\log n)$ queries. Thus, efficient algorithms correspond to decision trees (with leaves labelled by elements of $O_n$) of depth at most $\text{poly}(\log n)$. Note that this model is non-uniform: the problem admits an efficient algorithm, if for each $n \in \mathbb{N}$, there exists a shallow decision tree solving $R_n$.

The notion of total search problems as defined above does not quite correspond to TFNP yet, because it is missing the requirement for efficient verification of solutions. We enforce this in the following natural way. A total search problem \(R = (R_n)_n\) is in \(\mathsf{TFNP_{dt}}\) if for each \(o \in O_n\) there is a decision tree \(T_o\) with depth \(\text{poly}(\log n)\) such that for every \(x \in \{0, 1\}^n$, $T_o(x) = 1\) if and only if \((x, o) \in R_n\).

Let $\mathcal{F}_n = { f : {0, 1}^n \to {0, 1}^n }$ be the class of all circuits $f$ mapping $n$ bits into $n$ bits. We give a definition of a search problem for the family $\mathcal{F}_n$. In the white-box model, an algorithm is required to solve the search problem $S$ while given an explicit representation of the function $f$ (as a circuit). The white-box complexity of $S$ is the running time (as opposed to number of queries) needed (measured as a function of the size of the representation) to solve a search problem in the worst case. In the white-box setting, we are mostly interested in solvers that run in polynomial-time in the size of the function.

Definition (White-box complexity). The white-box complexity of $S$, denoted by $\textrm{wbc}(S)$, is bounded by a function $T(\cdot)$ if there exists an algorithm $A$ that for sufficiently large $n$, given $f \in \mathcal{F}_n$ (as a circuit) runs in time $T(|f|)$, and outputs $x_1, \ldots, x_q$ such that $(x_1, \ldots, x_q, f(x_1), \ldots, f(x_q)) \in S$.

Using the black box model we can make our life easier and theorems and separations that we prove in the black box model could be transported into other more interesting models of computation, so black box models are very useful.

So the main contribution of this paper comes from the observation that we can relate black box TFNP efficient problems to proof complexity.

Namely at the start of the paper they note that:

The width of a CNF formula is: the maximum number of literals in any single clause. Formally, if $F = \bigwedge_{i=1}^m C_i$ and each clause $C_i = (\ell_{i,1} \vee \ell_{i,2} \vee \cdots \vee \ell_{i,w_i})$, then

There is a correspondence between total query search problems and CNF contradictions. In one direction, a CNF contradiction $F := C_1 \wedge \cdots \wedge C_m$ over the variables $x = (x_1, \ldots, x_n)$ naturally gives rise to a corresponding total search problem $S(F)$: given an assignment $x \in {0, 1}^n$, find an unsatisfied clause of $F$. Formally, we define $S(F) \subseteq {0, 1}^n \times [m]$ by $(x, i) \in S(F)$ if and only if $C_i(x) = 0$. Thus, a sequence of unsatisfiable CNF formulas $\mathcal{F} = (F_n)$, where $F_n$ has $n$ variables, defines the total search problem $S(\mathcal{F}) = (S(F_n))$. Note that $S(\mathcal{F}) \in \textrm{TFNP}^{\textrm{dt}}$ if $F_n$ has width $\textrm{poly}(\log n)$.

In the other direction, a problem $\mathcal{R} = (R_n)$ in $\textrm{TFNP}^{\textrm{dt}}$ can be written equivalently as $S(\mathcal{F})$ for some sequence of CNF contradictions \(\mathcal{F} = (F_n)\). Specifically, for \(R_n \subseteq \{0, 1\}^n \times \mathcal{O}_n\) we define the formula \(F_n := \bigwedge_{o \in \mathcal{O}_n} \neg T_o(x)\), where we note that \(T_o(x)\) can naturally be written as a DNF formula of width at most \(\textrm{poly}(\log n)\) (with one term per accepting leaf of \(T_o\)), and thus \(\neg T_o(x)\) can be written as a CNF formula of the same width.

To see this more explicitly:

• Look at the path from the root to $\ell$.
• Suppose it queries variables $x_{i_1}, x_{i_2}, \ldots, x_{i_k}$, with outcomes $b_1, b_2, \ldots, b_k \in {0,1}$.
• This path corresponds to the term

i.e., a conjunction of literals.

Then:

So here we define some instances of TFNP problems, they are the main interests of the authors of the paper:

PPP: Pigeon ($\text{Pigeon}_n$). This problem features $n$ pigeons, denoted by $[n]$, and as input we are given, for each pigeon $u \in [n]$ a hole $s_u \in [n - 1]$. The goal is to output

1. $u, v \in [n]$, if $u \neq v$ and $s_u = s_v$. (pigeon collision)

PPADS: Sink-of-Line ($\text{SoL}_n$). This problem is defined on a set of $n$ nodes, denoted by $[n]$, where the node $1$ is “distinguished”. For input, we are given a successor $s_u \in [n]$ for each node $u \in [n]$ and a predecessor $p_u \in [n]$ for each node $u \neq 1$. Given this list of successor/predecessor pointers we create a directed graph $G$ where we add an edge $(u, v)$ if and only if $s_u = v$ and $p_v = u$. We say $u$ is a proper sink if it has in-degree $1$ and out-degree $0$, and it is a proper source if it has in-degree $0$ and out-degree $1$. The goal of the search problem is to output any of the following

1. $1$, if $1$ is not a proper source node in $G$, or (no distinguished source)
2. $i \neq 1$, if $i$ is a proper sink node in $G$. (proper sink)

PPAD: End-of-Line ($\text{EoL}_n$). Same as $\text{SoL}$, except we add the following feasible solution.

1. $i \neq 1$, if $i$ is a proper source node in $G$. (proper source)

So now it makes sense for us to talk about reductions from different search problems to another search problems, because just above we have defined several interesting instances of search problems which are total based on principles but maybe NP in general.

So we try to define a reduction for them:

Definition. Let $R \subseteq {0, 1}^n \times \mathcal{O}$ and $S \subseteq {0, 1}^m \times \mathcal{O}’$ be total search problems. An $S$-formulation of $R$ is a decision-tree reduction $(f_i, g_o)_{i \in [m], o \in \mathcal{O}’}$ from $R$ to $S$. Formally, for each $i \in [m]$ and $o \in \mathcal{O}’$ there are functions $f_i : {0, 1}^n \to {0, 1}$ and $g_o : {0, 1}^n \to \mathcal{O}$ such that

where $f(x) \in {0, 1}^m$ is the string whose $i$-th bit is $f_i(x)$. The depth of the reduction is

So what does this reduction do? Assume we have several decision trees, namely $m$, then based on the input we generate $m$ bits and we run the black box simulation on the shallow decision tree of the class which was reduced to, then when we have the answer, we use another decision tree to convert the answer back, so that is really logical for them to define it that way, in this way we keep everything deterministic and computable. (I love trees.)

So in this post we just focus on the following theorem: this theorem is used to establish some separation results further in the paper regarding different proof systems.

Theorem. There are $n$-variate CNF formulas $F$ that can be refuted by constant-width polynomial-size $\textrm{RevRes}$, but such that any $\mathcal{F}$-NS refutation (over any $\mathcal{F}$) of $F$ requires degree $n^{\Omega(1)}$.

Weak Nullstellensatz. Let $f_1, \ldots, f_m$ be polynomials. Then exactly one of the following statements holds:

1. The system of equations $f_1 = \cdots = f_m = 0$ has a solution $x \in \mathbb{C}^n$.
2. There exist polynomials $g_1, \ldots, g_m$ such that $\sum_{i=1}^m g_i \cdot f_i = 1$.

We define a generalization of $\mathbb{R}$-NS that we call $\varepsilon$-approximate Nullstellensatz ($\varepsilon$-NS) where $\varepsilon \in (0, 1)$ is an error parameter. An $\varepsilon$-NS refutation of a set of real polynomial equations ${a_i(x) = 0 : i \in [m]}$ is a set of polynomials ${p_i(x)}$ such that

Definition. Let $F = C_1 \wedge C_2 \wedge \cdots \wedge C_m$ be an unsatisfiable CNF over $n$ variables. Let $\mathbb{F}$ be any field. Then a Nullstellensatz refutation of $F$ is a set of polynomials (generalization: Polynomial Calculus) $g_1, g_2, \ldots, g_m, h_1, h_2, \ldots, h_n$ over $\mathbb{F}$ such that

where $p(C_i)$ denotes its polynomial encoding.

Why is this a refutation?

Suppose that $F$ had a solution! Then plugging in we get $0 = 1$! Contradiction.

So for them in order to prove the theorem they start by proving the following lemma:

Lemma. Every $\frac{1}{2}$-NS refutation of $\mathsf{SoPL}_n$ requires degree $n^{\Omega(1)}$.

They start to prove it using a nice problem solving technique here called decision-to-search reduction and then they use the following fact:

Fact. Suppose that $p$ is an $n$-variate real polynomial such that $p(x) = \mathsf{OR}_n(x) \pm 1/3$ for all $x \in {0, 1}^n$. Then $\deg(p) \geq \Omega(\sqrt{n})$.

Definition of reduction. We define a depth-$d$ deterministic reduction as a pair $(f, u)$ such that

1. $f : {0, 1}^{n-1} \to {0, 1}^{n’}$ is a function that maps an input $x$ of $\mathsf{OR}_{n-1}$ to an input $y = f(x)$ of $\mathsf{SoPL}_n$. Moreover, each output bit $f_i(x) \in {0, 1}$ is a depth-$d$ decision tree function of $x$.

2. For any input $x$, the only solutions of $y = f(x)$ are active sinks on the last row ${n} \times [n]$. We write $\text{Sol}(y) \subseteq {n} \times [n]$ for the set of solutions in $y$. Moreover, $u \in \text{Sol}(y)$ is a solution called the planted solution. (Note that $u$ does not depend on $x$.)

3. If $\mathsf{OR}(x) = 0$, then $y = f(x)$ contains a unique solution, namely $\text{Sol}(y) = {u}$.

4. If $\mathsf{OR}(x) = 1$, then $y = f(x)$ contains at least two solutions, $|\text{Sol}(y)| \geq 2$.

Ideal $(y, u)$: Let $\bar{y}$ be any outcome of $y$ and consider $u$ conditioned on $y = \bar{y}$, namely, $u’ := (u \mid y = \bar{y})$. Then $u’$ is uniformly distributed over $\text{Sol}(\bar{y})$; in short, $u’ \sim \text{Sol}(\bar{y})$ (we want to have the available solution which is displayed to have equal chance to be displayed).

So we are constructing instances of reduction such that in an ideal setting the solutions have equal chance of showing up in the reduction, and we add in some elements of randomness by using a distribution.

So suppose that we have a distribution over all the pairs $(f,u)$, namely the reductions and their solutions, we can define distribution over them obviously because the set itself is finite; but we want the distribution to be ideal.

Definition. Let $n$ be a positive integer, and for simplicity assume $n = 2^\lambda - 1$ for some integer $\lambda \geq 1$. Consider the following unsatisfiable CNF formula \(\mathsf{SoPL}_n\). For each \((i, j) \in \{2, \ldots, n-1\} \times [n]\) we have two blocks of \(\lambda\) variables \(s_{i,j} \in \{0, 1\}^\lambda\), \(p_{i,j} \in \{0, 1\}^\lambda\) encoding the successor and predecessor pointers of the node \((i, j)\) in binary, where null is encoded by \(0^\lambda\). For each \(j \in [n]\), we additionally have a block of \(\lambda\) variables \(s_{1,j} \in \{0, 1\}^\lambda\) encoding the successor of \((1, j)\), a block of \(\lambda\) variables \(p_{n,j} \in \{0, 1\}^\lambda\) encoding the predecessor of \((n, j)\), and a single variable \(s_{n,j} \in \{0, 1\}\) encoding whether or not \((n, j)\) is active.

The clauses of $\mathsf{SoPL}_n$ are the following:

• For each $j \in [n]$, $ s_{1,1} \neq j \vee p_{2,j} = 1 $ and $ s_{1,1} \neq 0 $ (active distinguished source)
• For each $j \in [n]$, $\overline{s_{n,j}}$ for each $j \in [n]$, (inactive sink)
• For each $(i, j) \in {1, \ldots, n-2} \times [n]$ and each $a, b \in [n]$, $c \in [n] \cup {0}$, $a \neq c$, (no proper sinks)

as well as $ s_{i,j} \neq a \vee p_{i+1,a} \neq j \vee s_{i+1,a} \neq 0 $. Similarly, for each $a, b \in [n]$,

As you can see, the above CNF formulation is unsatisfiable because the last term and inactive sink clauses are in contradiction with each other, so because Nullstellensatz is complete and sound then there exists the Nullstellensatz refutation of the fact. And the non-zero term of the refutation are the last row nodes which are supposed to be active.

Ideal reduction $\Rightarrow$ Approximation to OR. Next, we show that if we had an ideal reduction, we could construct an approximating polynomial for $\mathsf{OR}$. We write $i_u$ for the unique $i$ such that the polynomial equation $a_i(y) = 0$ encodes the $\mathsf{SoPL}_n$ constraint that $u$ is not an active sink. Namely, this corresponds to the equation $s_u = 0$, where the bit $s_u \in {0, 1}$ of the input $y$ encodes whether or not $u$ is active (see Definition above). If we think of $u \in {n} \times [n]$ as encoded by an $O(\log n)$-bit string, we can define an $[n’ + O(\log n)]$-variate polynomial

Here $p_i$ is the refutation polynomial:

Here, for every $i$, the indicator function $\mathbf{1}[i = i_u] \in {0, 1}$ is computed by an $O(\log n)$-degree polynomial. This means $q$ has degree $\deg(q) \leq O(k \log n)$ (keep in mind that it is a loose upper bound).

So we want to prove that the degree of refutation polynomials are $\Omega(1)$, so the constructions so far make sense, we are looking at each refutation polynomial separately.

If $(y, u)$ or $(f(x), u) = (y, u)$ is ideal, then for a specific $x$:

where we used the fact that $\sum_{u’ \in \text{Sol}(\bar{y})} p_{i_{u’}}(\bar{y}) a_{i_{u’}}(\bar{y}) = \sum_i p_i(\bar{y}) a_i(\bar{y})$, because $a_i(\bar{y}) = 0$ for all $i \notin {i_{u’} : u’ \in \text{Sol}(\bar{y})}$, given that $\bar{y}$ satisfies all the $\mathsf{SoPL}_n$ constraints, except the equations requiring that $u’$ not be an active sink, for $u’ \in \text{Sol}(\bar{y})$.

Suppose for a moment we had an ideal depth-$d$ randomized reduction $R$. Then, we could construct the polynomial

We have $\deg(r) \leq O(dk \log n)$. Moreover, if $\mathsf{OR}(x) = 0$ then $r(x) = 1 \pm \epsilon$; and if $\mathsf{OR}(x) = 1$ then $r(x) \in [0, (1 + \epsilon)/2]$, (it is basically equal to $(1 \pm \epsilon) \cdot \mathbb{E}[|\text{Sol}(y)|^{-1}]$) since $\mathbb{E}[|\text{Sol}(y)|^{-1}] \in [0, 1/2]$. Thus for $\epsilon = 0.01$, if we consider $t(x) := 1 - r^2(x)$ we get that $t$ approximates $\mathsf{OR}$ to within error $1/3$. Using Fact 1, we deduce that $k \geq \Omega(\sqrt{n}/(d \log n))$. So the degree must be more than one proving the lemma, but the thing is that we must be able to construct an ideal distribution which in the paper it seems we cannot do.

So how is the construction?

They try to construct it very cleverly, it’s a nice technique.

First they start off deterministically from the input of $x$, at each bit where $x_i$ is $1$ then make a path from that column to the sink, then after the deterministic construction is done, they permute the nodes of each row, then they have something nice.

A locally ideal reduction. Consider the following depth-$1$ randomized reduction $R$; see Figure 4.

1. Let \(\bar{y} = \bar{y}(x)\) be the input to \(\mathsf{SoPL}_n\) that has a directed path running down the first column of nodes, starting at distinguished node \((1, 1)\) and terminating at the active sink \(\bar{u} := (n, 1)\) (say \(\bar{u}\) is made active by being assigned \(1\) as successor). Moreover, we activate a path in \(\bar{y}\) down column \(i \geq 2\) iff \(x_{i-1} = 1\). Note that \(\bar{y}\) is a depth-$1$ decision tree function of \(x\), and \(\bar{u}\) does not depend on $x$ at all.

2. Let $y = y(x)$ be obtained from $\bar{y}$ so that, for each row except the first, $i \in [n] \setminus {1}$, randomly permute the nodes ${i} \times [n]$ on that row (updating the successor/predecessor pointers). Let $u$ be the sink node that $\bar{u}$ is mapped to.

3. Output $(f, u)$ where $f(x) := y(x)$.

As noted in the paper this reduction satisfies our requirements of SoPL but it is not ideal because we always have an active sink amongst the underlying set of the distribution, so it is not ideal, because it is based towards the existence of the active solution $u$ of our reduction, but in its local sense, for example in the local distribution by assuming $P(x \mid u)$ the rest are uniform because we are constructing them randomly and uniformly.

So that is a good trick, because basically we are simulating an ideal and uniform marginal in this way.

What we would really like instead is that $R_x$ was distributed as the ideal pair $(y, u) \sim I_x$ defined by the following procedure: Sample $(y, u’) \sim R_x$; define $u$ such that for every outcome $\bar{y}$, $(u \mid y = \bar{y}) \sim \text{Sol}(\bar{y})$; and output $(y, u)$, (reachable states based on the randomness of the reduction).

Define two functions ${0, 1}^{n-1} \to \mathbb{R}$ by

We know that $r$ has low degree as a polynomial, $\deg(r) \leq O(k \log n)$, and $r’$ has the ideal output behaviour, $r’(x) = (1 \pm \epsilon) \cdot \mathbb{E}[|\text{Sol}(f(x))|^{-1}]$ by (6). The following claim shows that, in fact, $r = r’$, and hence we can get the best of both worlds.

Claim. We have $r(x) = r’(x)$ for all $x \in {0, 1}^{n-1}$.

Proof. By linearity of expectation, it suffices to show \(\mathbb{E}_{R_x}[m(y, u)] = \mathbb{E}_{I_x}[m(y, u)]\) for any monomial \(m\) of \(q\) and every \(x\). Fix a monomial \(m\). We claim that $R_x$ and \(I_x\) have the same marginal distribution over the variables read by \(m\), which would prove the claim. We may assume that \(\deg(m) \leq O(k \log n) \leq o(n)\) because otherwise Lemma 1 is proved. Hence there exist two consecutive rows \(i, i + 1 \in [n/3, 2n/3]\) such that \(m\) does not read any variables associated with either row.

Starting with a sample $(y, u) \sim R_x$ we can generate a sample from $I_x$ as follows: Consider active nodes $A \subseteq {i} \times [n]$ and $B \subseteq {i + 1} \times [n]$ on rows $i$ and $i + 1$ in $y$ and the $|A| = |B| = 1 + |x|$ many directed edges joining them (defined by successor pointers for row $i$ and predecessor pointers for row $i + 1$). Reroute these edges by choosing a random bijection $A \to B$, and denote the resulting input by $y’$. Then $(y’, u) \sim I_x$. This proves our claim about the marginals, since our modification to the input $y$ was done outside the variables read by $m$. $\square$

Since $q$ is a polynomial,

linearity of expectation gives

Therefore, to show

it is enough to prove that for every monomial $m$,

• $r(x) := \mathbb{E}_{R_x}[q(y,u)]$ — expectation under the real reduction
• $r’(x) := \mathbb{E}_{I_x}[q(y,u)]$ — expectation under the idealized distribution

Each monomial only reads variables from few rows of the $\mathsf{SoPL}_n$ instance. Since there are $n$ rows total, by a pigeonhole argument there must exist two consecutive rows

such that no variable from either row appears in $m$. (So they don’t affect the value of the monomial but we must still compute their effect on the underlying distribution.)

And the basis of the proof is a coupling argument:

Choosing a uniformly random bijection $A \to B$, updating the successor pointers in row $i$ and predecessor pointers in row $i+1$ accordingly, leaving everything else unchanged. This random rerouting does not change the marginal distribution of $y$ outside rows $i,i+1$.

But it completely randomizes which sink in the bottom row is reachable from the distinguished node.

As a result, conditioned on the final graph $y’$, the sink $u$ is now uniformly distributed over $\mathrm{Sol}(y’)$.

Hence:

This proves our claim about the marginals, since our modification to the input $y$ was done outside the variables read by $m$.

References

The references for this blogpost:

• Separations in Proof Complexity and TFNP by mika goos et el
• On the degree of boolean functions as real polynomials by Noam Nisan and Mario Szegedy
• this lecture here