Interactive figure: find weights for the target tanh(1.5x). Enable JavaScript to play. One solution is (a, b) = (1, 1.5).
::: In the following exercises, we'll explore various different ways to implement this and other neural network functions using collections of one or more units. ## I. Flips and swaps First, let's get on top of the obvious different ways to represent a given neural network function. ### Unit flips The hyperbolic tangent is an odd function: $\tanh(-z)=-\tanh(z)$. Negating both of a unit's weights therefore preserves its contribution: $$(-a)\tanh((-b)x)=a\tanh(bx).$$ Your next challenge is to explore this *unit flip.* Move to the *other* parameter that implements the same function. ::: {.acrobatics-game data-game="flip"}Interactive figure: flip (1, 1.5) to (−1, −1.5). Both implement the same function.
::: ### Unit swaps Addition is commutative, so exchanging two units also preserves the function: $$ a_1\tanh(b_1x)+a_2\tanh(b_2x) =a_2\tanh(b_2x)+a_1\tanh(b_1x) .$$ So, your next challenge is to *swap* two labelled units. Pay close attention to the starting points, and then move them both and see if you can find this alternative way of implementing the same function. ::: {.acrobatics-game data-game="swap"}Interactive figure: exchange unit 1 at (1, 1.5) and unit 2 at (−0.5, 0.75).
::: ### Permutations and signs in general In a larger network with $h$ units, there are $h!$ permutations that implement the results of different unit swaps, times $2^h$ different combinations of unit flips to consider. In total, this gives up to $h! \cdot 2^h$ distinct implementations of a given single-hidden-layer biasless hyperbolic tangent network. These operations always preserve the implemented function for any parameter. In *most* cases, they're the *only* operations that do, and there are $h! \cdot 2^h$ different ways to implement the function exactly. This result is a special case of the result proved by [Sussmann (1992)](https://doi.org/10.1016/S0893-6080(05)80037-1). It holds for all parameters *unless* one of the following conditions holds: 1. $a_i=0$ for some unit $i$; 2. $b_i=0$ for some unit $i$; 3. $b_i=b_j$ for distinct units $i,j$; or 4. $b_i=-b_j$ for distinct units $i,j$. When any of these conditions hold, the parameter is called *reducible.* The flip and swap operations also preserve the functions implemented by reducible parameters. But for reducible parameters, there are **many more possibilities.** ## II. Reducibility Sussmann showed that reducible parameters implement functions redundantly, in that it's possible to implement the same function with fewer units. In a reducible network, units either contribute nothing to the function, or a set of two or more units contributes only a single sigmoid (which could be implemented by any one of them). Let's explore the consequences. ### Silent units If $a=0$, the incoming weight can be anything; if $b=0$, the outgoing weight can be anything: $$0\tanh(bx)=0,\qquad a\tanh(0x)=0.$$ This leads to a continuum of possible parametisations that all implement the same (zero) function. Your next challenge is to find this entire set of parameters. Trace both axes, passing through their intersection to change direction. Valid motions leave a trail. ::: {.acrobatics-game data-game="zero"}Interactive figure: the zero function's equivalence class is the coordinate cross {a = 0} ∪ {b = 0}.
::: ### Matching units When $b_1=b_2=B$, only the sum of the outgoing weights matters: $$a_1\tanh(Bx)+a_2\tanh(Bx)=(a_1+a_2)\tanh(Bx).$$ For the next challenge, we want to move both units around while holding the sum of their outgoing weights constant. So, we introduce a second slice of parameter space, showing the point $(a_1,a_2) \in \mathbb{R}^2$. Dragging this point around moves both units' outgoing weights in tandem, leaving the incoming weights fixed. Your next challenge is to trace the line $a_1+a_2=1$ to see how it preserves the target $\tanh(x)$. ::: {.acrobatics-game data-game="matching"}Interactive figure: in the (a₁, a₂) plane, follow a₁ + a₂ = 1. Its intercepts (1, 0) and (0, 1) put all the contribution in one unit.
::: At either intercept, one unit becomes silent. Beyond the intercepts, one unit contributes more than the target while the other cancels the excess. ### Opposite units When $b_1=-b_2=B$, we have a similar situation. Because hyperbolic tangent is odd, it's as if the units have the same incoming weight, and all that matters is the *difference* between their outgoing weights: $$a_1\tanh(Bx)+a_2\tanh(-Bx)=(a_1-a_2)\tanh(Bx).$$ Therefore, your next challenge is to repeat the previous game with opposite incoming weights. ::: {.acrobatics-game data-game="opposite"}Interactive figure: with (b₁, b₂) = (1, −1), follow a₁ − a₂ = 1.
::: ## III. Staying in the fibre The above examples show that for reducible parameters, the set of parameters that implement the same function is much more intricate than in the irreducible case. This set of parameters is called a *fibre* of the parameter--function map, or the *functional equivalence class* of the parameter $w$: $$[w]=\{w'\in\mathbb{R}^{2h}:f_{w'}=f_w\}.$$ For irreducible parameters, this is the discrete set generated by flips and swaps. Reducible fibres are not only infinite. In this architecture, they are also *piecewise linear path-connected*: any two equivalent reducible parameters can be joined by a path of straight segments that stays entirely within their fibre. In particular, we should be able to implement a flip or a swap *while staying entirely inside the fibre.* Let's see how the moves we discovered above can combine to implement continuous flips and swaps, while preserving the implemented function continuously. ### A continuous flip Start with one active unit and one spare unit. The idea is to let the spare carry the function while the original unit changes its incoming weight. First move the silent spare to the opposite incoming weight. Transfer the contribution to it using the outgoing-weight panel. The original unit is now silent, so you can move it to its flipped incoming weight. Transfer the contribution back, then return the spare to the origin. ::: {.acrobatics-game data-game="continuous-flip"}Interactive figure: move ((1, 1.5), (0, 0)) to ((−1, −1.5), (0, 0)) through transfers using the spare. The sum stays tanh(1.5x).
::: The incoming-weight panel shows $(b_1,b_2)$, just as the outgoing-weight panel shows $(a_1,a_2)$. All views describe the same parameter. When one unit is silent, you can change its incoming weight in either view. ### A continuous swap For a swap of two active units, use a third unit as temporary storage. Select the pair whose weights you want to coordinate. Transfer the first unit's contribution into the spare, the second into the first, and the spare into the second. Before each transfer, move the silent receiving unit to the donor's incoming weight. Writing the contributions in their labelled slots, the plan is $$ (F,G,0) \longrightarrow (0,G,F) \longrightarrow (G,0,F) \longrightarrow(G,F,0) .$$ Finally, return the spare's incoming weight to zero to complete the exact swap of the full parameter vector. ::: {.acrobatics-game data-game="continuous-swap"}Interactive figure: use unit 3 to swap the two active units, keeping their total function fixed and returning unit 3 to (0, 0).
::: ## Conclusion Reducible parameters have quite complicated functional equivalence classes, but their geometry can still be understood through simple motions of moving silent units and redistributing contributions. In my Master's thesis, [*Structural Degeneracy in Neural Networks*](/mthesis), I pioneered an approach to characterising these functional equivalence class through combinatorial algorithms for navigating the functional equivalence class, enumerating paths like the ones you explored above. This theory is also summarised in my 2023 NeurIPS paper ["Functional equivalence and path connectivity of reducible hyperbolic tangent networks"](https://proceedings.neurips.cc/paper_files/paper/2023/hash/fb64a43508e0cfe53ee6179ff31ea900-Abstract-Conference.html). So far, the theory only covers single-hidden-layer hyperbolic tangent networks, but more recently, I've been working on a generalisation to a much broader class of activation functions, and I think it might be possible to consider other architectures too. ``{=html}