The system of neural architectures is key to the mathematical richness and success of neural networks for at least two reasons.  Firstly, the stratification of the space of all functions according to which architecture(s) can realize the function exploits a tension between flexibility and rigidity -- in the sense that any function can be approximated within a sufficiently complicated architecture, but each architecture can represented only a small subset of the space of all functions.  Secondly, a neural architecture provides a parameterization of the set of associated functions -- the realization map takes a point in parameter space (a list of weight and biases) and outputs a function.  The image of the realization map (for a fixed choice of architecture) is the set of functions that can be represented by that architecture; a fiber of the realization map is a set of parameters that  all determine the same function.  Neural networks practitioners interact with the space of all functions by selecting an architecture and then  using (e.g. during training) the parameterization of function space provided by the realization map on parameter space.  Thus, for any fixed choice of architecture, an important question is "What are the image and fibers of the realization map?"

I will present some partial answers to this question for architectures of feedforward ReLU neural networks.  In particular, I will present examples of fibers with nontrivial topological structures.  I will also describe results about the unbounded rays in and dimension of the images of neural network maps of given architectures, and constraints imposed on the topology of sublevel sets by the architecture.  This is based on joint work with Eli Grigsby, David Rolnick, Robert Meyerhoff, and Chenxi Wu