Why are machine learning experts talking about Xavier initialization?

Xavier initialization is an important idea in the engineering and training of neural networks. Professionals talk about using Xavier initialization in order to manage variance and the ways that signals emerge through neural network layers.

Xavier initialization is essentially a way to sort initial weights for individual inputs in a neuron model. The net input for the neuron consists of each individual input, multiplied by its weight, which leads into the transfer function and an associated activation function. The idea is that engineers want to manage these initial network weights proactively, in order to make sure that the network converges properly with appropriate variance at each level.

Experts point out that engineers can, to some extent, use stochastic gradient descent to adjust the weights of the inputs in training, but that if they start out with improper weighting, they may not converge correctly as neurons can become saturated. Another way that some professionals put this is that signals can "grow" or "shrink" too much with improper weights, and that's why people are using Xavier initialization in accordance with various activation functions.

Part of this idea is related to the limitations of dealing with systems that are not yet developed: Before training, engineers are in some ways working in the dark. They don't know the data, so how do they know how to weight the initial inputs?

For that reason, Xavier initialization is a popular topic of conversation in programming blogs and forums, as professionals ask how to apply it to different platforms, for instance, TensorFlow. These types of techniques are part of the refining of machine learning and artificial intelligence designs that are having big impacts on progress in consumer markets and elsewhere.