Library implementing quaternion-valued linear transformation.
Titouan Parcollet 2020
This function implements a fully connected quaternion-valued linear layer: y = Wx + b.
QLinear(n_neurons, input_shape, bias=True, init_criterion='glorot', weight_init='quaternion', autograd=True, spinor=False, vector_scale=False)¶
This function implements a fully connected quaternion-valued linear layer: y = Wx + b. y, W, x and b are thus quaternion numbers. A quaternion number is written as: r + xi + yj + zk. A tensor of quaternion numbers x = [batch, 32] can be understood as [batch, 0:7] = R, [batch, 8:15] = Xi, [batch, 16:23] = Yi, and [batch, 24:31] = Xi. Thus the features dimension is cut in four (must be divisible by 4).
n_neurons (int) – It is the number of output neurons (i.e, the dimensionality of the output). Please note that these are quaternion-valued neurons. If 256 neurons are specified, the output dimension will be 1024.
input_shape (tuple) – Expected size of the input.
bias (bool) – If True, the additive bias b is adopted.
init_criterion (str , optional) – (glorot, he). This parameter controls the initialization criterion of the weights. It is combined with weights_init to build the initialization method of the quaternion-valued weights (default “glorot”).
weight_init (str, optional) – (quaternion, unitary). This parameter defines the initialization procedure of the quaternion-valued weights. “quaternion” will generate quaternion-valued weights following the init_criterion and the quaternion polar form. “unitary” will normalize the weights to lie on the unit circle (default “quaternion”). More details in: “Quaternion recurrent neural networks”, Parcollet T.
autograd (bool, optional) – When True, the default PyTorch autograd will be used. When False, a custom backpropagation will be used, reducing by a factor 3 to 4 the memory consumption. It is also 2x slower. This only works with spinor = False (default True).
spinor (bool, optional) – When True, the layer will be turned into a spinor layer. More precisely W*x will be turned into W*x*W-1. The input x will be rotated by W such as in a spinor neural network. However, x MUST be a quaternion with the real part equal to zero. (0 + xi + yj + zk). Indeed, the rotation operation only acts on the vector part. Note that W will always be normalized before the rotation to ensure the quaternion algebra (default False). More details in: “Quaternion neural networks”, Parcollet T.
vector_scale (bool, optional) – The vector_scale is only used when spinor = True. In the context of a spinor neural network, multiple rotations of the input vector x are performed and summed. Hence, the norm of the output vector always increases with the number of layers, making the neural network instable with deep configurations. The vector_scale parameters are learnable parameters that acts like gates by multiplying the output vector with a small trainable parameter (default False).
>>> inputs = torch.rand(10, 50, 40) >>> lin = QLinear(n_neurons=100, input_shape=inputs.shape, weight_init='unitary') >>> output = lin(inputs) >>> output.shape torch.Size([10, 50, 400])