WO2022053502A1

WO2022053502A1 - Device implementing a convolution filter of a neural network

Info

Publication number: WO2022053502A1
Application number: PCT/EP2021/074687
Authority: WO
Inventors: Nathan LEROUX; Julie Grollier; Alice MIZRAHI; Danijela Markovic
Original assignee: Thales; Centre National De La Recherche Scientifique
Priority date: 2020-09-08
Filing date: 2021-09-08
Publication date: 2022-03-17
Also published as: FR3113971A1; FR3113971B1

Abstract

The invention relates to a device (14) implementing a convolution filter of a neural network, wherein the convolution filter transforms an input table coded as input channels into an output table coded as output channels by using convolution kernels, the input and output tables being implemented respectively by layers of neurons (12) in which each neuron is a frequency oscillator, and wherein the device (14) comprises sets of synaptic chains (20) each formed by series resonators (22), each set applying a respective convolution kernel, and each resonator (22) having a frequency that is adjusted by an adjustment unit so as to be equal to its resonant frequency offset by a frequency offset that depends on the convolution coefficient of the convolution kernel to be applied so that the set applies the convolution kernel to be implemented.

Description

Device implementing a convolutional filter of a neural network

The present invention relates to a device physically implementing a convolutional filter of a neural network as well as a system physically implementing a convolutional neural network comprising such a device.

The development of the Internet and connected sensors has made it possible to obtain a considerable amount of data. This phenomenon often referred to as “big data” involves the use of computers to be able to exploit all the data obtained. Such exploitation can be used in multiple fields, including automatic data processing, diagnostic assistance, predictive analysis, autonomous vehicles, bioinformatics or monitoring.

To implement such exploitation, it is known to use machine learning algorithms that are part of programs that can be executed on processors such as CPUs or GPUs. A CPU is a processor, the acronym CPU coming from the English term “Central Processing Unit” literally meaning central processing unit while a GPU is a graphics processor, the acronym GPU coming from the English term “Graphie Processing Unit” literally meaning graphics unit treatment.

Among the techniques for implementing learning, the use of formal neural networks, and more specifically deep neural networks, is more and more widespread, these structures being considered as very promising because of their performance for many tasks such as automatic data classification and pattern recognition. This is particularly the case with convolutional neural networks.

A neural network is generally made up of a succession of layers of neurons, each of which takes its inputs from the outputs of the previous layer. More precisely, each layer comprises neurons taking their inputs from the outputs of the neurons of the previous layer. Each layer is connected by a plurality of synapses. A synaptic weight is associated with each synapse. It is a real number, which takes both positive and negative values. For each layer, the input of a neuron is the weighted sum of the outputs of the neurons of the previous layer, the weighting being made by the synaptic weights.

By definition, a deep neural network is a network with more than three layers of neurons and a large number of neurons per layer.

In the case of a convolutional neural network, the network further comprises a large number of layers capable of performing convolution operations. For an implementation of such a network in a CPU or a GPU, a Von Neumann funnel problem (also called Von Neumann bottleneck according to its English name) arises from the fact that the implementation of a deep neural network involves use both the memory(s) and the processor while these latter elements are spatially separated. This results in congestion of the communication bus between the memory or memories and the processor.

It is therefore desirable to develop dedicated hardware architectures, bringing together memory and calculation, to create fast, low-power neural networks capable of learning in real time.

It is known to produce neural networks based on a CMOS-type technology. It is understood by the acronym "CMOS", Complementary Metal Oxide Semiconductor (acronym from the English expression "Complementary Metal-Oxide-Semiconductor"). The acronym CMOS designates both a manufacturing process and a component obtained by such a manufacturing process.

A neural network based on technologies of the optical type is also known.

More specifically, three architectural proposals are the subject of specific studies: CMOS neural networks and CMOS synapses, optical neural networks and optical synapses and CMOS neural networks and memristive synapses. Memristive synapses are synapses using memristors. In electronics, the memristor (or memristance) is a passive electronic component. The name is a portmanteau word formed from the two English words memory and resistor. A memristor stores information efficiently because the value of its electrical resistance changes permanently when a current is applied.

However, according to each of these technologies, each neuron occupies several tens of micrometers per side. For CMOS and optical technologies, each synapse also occupies several tens of micrometers per side. As a result, on a limited surface corresponding for example to an electronic chip, the number of neurons and synapses which can be integrated is limited, which results in a reduction in the performance of the neural network.

None of the aforementioned physical implementations of neural networks are satisfactory for the physical implementation of a convolutional neural network for which the previous problems are even more troublesome.

There is therefore a need for a circuit physically implementing a convolutional neural network or a part of such a network which has improved performance. To this end, the present description proposes a device physically implementing a convolution filter of a neural network, the convolution filter taking as input an input table coded according to at least one input channel to obtain as output a table output coded according to at least one output channel, the input table being a set of input boxes arranged according to an input matrix, each box presenting a value specific to each input channel, a first layer of neurons physically implementing the input table, each neuron of the first layer of neurons being an oscillator, in particular a radio frequency oscillator, oscillating at a natural frequency, each neuron of the first layer of neurons corresponding in a one-to-one manner to the value of a box of an input channel, the output table being a set of output boxes arranged according to an output matrix, each box presenting a value specific to each output channel, a second layer of neurons physically implementing the output table, each neuron of the second layer of neurons being an oscillator, in particular a radio frequency oscillator, oscillating at a natural frequency, each neuron of the second layer of neurons corresponding in a one-to-one manner to the value d a cell of an output channel, the convolution filter being a set of convolution kernels, each convolution kernel being specific to an output channel, each convolution kernel delimiting a matrix subset having a strictly smaller size to the size of the input matrix, the matrix subset of a convolution kernel comprising a set of convolution coefficients, the device comprising sets of synaptic chains, each synaptic chain comprising synapses, each synapse being a resonator , in particular a spintronic resonator, the resonators being in series, each resonator having a frequency adjustable resonance, each set of synaptic chains being capable of applying a respective convolution kernel, each synaptic chain of the same set of synaptic chains bringing together the resonators having as resonance frequency the frequency of the neurons implementing the values of the boxes of the sub- matrix set of the input matrix associated with the convolution kernel of the set of synaptic chains and whose first element is an input box having respective coordinates, each resonator having a frequency adjusted by a frequency adjustment unit so that the frequency of the resonator is equal to the resonance frequency to which is added a frequency shift depending on the convolution coefficient of the convolution kernel to be applied to the neuron, the number of synaptic chains of the same set being equal to the number of squares of the output table that the convolution kernel makes it possible to obtain by application on the input table e. Otherwise formulated, the device physically implements a convolution filter of a neural network, the convolution filter transforming an input table coded according to at least one input channel to obtain at output an output table coded according to at least one output channel using a set of convolution kernels, the input array and the output array being physically implemented by respective neurons of the first layer and neurons of a second layer, respectively, each neuron being an oscillator oscillating at a natural frequency, the device comprising sets of synaptic chains each formed by resonators in series, each set of synaptic chains applying a respective convolution kernel, each resonator having a frequency adjusted by a frequency adjustment unit so that the frequency of the resonator is equal to the resonant frequency to which is added a frequency shift depending on the convolution coefficient of the convolution kernel to be applied to the neuron for the set of synaptic chains to apply the convolution kernel to be implemented.

A device is also described that physically implements a convolution filter of a neural network, the convolution filter being capable of applying a set of convolution coefficients to a set of input values ordered according to a first order to obtain a set of output values ordered according to a second order, the set of convolution coefficients being arranged according to a third order, the first order, the second order and the third order are each a line-to-line traversal successively for each channel of a matrix corresponding in a representation of the operation of a convolution filter in the form of a matrix operation where each matrix is a set of boxes where each box has a value specific to at least one channel. The device comprises a first layer of neurons, each neuron of which is an oscillator, in particular a radiofrequency oscillator, oscillating at a natural frequency and physically implements a respective input value, the neurons being aligned on the same first column according to the first order. The device also comprises a second layer of neurons, each neuron of which is an oscillator, in particular a radio frequency oscillator, oscillating at a natural frequency and physically implements a respective output value, the neurons being aligned on the same second column according to the second order, the second column being parallel to the first column. The device also comprises sets of synaptic chains, each synaptic chain comprising synapses in series physically implementing a respective convolution coefficient on at least one input value, each synapse being linked to the neuron of the first layer of neurons physically implementing the value input on which the convolution coefficient of said synapse is to be applied, each synapse being a resonator, in particular a spintronic resonator, having a resonance frequency, and having a frequency adjustable by a frequency adjustment unit so that the frequency of the resonator is equal to the resonance frequency to which is added an offset in frequency depending on the respective convolution coefficient that the resonator physically implements, the resonators of the same synaptic chain being arranged according to the third order, each synaptic chain being connected to a neuron of the second layer of neurons, the synaptic chains being parallel to each other and arranged in a direction perpendicular to the two columns, the resonators physically implementing the same coefficients of the convolution filter being spatially aligned along the same line, the different lines being parallel to each other.

According to particular embodiments, the device comprises one or more of the following characteristics when technically possible:

- the application of a convolution kernel for the or each input channel to a matrix subset of the input matrix whose first element is an input box having first coordinates giving the value of the box output having the first coordinates for the output matrix according to the output channel specific to the applied convolution kernel.

- the frequency adjustment unit is the same for the resonators of the same set of synaptic chains for which the convolution coefficient to be applied is identical.

- an input and an output are defined for each synaptic chain, the device comprising at least one amplifier positioned either at the input or at the output.

- each frequency adjustment unit is chosen from the list consisting of a direct current applicator, a current pulse applicator and a voltage applicator.

- the resonators are arranged in a matrix whose columns are the resonators having the same convolution coefficient to be applied.

- the device comprises a summing of the signals coming from the neurons of the first layer of neurons.

- the resonators are connected to at most two neurons of the first layer of neurons.

- the synaptic chains are parallel to a line along which the neurons of the first layer of neurons are arranged.

- each resonator is connected to a single neuron of the first layer of neurons. The present description also proposes a system physically implementing a convolutional neural network comprising a device as previously described.

Other characteristics and advantages of the invention will appear on reading the following description of embodiments of the invention, given by way of example only and with reference to the drawings which are:

- Figure 1, a schematic representation of a system physically implementing a convolutional neural network;

- Figure 2, a schematic representation of the action of a convolution filter;

- Figure 3, a schematic representation of an example of a device forming part of the system of Figure 1, the device physically implementing the convolution filter of Figure 2;

- Figure 4, a schematic representation of another example of a device physically implementing a convolution filter, and

- Figure 5, a schematic representation of yet another example of a device physically implementing a convolution filter, and

- Figure 6, a more precise representation of the device of Figure 6.

A system 10 physically implementing a neural network is shown in FIG.

The system 10 physically implements a neural network comprising a set of layers of neurons and interconnections connecting one layer of neurons to another layer of neurons.

This means that system 10 includes circuits 12 implementing layers and devices 14 implementing interconnects.

Each layer 12 of neurons is a set of at least two neurons 16.

By definition, in biology, a neuron, or a nerve cell, is an excitable cell constituting the basic functional unit of the nervous system. Neurons transmit a bioelectrical signal called a nerve impulse. Neurons have two physiological properties: excitability, i.e. the ability to respond to stimulation and convert it into nerve impulses, and conductivity, i.e. the ability to transmit signals. impulses. In formal neural networks, the behavior of biological neurons is imitated by a mathematical function which has the property of being non-linear (to be able to transform the input in a useful way) and preferentially of being differentiable (to allow the gradient backpropagation learning). In the context of this application, a neuron 16 is a component performing an equivalent function. The layers of neurons 12 are ordered, so that it is possible to define an index for each layer of neurons 12. In the remainder of the remainder of the present application, the terms “upstream” and “downstream” are defined in relation in the increasing sense of this index.

In the example described, each neuron 16 is an oscillator.

In addition, each oscillator is an oscillator whose frequency is between 1 MegaHertz (MHz) up to several TeraHertz (THz). Hereafter we will use the term “radiofrequency” to refer to this frequency range.

An oscillator is a device suitable for generating oscillations having a controlled amplitude and a fixed or controlled frequency on one or more output(s).

An oscillator being likely to present several frequencies of oscillations, by definition, the frequency of oscillation of a neuron 16 is the frequency of the oscillation presenting the greatest amplitude (the amplitude being defined in peak to peak) .

In the illustrated case, the neurons 16 have a respective oscillation frequency. This means that the oscillation frequencies of all the oscillators of the same layer are distinct two by two.

Depending on the implementations, the output signal of a neuron 16 is a radiofrequency electric current, a radiofrequency electromagnetic field or a spin wave.

Spin waves are fluctuations in the magnetization of ferromagnetic materials around the magnetization equilibrium position. The spin wave can be localized or propagate. A ferromagnetic material has a spontaneous magnetization, unlike non-magnetic materials.

In physics, magnetization is a vector quantity that characterizes the magnetic behavior of a sample of matter on a macroscopic scale. Magnetization originates from the orbital magnetic moment and the spin magnetic moment of electrons or atoms.

According to a first implementation, each neuron 16 is a CMOS oscillator.

The creation of such an oscillator is based on the transposition of existing electronic assemblies, such as the Colpitts oscillator, the Clapp oscillator, the phase shift oscillator, the Pierce oscillator, the Hartley oscillator, the variable oscillator of state or the ring oscillator (more often referred to as the “ring oscillator”).

This makes it possible to obtain a CMOS oscillator presenting an oscillation signal with a fixed frequency and a controllable amplitude. This results in high emitted power and low noise. According to a second implementation, the oscillators are made with laser signals, and elements having a non-linear optical transmission, which would have the advantage of the high speed of information (speed of light), but also the disadvantage of the size. components, and the high energy consumption of lasers.

According to a third implementation which is that of the example described, each neuron 16 is a spintronic oscillator.

Such an implementation reduces the size of the first implementation.

Spintronic oscillators make it possible to obtain oscillation frequencies over a wide range of frequencies, between 1 MegaHertz (MHz) and several TeraHertz (THz), especially when antiferromagnetic materials are used.

Antiferromagnetism is a property of certain magnetic media. Unlike ferromagnetic materials, in antiferromagnetic materials, the exchange interaction between neighboring atoms leads to an antiparallel alignment of atomic magnetic moments. The total magnetization of the material is then zero. Like ferromagnets, these materials become paramagnetic above a transition temperature called the NéeL temperature.

Antiferromagnetism is distinct from ferromagnetism, which designates the ability of certain bodies to become magnetized under the effect of an external magnetic field and to keep part of this magnetization.

In addition, spintronics, spin electronics or magnetoelectronics, is a technique that exploits the quantum property of the spin of electrons in order to store information or perform calculation operations on this information. By extension, a spintronic component is a component that exploits the quantum property of the spin of electrons in order to store or process information.

According to the proposed example, spintronic oscillators generate harmonic signals.

A neuron 16 comprises, for example, a pillar with a characteristic diameter of between 3 nanometers (nm) and 1 micrometer (pm) and means for injecting a supply current through the pillar.

The pillar comprises a pattern consisting of several layers, superimposed along a stacking direction of the layers, namely a first layer of a ferromagnetic material, an intermediate layer of a non-magnetic material, and a second layer of a ferromagnetic material. At each of its ends, the pillar respectively comprises lower and upper layers which are arranged on either side of the pattern and constitute contacts allowing the injection of a supply current through the layers.

The means suitable for allowing the injection of a supply current through the pillar are a current source capable of delivering either a direct current, or an alternating current adjustable in intensity and frequency, or both, and electrodes.

Such oscillators have the disadvantage of their relatively high noise level, but the advantage of low power consumption and small size.

In summary for the present example, spintronic oscillators are used, such as transfer oscillators whose stack is magnetic tunnel junctions. The direct current crossing the junction makes it possible to oscillate the magnetization of the upper ferromagnetic layer, and therefore to oscillate the resistance of the junction by tunnel magnetoresistance effect. A direct current therefore makes it possible to generate an alternating voltage.

The interconnects each perform one or more specific operations.

In this case, the neural network is a convolutional neural network.

Such a neural network is capable of implementing a plurality of convolution filters.

The number of filters used increasing with the depth of the network, the number of filters is sometimes very important.

An example of the operation of such a convolution filter FC is shown in Figure 2.

The convolution filter FC takes as input an input table IE coded according to at least one input channel to obtain as output an output table IS coded according to at least one output channel.

The IE input array is a set of input bins arranged in an input matrix, each bin presenting a value specific to each input channel.

In the "IE" notation, the letter "I" is used to designate the table because any table corresponds to an image, that is to say a set of pixels.

A specific example of such an IE input array is a color image.

A color image is an IE input table coded according to the values of the pixel in three colors, typically according to an RGB coding. In such an example, the input table IE is a first set of input boxes arranged according to an input matrix whose value is the red component of the color image, a second set of input boxes also arranged according to the input matrix having as its value the green component of the color image and a third set of boxes also arranged according to the input matrix and having as value the blue component of the color image.

In the representation of Figure 2, the input matrix is a set of rows each identified by an index i and by a set of columns identified by an index j.

The index i is an integer whose value is between 1 and NH, NH being the total number of rows of the input matrix.

The index j is an integer whose value is between 1 and Nw, Nw being the total number of columns of the input matrix.

Thus, each element of the input matrix is identified by a couple (i, j).

Furthermore, as each element of the input matrix has a different value depending on the input channel considered, each element of the input array IE is identified, not simply by the pair (i, j), but indeed by a triplet (i, j, c), c being the channel index which is an integer varying between 1 and Ne the number of channels of the IE input table.

Subsequently, an element of the input matrix will thus be denoted Xj,j, _c and an element of the input array IE will be denoted Xij.

Similarly, it is possible to locate each element of the output matrix by a triplet (a, b, m), hence the notation z _a ,b, _m for an element of the output matrix and z _a ,b for an element of the IS output image.

The index a is an integer whose value is between 1 and h, where h is the total number of rows of the output matrix.

The subscript b is an integer whose value is between 1 and I, where I is the total number of columns of the output matrix.

The index m is the channel index which is an integer varying between 1 and N _m the number of channels of the output table IS.

The convolution filter FC is a set of convolution kernels NC, the set of convolution kernels NC applied to the input array IE to obtain the output image IS.

Each NC convolution kernel is specific to an output channel.

The example NC convolution kernel shown in Figure 2 is specific to output channel m.

Each convolution kernel NC delimits a subset presenting a size strictly lower than the size of the input matrix.

This means that, in this example, the total number of rows of the matrix subset is strictly less than the total number NH of rows of the input matrix and that the total number of columns of the matrix subset is strictly less than the number total Nw of columns of the input matrix. However, it is possible to envisage cases where the total number of rows of the matrix subset is equal to the total number NH of rows of the input matrix and/or the total number of columns of the matrix subset is equal to the total number N _w of columns of the input matrix. This is particularly relevant when one-dimensional convolutions are applied successively, for example a horizontal convolution and a vertical convolution.

In this case, the convolution kernel NC has three rows and three columns, which means that the convolution kernel NC is square in the particular case of Figure 2.

The convolution kernel matrix subset NC includes a set of convolution coefficients.

In the representation of FIG. 2, the convolution kernel NC is a set of rows each identified by an index p and a set of columns identified by an index q.

The index p is an integer whose value is between 1 and HH, HH being the total number of rows of the convolution kernel NC.

The index q is an integer whose value is between 1 and nw, where nw is the total number of columns of the convolution kernel NC.

In the case shown, n = n _w = 3.

Moreover, the value of the convolution coefficient also depends on the input channel and the output channel considered.

This means that an element of the NC convolution kernel is no longer identified by a triplet as in the case of the input and output images but by a quadruplet (p, q, c, m).

An element of the convolution kernel NC will thus be denoted w _p ,q, _c , _m or alternatively w™ _qc according to the notations of figure 3.

The application of the NC convolution kernel for the or each input channel to a matrix subset of the input matrix, a first element of which is an input box having first coordinates gives the value of a box of output having the first coordinates for the output matrix according to the output channel specific to the applied NC convolution kernel.

More precisely, this means that the convolution kernel NC is applied to each element of the input matrix spatially included in the matrix subset positioned at the input box with the first coordinates and this, on all possible channels.

In this case, this means that the first element of the IS output array, namely zi,i,m is the result of applying the convolution kernel NC corresponding to the channel m to the nine elements Xu , xi,2, xi ,3, X2,i, X2,2, X2,3, X3,i, X3,2 and x 3,3 of the IE input table. More precisely, this implies that the first element of the output table IS zi ,1 , _m is a weighted sum of 27 terms corresponding to the value of the aforementioned nine elements of the input image IE for channel 1 , channel 2 and channel 3. The weighting is done using the convolution coefficients w _p ,q, _c , _m of the convolution kernel NC.

Mathematically, this is written:

The previous considerations show that the first term Xu of the sum has the same coordinates as the element z-i,i .

The generalization of the previous formula gives the following formula:

In general, a bias weight could also be added to the overall sum. Such a bias weight is specific to the output channel considered and would be denoted PB _m .

This bias weight is neglected in the rest of the presentation and would be easily implemented by adding a voltage with a CMOS component before the input of each neuron 16.

The previous formulation clearly shows that the convolution coefficient w _{i J:C m} is to be interpreted as a synaptic weight.

Devices 14 in Figures 3-5 show physical implementations of such an FC convolution filter.

Devices 14 of FIGS. 3 to 5 are devices physically implementing an interconnection of FIG. 1 .

Each of the devices 14 corresponds to a frequency implementation of the preceding relationships.

Devices 14 are an interconnection between a first layer of neurons 12 and a second layer of neurons 12.

Each layer comprises a set of neurons 16 arranged along a line.

A line is defined as vertical or according to a column as being the line along which the neurons 16 of the first layer 12 are arranged.

As the frequencies of oscillation of the neurons 16 are intended to represent the cells of the input table IE, it is possible for the case illustrated to write the frequencies of the oscillators as, in the order, fi,i,i,f -1,2,1, ■■■ , fi.Nw.-i, f2,i,2> ■■■ , fNh.Nw , f2,i,2, ... , fNh,Nw,Nc- In each case, the device 14 comprises sets of synaptic chains 20 connected to voltage-current conversion circuits 18 as well as frequency adjustment units.

A synaptic chain 20 is a set of synapses 22 connected in series by at least one transmission line.

Each transmission line or field line is made by a metal track placed at a reasonably close distance above or below synapses 22 but electrically isolated from synapses 22.

It should be noted that, for the embodiment corresponding to FIG. 3, the transmission of the signal can be done by field line, but also electrically with conventional lines. In the latter case, the transmission lines are not isolated from the synapses 22.

The same implementation is possible for the field lines used to modify the frequencies of the synapses 22 which will be described later with reference to the frequency adjustment units.

In biology, the synapse designates a functional contact zone which is established between two neurons 16. Depending on its behavior, the biological synapse can excite or even inhibit the downstream neuron in response to the upstream neuron. In formal neural networks, a positive synaptic weight corresponds to an excitatory synapse while a negative synaptic weight corresponds to an inhibitory synapse. Biological neural networks learn by altering synaptic transmissions throughout the network. Similarly, formal neural networks can be trained to perform tasks by modifying synaptic weights according to a learning rule. One of the most powerful learning rules today for training deep networks is gradient backpropagation. In the context of this application, a synapse 22 is a component performing a function equivalent to a synaptic weight of modifiable value.

Furthermore, in what follows, we will call “synaptic chain” or “chain 20” a set of synapses 22 connected in a chain, whose function is to connect all or a subset of neurons 16 from the upstream layer to the layer downstream. More specifically, the output of a synaptic chain 20 is proportional to the weighted sum of the outputs of the neurons 16 of the previous layer which are connected to the input of the chain 20, the weighting being made by the synaptic weights of the synapses 22 which constitute the chain 20.

Each synapse 22 is a spintronic resonator. Alternatively, each synapse 22 is realized by optical resonators if the neurons 16 are optical signals.

A resonator 22 is an electrical component having a resonant frequency. More precisely, the response of a resonator 22 to a radio frequency signal is higher in a certain range around the resonant frequency.

A spintronic resonator 22 is a magnetoresistive resonator.

More specifically, a spintronic resonator 22 is an electrical component comprising one or more ferromagnetic layers and the magnetization of at least one of the layers of which can be put into resonant precession by a radiofrequency signal. The precession of the magnetization causes a variation of the resistance of the resonator 22 by magneto-resistive effect. The resonant frequency of the resonator 22 depends on the dimensions of the ferromagnetic layer, the magnetic field which is applied to the ferromagnetic layer and the ferromagnetic material or materials which form(s) the ferromagnetic layer.

Furthermore, each spintronic resonator 22 has an adjustable resonant frequency.

Thus, according to a first example, the resonators 22 are magnetic tunnel junctions (namely a ferromagnetic/insulator/ferromagnetic stack).

The current or the magnetic field makes the upper magnetic layer resonate, and therefore causes the resistance to oscillate thanks to the tunnel magnetoresistance effect.

According to a second example, the resonators 22 are spin valves (namely a ferromagnetic/metal/ferromagnetic stack).

According to a third example, the resonators 22 are simple lines of ferromagnetic metals

The Oersted field then created by the current crossing the magnetic line allows it to enter into resonance. The resistance oscillates thanks to the anisotropy effect of magnetoresistance (weaker effect than that of giant magnetoresistance).

According to a fourth example, the resonators 22 are junctions with a Heavy Metal/Ferromagnetic/Insulator stack with a field line above. A heavy metal exhibiting a strong reverse-spin Hall effect. In this case, the heavy metal is an alloy comprising one or more of the elements from the group consisting of Pt, W, Pd, Au, Ir, Ag and Bi.

The field line creates a radio frequency magnetic field which causes the ferromagnetic part to resonate. The precession of the magnetization induces a spin current in the heavy metal part (metals with strong spin-orbit coupling, such as Pt, W, Pd...), and this spin current is converted into voltage thanks to the inverse spin Hall effect.

In the example where the resonators 22 are magnetic tunnel junctions, the voltage at the terminals of a resonator 22 as a function of the signal received is written according to the following formula:

Or :

• P _RF is the power of the received signal,

• f _RF is the frequency of the received signal,

• f _res is the resonance frequency of the resonator 22,

• a is the magnetic damping coefficient of resonator 22, and

• A is a coefficient which depends in particular on the resistance of resonator 22, the magnetoresistance of resonator 22 and the impedance adaptation of resonator 22.

Each set of chains 20 is capable of applying a respective convolution kernel NC.

The number of strings 20 of the same set being equal to the number of boxes of the output table IS that the convolution kernel NC makes it possible to obtain by application on the input table IE.

Each chain 20 of a same set of chains 20 brings together the spintronic resonators 22 having a resonant frequency equal to the frequency of the neurons 16 implementing the values of the pixels of the matrix subset of the input matrix associated with the convolution kernel NC of the set of strings 20 and whose first element is an input pixel having respective coordinates.

By equality in this context, it is understood that the resonant frequency of the resonator 22 is close to the frequency of the neurons 16, which means that the frequency difference f _RF - f _res between the frequency of a resonator 22 and of its associated neuron 16 has an absolute value less than the term _afres .

In the example given, this means that the first chain 20 comprises the 27 spintronic resonators 22 associated with the value of the aforementioned nine elements xi ,1 , xi, ₂ , xi,3, x ₂ ,i, x ₂ , ₂ , x ₂ ,3, x ₃ ,i, x ₃ , ₂ and x 3,3 of the IE input table for channel 1 , channel 2 and channel 3. By the term “associated” in this context, it is understood that each resonator 22 has a respective frequency equal to the (“close” among the) frequencies of the neurons 16 encoding the value of the aforementioned elements.

In operation, each spintronic resonator 22 has a frequency adjusted by a frequency adjustment unit U so that the frequency of the resonator 22 is equal to the resonant frequency of the resonator to which is added a frequency offset depending on the convolution coefficient of the NC convolution kernel to be applied to the oscillator associated with the spintronic network. The added frequency shift denoted S _pqcm is such that the voltage across a resonator 22 is written: v

So so that

The frequency offset has the value 5 _vacm =

— - .

' ^2w p,q,c,m

This is indicated in FIG. 3 for each resonator 22 by a rectangle comprising a value corresponding to the resonance frequency and a value corresponding to the frequency shift to be applied.

More precisely, in the example proposed, assuming that the output channel concerned is channel m, the first chain 20 successively comprises a first resonator 22 at the frequency (fi,i,i + 5i,i,i,m), a second resonator 22 at the frequency (f-1,2,1 + 5i,2,i,m), ...., a ninth resonator 22 at the frequency (fs, 3,1 + δ _3,3 ,i ,m), a tenth resonator 22 at the frequency (f 1 ,1 ,2 + 5i ,1 ,2,m), an eleventh resonator 22 at the frequency (fi ,2,2 + 5i ,2,2,m) , . . . . , an eighteenth resonator 22 at the frequency (fs, 3, 2 + o _3,3,2 ,m), an eighteenth resonator 22 at the frequency (f 1,1,3 + 5i,1,3, m) a twentieth resonator 22 at the frequency (f 1 ,2,3 + ôi ,2,3,m), . . . . and a twenty-seventh resonator 22 at the frequency (fs, 3, 3 + Ô3,3,3,m).

Similarly, the second chain 20 successively comprises a first resonator 22 at the frequency (fi, ₂ ,i + 5i,i,i,m), a second resonator 22 at the frequency (f-1,3,1 + ôi,2 ,i, _m ), a ninth resonator 22 at the frequency (fs,4,1+Ô3,3,i,m), a tenth resonator 22 at the frequency (f 1,2,2+5i,1,2, m), an eleventh resonator 22 at the frequency (fi, 3,2 + ôi, 2,2, _m ), an eighteenth resonator 22 at the frequency (fs, 4, 2 + ô ₃ , 3,2, m ), a nineteenth resonator 22 at the frequency (f 1 ,2,3 + 5i ,1 ,3,m) a twentieth resonator 22 at the frequency (f 1 ,3,3 + ôi ,2,3,m) , . . . . and a twenty-seventh resonator 22 at the frequency (fs,4,3+Ô3,3,3,m).

And so on to browse the set of possible values for the indices a and b. The (h+1)-th chain 20 thus successively comprises a first resonator 22 at the frequency (fNa,Nt>,i +5i,i,i, _m ), a second resonator 22 at the frequency (fNa,Nt>+ i,i + ôi ,2,i , _m ), ■ ■ ■ ■ , a ninth resonator 22 at the frequency (fNa+2,Nb+2,i + Ô3,3,i,m), a tenth resonator 22 at the frequency (fNa,Nt>,2 + ôi,i,2,m), an eleventh resonator 22 at the frequency (fNa,Nb+i ,2 + ôi ,2,2,m), ■ ■ ■ ■ , a eighteenth resonator 22 at the frequency (fNa+2,Nb+2,2 + Ô3,3,2,m), an eighteenth resonator 22 at the frequency (fNa,Nb,3+5i,1,3, m) a twentieth resonator 22 at the frequency (fNa,Nb+i,3 + ôi,2,3, _m ), .... and a twenty-seventh resonator 22 at the frequency (fNa+2,Nb+2, 3 + Ô3,3,3,m).

This gives a set of h*l strings 20.

According to the example of Figure 3, a frequency adjustment unit U is a direct current applicator.

In the example of Figure 3, the same frequency shift is applied to each of the i-th resonators 22.

In such a case, the frequency adjustment unit U is the same for the spintronic resonators 22 of the same set of chains 20 for which the convolution coefficient to be applied is identical.

This means that it is the same frequency adjustment unit U for each of the i-th resonators 22 of the assembly described in FIG. 3.

Furthermore, by arranging the resonators 22 according to a matrix whose columns are the i-th resonators 22, the frequency adjustment unit U is able to apply the same frequency shift to a rectilinear field line extending along a column, that is to say perpendicular to the direction in which the chain 20 extends.

In the example of figure 3, the device 14 also comprises a summer 23 and an amplifier 24 placed between the first layer of neurons 12 and the input of the chains 20.

A summer 23 is a circuit making it possible to sum different types of signals on the same transmission channel.

In this case, the adder 23 is able to perform a sum of the signals from each of the neurons 16 of the first layer of neurons 12.

The adder 23 is, for example, produced by CMOS technology.

An amplifier 24 is an electronic system that increases the voltage and/or intensity of an electrical signal.

Here, the amplifier 24 is able to amplify the summed signal.

By way of example, the amplifier 24 is produced using CMOS technology.

Each conversion circuit 18 acts as a DC to DC converter. According to the example proposed, the conversion circuit 18 comprises diodes.

A diode is connected to the output of each chain 20.

The diode is, for example, a CMOS type diode.

The diode makes it possible to transform the incident radiofrequency signal into a continuous signal. Any other device for performing the same function is possible.

As an example, instead of a diode, a radio frequency power detector is used.

The device 14 also comprises an amplifier 26 at the output, that is to say an amplifier 26 between the conversion circuit 18 and the corresponding neuron 16 of the second layer of neurons 12.

The operation of the device 14 of FIG. 3 is now described.

The signal from neurons 16 is summed by adder 23 to obtain a summed signal.

Amplifier 24 amplifies the summed signal.

The signal thus amplified is sent to the input of each chain 20.

In parallel, the frequency adjustment units apply the frequency shifts corresponding to the convolution coefficients to each resonator 22.

More precisely, in each set of chain 20, the first frequency adjustment unit U1 adds a first frequency offset (first weight), the second frequency adjustment unit U2 adds a second frequency offset (second weight), the third frequency adjustment unit U3 adds a third frequency offset (third weight)... and so on up to the twenty-seventh frequency adjustment unit U27 which adds a twenty-seventh frequency offset ( twenty-seventh weight).

For this, each frequency adjustment unit U being a direct current applicator, a direct current is applied and passes through a field line. A magnetic field is thus created. The magnetic field then acts on the magnetization of a layer of the resonator 22. The resonance frequency of a magnetic material evolves, in fact, in a linear manner with the applied magnetic field.

Then, an interaction between the summed signal and each resonator 22 of the chain 20 takes place.

The signal obtained after interaction is then rectified by the resonator 22.

The value of a cell of the IS output table is thus obtained.

Otherwise formulated, each synaptic chain 20 calculates the value of a box of the IS output table by interaction between the summed signal and each resonator 22 of the chain 20. A set of strings 20 thus calculates the value of the set of cells of the output table IS for a channel.

Each set of chains 20 calculates an IS output table for a respective channel.

This principle illustrated for the convolution filter FC of figure 2 can be applied for a general case. As a result, it is in this case that Figure 3 has been produced with any numbers for the indices.

The device 14 of FIG. 3 thus physically realizes a convolution filter FC on an input array IE implemented by the first layer of neurons 12 and an output array IS implemented by the second layer of neurons 12.

In other words, this means that the device 14 is a physical implementation of a convolution filter FC on an input array IE implemented by the first layer of neurons 12 and an output array IS implemented by the second layer of neurons 12.

Such a device 14 is capable of performing large matrix products in parallel with non-linear transformations. As explained previously, such a non-linearity comes from the fact that neurons 12 only oscillate above a threshold current.

Such a device 14 is thus compatible with a system 10 comprising a number of layers greater than 3, preferably greater than 5 and/or the number of chains 20 is greater than 9, preferably greater than 100.

In addition, the device 14 having memory all integrated on the same chip would drastically reduce the energy consumption, and increase the execution speed of the image recognition.

This increase in speed is all the more significant since the device 14 does not involve any time-division multiplexing.

In addition, the device 14 is very compact since the components forming the oscillators and the resonators 22 can have dimensions of the order of 20 nm.

Such components are, moreover, compatible with CMOS technology, which makes their integration easy.

Furthermore, it is proposed to modify the synaptic weights simultaneously using a frequency adjustment unit U common to several resonators 22, which saves a lot of writing time.

Thus, the use of a current applicator for the frequency adjustment unit U allows easy control of synaptic weights through analog control.

However, because the direct current must be applied continuously, such an embodiment corresponds to volatile operation: According to another example, the frequency adjustment unit U is a current pulse applicator.

In operation, when a current pulse passes through a field line, a strong magnetic field is produced. Such a strong magnetic field causes the magnetization of a layer of the resonator 22. It is then possible to alternate between two different frequencies, which allows the resonator 22 to have two states depending on whether or not a current pulse by the current pulse applicator.

Such an example therefore corresponds to non-volatile operation but with only binary weights.

According to yet another example, the frequency adjustment unit U is a voltage applicator.

A gate voltage applied above the resonator 22 moves oxygen atoms to the interface between the tunnel barrier (oxide) and the upper layer of the resonator 22. This makes it possible to modify the magnetic anisotropy, and therefore to modify the effective applied magnetic field.

Such an example therefore corresponds to analog and non-volatile operation.

Depending on the needs, the frequency adjustment units are according to one or more of the preceding examples.

For example, in a part of the neural network where the weights are imposed to be binary, the use of a current pulse applicator is indicated, but in a part of the neural network where the weights are arbitrary, the use of a voltage applicator is indicated. In such an example, the frequency adjustment units are either current pulse applicators or voltage applicators.

The frequency adjustment unit is thus an element suitable for adjusting the frequency of a resonator 22 by modifying one of the voltage, the current or the magnetic field applied to a resonator 22.

Figure 4 shows a variant of the device 14 of Figure 3.

The device 14 of FIG. 4 has the same elements as the device 14 of FIG. 3. For the sake of simplification, the remarks which are valid for the two devices 14 are not repeated in what follows. Only the differences are underlined.

In such a device 14, the chains 20 do not extend along a line perpendicular to the line along which the neurons 16 of the first layer of neurons 12 extend.

The chains 20 thus extend along a column.

Furthermore, instead of using a summer 23 and an amplifier 24, the signals from each neuron 16 are transmitted along lines of radiofrequency fields horizontal lines visible in dotted lines in FIG. 4. Each signal from a neuron 16 is transmitted along a line of fields specific to it.

As a result, the resonators 22 associated with the same weight (or the same frequency addition) are not aligned along a column or a line.

The frequency adjustment units are then arranged to operate in a diagonal line.

The device 14 according to figure 4 has the same advantages as the device 14 of figure 3.

Figure 5 shows a variant of the device 14 of Figure 3.

The device 14 of FIG. 5 has the same elements as the device 14 of FIG. 3. For the sake of simplification, the remarks which are valid for the two devices 14 are not repeated in what follows. Only the differences are underlined.

The resonators 22 are arranged similarly to those of Figure 3.

However, instead of using a summer 23 and an amplifier 24, the signals from each neuron 16 are transmitted along diagonal radiofrequency field lines visible in dotted lines in Figure 5.

Each field line transmits a signal which is specific to it, the signal being composed of at most two signals from two different neurons 16 of the first layer of neurons 12.

More specifically, each field line is arranged to transmit all of the natural frequencies of the resonators 22 of the diagonal.

This means that each field line arranged to transmit the signal to at most three resonators 22 is connected to a single neuron 16 while each field line arranged to transmit the signal to more than three resonators 22 is connected to two neurons 16.

According to the proposed example, the first line of fields transmits the signal coming from the neuron 16 of the first layer of neurons 12 whose frequency is fi,i,i , (a single resonator 22 on the path of the first line of fields) the second line of fields transmits the signal coming from the neuron 16 of the first layer of neurons 12 whose frequency is fi, ₂ ,i (two resonators 22 on the path of the second line of fields), the third line of fields transmits the signal from neuron 16 of the first layer of neurons 12 whose frequency is f 1,3,1 (three resonators 22 on the path of the third line of fields), the fourth line of fields transmits the signal from neuron 16 of the first layer of neurons 12 whose frequency is f ₂ ,i,i and the signal of the neuron 16 of the first layer of neurons 12 whose frequency is f 1 ,4,1 and so on. This only describes the operation of the device 14 without specifying the connections which appear better in figure 6 which can be described as follows using a slightly modified formalism.

The convolution filter FC is adapted to apply a set of convolution coefficients to a set of input values ordered according to a first order to obtain a set of output values ordered according to a second order, the set of convolution coefficients being arranged according to a third order, the first order, the second order and the third order are each a row-by-row traversal successively for each channel of a corresponding matrix in a representation of the operation of an FC convolution filter in the form of a matrix operations where each matrix is a set of boxes where each box presents a value specific to at least one channel.

These line-to-line traversal orders are visible schematically in FIG. 2 for a channel. We understand that the same operation takes place for the next channel and so on until the last channel.

Each cell is thus ordered as follows: the first cell of the first row of the matrix corresponding to the first channel, ..., the last cell of the first row of the matrix corresponding to the first channel, the first cell of the second row of the matrix corresponding to the first channel, ... , the last cell of the second row of the matrix corresponding to the first channel, ... , the first cell of the last row of the matrix corresponding to the first channel, ... , the last box of the last row of the matrix corresponding to the first channel, the first box of the first row of the matrix corresponding to the second channel, ..., the last box of the first row of the matrix corresponding to the second channel, the first box of the second row of the matrix corresponding to the second channel, ... , the last box of the second row of the matrix corresponding to the second channel, ... , the first box of the last row of the matrix corresponding to the second xth channel, ... , the last cell of the last row of the matrix corresponding to the second channel > the first cell of the first row of the matrix corresponding to the last channel, ... , the last cell of the first row of the matrix corresponding to the last channel, the first cell of the second row of the matrix corresponding to the last channel, ... , the last cell of the second row of the matrix corresponding to the last channel, ... , the first cell of the last row of the matrix corresponding to the last channel, ... , the last cell of the last row of the matrix corresponding to the last channel.

The arrangement of device 14 is particular here. The first layer of neurons 12 comprises neurons 16 which physically implement a respective input value. These neurons 16 are aligned on the same first column according to the first order. The top neuron 16 implements the input value corresponding to the first box of the first line of the first channel and the one below implements the input value corresponding to the second box of the first line of the first channel and so right now.

The second layer of neurons 12 comprises neurons 16 which physically implement a respective output value. These neurons 16 are aligned on the same second column according to the second order. The top neuron 16 implements the output value corresponding to the first box of the first line of the first channel and the one below implements the output value corresponding to the second box of the first line of the first channel and so on . The second column is parallel to the first column.

The resonators 22 of the same synaptic chain 20 are arranged according to the third order. The resonators 22 have been grouped together in the diagram of Figure 6 to illustrate this.

As shown in the connections in Figure 6, each synaptic chain 20 is connected to a neuron of the second layer of neurons 12 and each synapse 22 is connected to the neuron of the first layer of neurons 12 physically implementing the input value on which the convolution coefficient of said synapse is to be applied.

The synaptic chains 20 are parallel to each other and arranged in a direction perpendicular to the two columns, the resonators 22 physically implementing the same coefficients of the convolution filter FC being spatially aligned along the same line, the different lines being parallel to each other.

Such an arrangement makes it possible to obtain the same compactness as the device 14 of FIG. 3 by dispensing with the use of an adder 23 thanks to a clever use of the recurrences specific to convolutional neural networks.

In addition, such an arrangement allows no connection to cross.

Other embodiments can be envisaged by combining the aforementioned embodiments when technically possible.

Claims

24 CLAIMS

1. Device (14) physically implementing a convolution filter (FC) of a neural network, the convolution filter (FC) taking as input an input table (IE) coded according to at least one input channel for obtain at output an output table (IS) coded according to at least one output channel, the input table (IE) being a set of input boxes arranged according to an input matrix, each box presenting a value specific to each input channel, a first layer of neurons (12) physically implementing the input array (IE), each neuron (16) of the first layer of neurons (12) being an oscillator, in particular a radiofrequency oscillator, oscillating at a natural frequency, each neuron (16) of the first layer of neurons (12) corresponding in a one-to-one manner to the value of a box of an input channel, the output table (IS) being a set of boxes of output arranged according to an output matrix, each box presenting a value specific to each c output channel, a second layer of neurons (12) physically implementing the output array (IS), each neuron (16) of the second layer of neurons (12) being an oscillator, in particular a radiofrequency oscillator, oscillating at a natural frequency , each neuron (16) of the second layer of neurons (12) corresponding one-to-one to the value of a box of an output channel, the convolution filter (FC) being a set of convolution kernels (NC) , each convolution kernel (NC) being specific to an output channel, each convolution kernel (NC) delimiting a matrix subset having a size strictly less than the size of the input matrix, the matrix subset a convolution nucleus (NC) comprising a set of convolution coefficients, the device (14) comprising sets of synaptic chains (20), each synaptic chain (20) comprising synapses (22), each synapse (22) being a resonator (22), not ment a spintronic resonator, the resonators (22) being in series, each resonator (22) having an adjustable resonant frequency, each set of synaptic chains (20) being capable of applying a respective convolution nucleus (NC), each synaptic chain (20) of the same set of synaptic chains (20) bringing together the resonators (22) whose resonance frequency is the frequency of the neurons (16) implementing the values of the cells of the matrix subset of the input matrix associated with the convolution kernel (NC) of the set of synaptic chains (20) and whose first element is an entry box having respective coordinates, each resonator (22) having a frequency adjusted by a frequency adjustment unit (U1, U2, U3) so that the frequency of the resonator (22) is equal to the resonance frequency to which is added a frequency shift depending on the convolution coefficient of the convolution kernel (NC) to be applied to the neuron (16), the number of synaptic chains (20) of a same set being equal to the number of cells of the output table (IS) that the convolution kernel (NC) makes it possible to obtain by application to the input table (IE).

2. Device according to claim 1, in which the frequency adjustment unit (U1, U2, U3) is the same for the resonators (22) of the same set of synaptic chains (20) for which the coefficient of convolution to be applied is identical.

3. Device according to claim 1 or 2, in which there is defined for each synaptic chain (20) an input and an output, the device (14) comprising at least one amplifier (24, 26) positioned either at the input or to the output.

4. Device according to any one of claims 1 to 3, wherein each frequency adjustment unit (UU1, U2, U3) is chosen from the list consisting of a direct current applicator, a pulse applicator of current and a voltage applicator.

5. Device according to any one of claims 1 to 4, wherein the resonators (22) are arranged in a matrix whose columns are the resonators (22) having the same convolution coefficient to be applied.

6. Device according to any one of claims 1 to 5, wherein the device (14) comprises a summer (23) of the signals from the neurons (16) of the first layer of neurons (12).

7. Device (14) physically implementing a convolution filter (FC) of a neural network, the convolution filter (FC) being capable of applying a set of convolution coefficients to a set of input values ordered according to a first order to obtain a set of output values ordered according to a second order, the set of convolution coefficients being arranged according to a third order, the first order, the second order and the third order are each a row-to-row traversal successively for each channel of a corresponding matrix in a representation of the operation of a convolution filter (CF) as a matrix operation where each matrix is a set of bins where each bin has a value specific to at least one channel, the device (14) comprising:

- a first layer of neurons (12) in which each neuron (16) is an oscillator, in particular a radiofrequency oscillator, oscillating at a natural frequency and physically implements a respective input value, the neurons (16) being aligned on a same first column according to the first order,

- a second layer of neurons (12) of which each neuron (16) is an oscillator, in particular a radiofrequency oscillator, oscillating at a natural frequency and physically implements a respective output value, the neurons (16) being aligned on the same second column according to the second order, the second column being parallel to the first column,

- sets of synaptic chains (20), each synaptic chain (20) comprising synapses (22) in series physically implementing a respective convolution coefficient on at least one input value, each synapse (22) being connected to the neuron of the first layer of neurons (12) physically implementing the input value to which the convolution coefficient of said synapse is to be applied, each synapse (22) being a resonator (22), in particular a spintronic resonator, having a resonant frequency , and having a frequency adjustable by a frequency adjustment unit (U1, U2, U3) so that the frequency of the resonator (22) is equal to the resonant frequency to which is added a frequency offset depending on the convolution coefficient respective that the resonator (22) physically implements, the resonators (22) of the same synaptic chain (20) being arranged according to the third order, each synaptic chain (20) being connected to a neur one of the second layer of neurons (12), the synaptic chains (20) being parallel to each other and arranged in a direction perpendicular to the two columns, the resonators (22) physically implementing the same coefficients of the convolution filter (FC) being spatially aligned along the same line, the different lines being parallel to each other.

8. System (10) physically implementing a convolutional neural network comprising a device (14) according to any one of claims 1 to 7.