CN115809694A - Multi-wavelength parallel based multi-task diffraction neural network equipment and processing method - Google Patents

Abstract

The invention relates to the technical field of optical computation and artificial intelligence, and provides a multi-wavelength parallel based multitask diffraction neural network device and a processing method thereof, wherein the device comprises an input unit, a diffraction modulation structure, an optical detection component and a processing unit; the input unit is used for modulating the input of the N tasks to N wavelengths, and inputting a diffraction modulation structure after a mixed light beam is formed by superposing light fields, wherein the diffraction modulation structure is used for outputting the input mixed light beam after parallel processing of each wavelength component; the optical detection component is used for detecting the light intensity of the output plane of the diffraction modulation structure; the output plane comprises M categories of detection areas, and each detection area comprises N sub-areas; the processing unit is used for determining inference results of N tasks corresponding to N wavelengths according to the light intensity distribution of the corresponding sub-area in each detection area. The problem that the D2NN can only adapt to a single deep learning task and is poor in universality is solved, multi-task parallelism is achieved, and the universality is improved.

Description

Multi-wavelength parallel based multi-task diffraction neural network equipment and processing method

Technical Field

The invention relates to the technical field of optical computation and artificial intelligence, in particular to multi-task diffraction neural network equipment based on multi-wavelength parallelism and a processing method.

Background

With the development of scientific technology, photon computation has been widely applied due to advantages of light speed processing, low power consumption, high throughput and the like, for example, photons can be used instead of electrons to execute an Artificial Intelligence (AI) task, at present, an Artificial Neural Network model is realized by a photon Neural Network based on photon computation, which can greatly improve computation speed and energy efficiency, and in different photon Neural Network architectures, a diffraction Deep Neural Network (D2 NN) therein can perform large-scale Neural information processing, and the D2NN includes a plurality of diffraction layers and is composed of diffraction optical elements. In practical application, the D2NN can only adapt to a single task, and the universality is poor.

Disclosure of Invention

The invention provides multi-wavelength parallel-based multi-task diffraction neural network equipment and a processing method, which are used for solving the defects that D2NN in the prior art can only adapt to a single task and is poor in universality, realizing multi-task parallel and improving the universality.

The invention provides a multitask diffraction neural network device based on multi-wavelength parallelism, which comprises: the device comprises an input unit, a diffraction modulation structure, a light detection assembly and a processing unit; wherein the diffractive modulation structure comprises a plurality of diffractive layers, each of the diffractive layers comprising a plurality of diffractive optical elements;

the input unit is used for modulating the input of the N tasks to N wavelengths, and inputting the N wavelengths into the diffraction modulation structure after forming a mixed light beam through light field superposition, wherein the N wavelengths correspond to the input of the N tasks one by one;

the diffraction modulation structure is used for parallelly processing and outputting each wavelength component of the input mixed light beam;

the light detection component is used for detecting the light intensity of the output plane of the diffraction modulation structure; the output plane comprises M categories of detection regions, each detection region comprises N sub-regions, and the N sub-regions correspond to the N wavelengths one by one; wherein M and N are both positive integers;

the processing unit is used for determining inference results of N tasks corresponding to N wavelengths according to the light intensity distribution of each sub-region in each detection region.

According to the multi-wavelength parallel based multitask diffraction neural network device provided by the invention, the processing unit is specifically used for:

and for each task, selecting the sub-region with the maximum light intensity from the sub-regions corresponding to the wavelengths corresponding to the input of the task in the M detection regions, and taking the class of the detection region in which the sub-region with the maximum light intensity is positioned as an inference result of the task.

According to the multi-wavelength parallel based multitask diffraction neural network equipment provided by the invention, the phase modulation coefficient of each diffraction optical element is obtained by the following method:

determining a first loss function based on an error between a detected value and a real value of the light intensity of the sub-region corresponding to the wavelength corresponding to the input of the task;

determining a second loss function based on the sum of the light intensities outside each sub-region corresponding to the wavelength corresponding to the input of the task;

determining a target loss function based on the first loss function and the second loss function;

determining a phase modulation factor for each of the diffractive optical elements based on the target loss function.

According to the multi-wavelength parallel based multitask diffraction neural network device provided by the invention, the N tasks are from different data sets.

According to the multi-wavelength parallel based multitask diffraction neural network device provided by the invention, the optical detection component comprises an optical detector for detecting the light intensity of the whole output plane; alternatively, the light detection assembly comprises a light detector corresponding to each of the sub-regions.

According to the multi-wavelength parallel based multitask diffraction neural network equipment provided by the invention, the input unit comprises N-1 beam splitters which are sequentially arranged along the direction of an optical path; when the beam splitter is the first beam splitter in the optical path direction, the input of the beam splitter comprises the input of the task corresponding to two wavelengths; when the beam splitter is not the first beam splitter in the direction of the optical path, the inputs of the beam splitter comprise the output of the previous beam splitter and the input of the task corresponding to one wavelength; when the beam splitter is the last beam splitter in the optical path direction, the output of the beam splitter is the mixed beam.

The invention also provides a processing method applied to any one of the above multi-wavelength parallel based multitask diffraction neural network devices, which comprises the following steps:

after the input unit modulates the input of N tasks to N wavelengths and forms mixed light beams through light field superposition to be input to the diffraction modulation structure, the light detection assembly detects the light intensity of an output plane of the diffraction modulation structure;

and the processing unit determines inference results of N tasks corresponding to N wavelengths according to the light intensity of each sub-region in each detection region of the output plane.

According to the processing method provided by the invention and applied to any one of the above multi-wavelength parallel based multitask diffraction neural network devices, the processing unit determines inference results of N tasks according to the light intensity of each subarea in each detection area of the output plane, and the method comprises the following steps:

and for each task, selecting the sub-region with the maximum light intensity from the sub-regions corresponding to the wavelengths corresponding to the input of the task in the M detection regions, and taking the class of the detection region in which the sub-region with the maximum light intensity is positioned as an inference result of the task.

According to the processing method provided by the invention and applied to any one of the above multi-wavelength parallel based multitask diffraction neural network devices, the phase modulation coefficient of each diffractive optical element is obtained by:

determining a first loss function based on an error between a detected value and a real value of the light intensity of the sub-region corresponding to the wavelength corresponding to the input of the task;

determining a second loss function based on the sum of the light intensities outside each sub-region corresponding to the wavelength corresponding to the input of the task;

determining a target loss function based on the first and second loss functions;

determining a phase modulation factor for each of the diffractive optical elements based on the target loss function.

According to the processing method provided by the invention, which is applied to any one of the multi-wavelength parallel based multitask diffraction neural network equipment, the task is an image classification task.

The invention provides multi-wavelength parallel based multi-task diffraction neural network equipment, which modulates the input of N tasks to N wavelengths through an input unit, forms a mixed light beam through light field superposition, inputs a diffraction modulation structure after the mixed light beam is input, the diffraction modulation structure is used for outputting each wavelength component of the input mixed light beam after parallel processing, and an optical detection assembly detects the light intensity of an output plane of the diffraction modulation structure, wherein the output plane comprises M types of detection areas, each detection area comprises N sub-areas, and the N sub-areas correspond to the N wavelengths one by one.

Drawings

In order to more clearly illustrate the technical solutions of the present invention or the prior art, the drawings needed for the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and those skilled in the art can also obtain other drawings according to the drawings without creative efforts.

FIG. 1 is one of the schematic diagrams of a multi-wavelength parallel based multitask diffractive neural network device provided by the present invention;

FIG. 2 is a second schematic diagram of the multi-wavelength parallel based multitask diffraction neural network device provided by the present invention;

FIG. 3 is a third schematic diagram of a multi-wavelength parallel based multitask diffraction neural network device provided by the present invention;

FIG. 4 is a fourth schematic diagram of the multi-wavelength parallel based multitask diffraction neural network device provided by the present invention;

FIG. 5 is a fifth schematic diagram of a multi-wavelength parallel based multitask diffraction neural network device provided by the present invention;

FIG. 6 is a sixth schematic diagram of a multi-wavelength parallel based multitask diffractive neural network device provided by the present invention;

fig. 7 is a schematic diagram of a processing method applied to a multitask diffraction neural network device based on multi-wavelength parallel provided by the invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is obvious that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

In the prior art, the D2NN cannot multiplex different tasks in parallel, and the inventor finds that realizing different task multiplexing in the D2NN is significant for improving the universality and expanding the application of the D2NN in different scenes, but performing multiple AI tasks in parallel by using the D2NN is still very troublesome, and competition among different tasks is a major obstacle in training because of possible disastrous forgetting caused by the fact that knowledge of previously learned tasks is suddenly lost in a multi-task training process, so that the performance of each task is reduced.

Therefore, the invention provides a multitask diffraction neural network device based on multi-wavelength parallel, which can realize multitask in D2NN. The multi-wavelength parallel based multitask diffraction neural network device of the present invention is described below with reference to fig. 1 to 6.

The embodiment provides a multitask diffraction neural network device based on multi-wavelength parallelism, as shown in fig. 1, including: an input unit 110, a diffractive modulation structure 120, a light detection assembly 130 and a processing unit 140; wherein the diffractive modulation structure 120 comprises a plurality of diffractive layers, each of the diffractive layers comprising a plurality of diffractive optical elements;

the input unit 110 is configured to modulate inputs of the N tasks to N wavelengths, and input the diffraction modulation structure after a mixed beam is formed by superimposing optical fields, where the N wavelengths correspond to the inputs of the N tasks one to one;

the diffraction modulation structure 120 is configured to output the input wavelength components of the mixed light beam after parallel processing;

the light detection component 130 is used for detecting the light intensity of the output plane of the diffraction modulation structure; the output plane comprises M categories of detection regions, each detection region comprises N sub-regions, and the N sub-regions correspond to the N wavelengths one by one; wherein M and N are both positive integers;

the processing unit 140 is configured to determine inference results of N tasks corresponding to N wavelengths according to light intensity distribution of each of the sub-regions in each of the detection regions.

Wherein the plurality of diffractive optical elements in the diffractive layer are densely arranged. The input unit 110, the diffractive modulation structure 120, the light detection assembly 130 and the processing unit 140 in this embodiment form a D2NN.

The values of M and N may be greater than or equal to 2, so that processing of multiple tasks can be performed simultaneously. The N tasks may be tasks from the same dataset. Alternatively, the N tasks may be tasks from different data sets, enabling multiple tasks to be performed in parallel, also known as optical multi-tasking. And simultaneously modulating the input of the N tasks to N wavelengths, forming a mixed beam through light field superposition, inputting a diffraction modulation structure, carrying out parallel processing on each wavelength component of the input mixed beam by the diffraction modulation structure, outputting, and matching subsequent optical detection components and processing units to obtain an inference result of the task carried by each wavelength, wherein based on the inference result, the corresponding D2NN can be called as a multi-wavelength D2NN. Here, the wavelength dimension is exploited to improve computational throughput, different inputs are encoded (i.e., modulated) into different wavelengths, and photon calculations are performed in the spatial and spectral dimensions. High parallel processing of multiple inputs is achieved.

The task may be an image processing task, such as an image classification task, or may be another image processing task. Correspondingly, the inference result of the task is the classification result of the image. The N tasks include N images, which may be images acquired under illumination of different wavelengths, which may contain target objects to be classified, and so on.

Therefore, the multi-wavelength D2NN has the inherent advantages of parallel processing of a plurality of tasks, and is high in processing speed, low in power consumption and high in throughput. By encoding different tasks to different wavelengths, competition between different tasks can be significantly mitigated and high performance per task is maintained. For each new task, new wavelengths can be easily added to realize the new task, and the realization cost is extremely low. Therefore, the multi-wavelength D2NN can make full use of photon calculation, and is beneficial to realizing a more universal brain-like intelligent architecture. In implementation, multiple tasks may be performed simultaneously on different data sets without inter-task adjustment.

In an implementation, there may be M categories, and the detection area of each category, that is, the detection area of M categories, may be set on the output plane of the D2NN, and further, the detection area of each category is divided into N parts to obtain N sub-areas, where the N sub-areas correspond to N wavelengths one to one, and each sub-area represents the input category at the corresponding wavelength. The 9 categories are illustrated in fig. 2, and correspondingly, the detection areas include detection areas No. 0, no. 1, no. 2, … …, and No. 9, and each detection area has a plurality of different sub-areas.

As shown in fig. 2, according to the preset sequence, the ith task corresponds to the ith wavelength λ in the N wavelengths _i Wavelength λ _i The information of the ith task is encoded, i =1, …, N. In FIG. 2, the number of diffraction layers is shown as 5, denoted by L ₁ 、L ₁ 、…L ₅ . The mixed beam may be input to the diffractive modulation structure via an aperture stop.

Based on the approximation theory of the multi-wavelength optical system, the transformation of the multi-wavelength optical field can be regarded as the combination of independent transformation of each wavelength coherent optical field and follows the principle of light intensity superposition, so that the input optical field with the wavelength λ i encodes the ith task and is detected after being transmitted through the diffraction modulation structure. In the implementation, a linear D2NN with a complex transformation function M (Φ) is considered, where Φ represents the phase modulation coefficients of the diffractive optical elements in the plurality of phase-only diffractive layers. The complex transformation function can refer to the related art, and is not described herein.

In this embodiment, the phase modulation coefficients of each diffraction layer are the same at different wavelengths designed based on a multi-wavelength Optical element (DOE). Thus, the wavelength λ _i Output light field U of _i "may be expressed as:

U _i `＝M _i (Φ)U _i (1)

wherein, U _i Represents the wavelength lambda _i And accordingly, the output optical field intensity distribution of the diffractive modulation structure can be expressed as:

I _i ＝|U _i `| ² ＝|M _i (Φ)U _i | ² (2)

I _i represents the wavelength lambda _i For multi-wavelength D2NN, the total optical field intensity distribution I of different wavelengths can be expressed as corresponding to each wavelengthSuperposition of the light field intensity distributions detected at the subregions:

I＝Σ _i I _i ＝Σ _i |M _i (Φ)U _i | ² (3)

correspondingly, in the detection areas of M categories on the output plane, according to the preset sequence, the jth detection area includes N sub-areas, j =1, …, M, which are indexes of different detection areas respectively, and can represent different categories, and the ith sub-area of the jth detection area is recorded as D ^j _i Wherein i =1, …, N, is characterized at wavelength λ _i An index of the corresponding task.

In practice, the wavelength λ may be selected by an optical filter _i And then detected by an optical detection component, and an optical filter is an instrument for selecting wavelength and is also called a wavelength selection filter. A wavelength selective filter may be applied to each sub-region to eliminate cross-talk between wavelength channels during intensity detection, improving task performance, in which case each sub-region only detects light intensity at the corresponding wavelength, sub-region D ^j _i Light intensity I (D) of ^j _i )＝I _i (D ^j _i ) I.e. sub-region D ^j _i At a wavelength of λ _i To the light intensity of (c). Or not through the wavelength selective filter, subregion D ^j _i The light intensity of (a) is the sum of the light intensities of all the wavelength components.

Compared with the spatial multiplexing of a plurality of D2NNs, in the multi-wavelength diffraction light calculation, optical signals with different wavelengths are independent of each other, and no crosstalk exists. Thus, increasing the number of wavelengths and diffractive optical elements in D2NN increases computational throughput and simplifies more tasking.

Thus, in this embodiment, the input of N tasks is modulated to N wavelengths by the input unit, a mixed beam is formed by superimposing optical fields, and then the mixed beam is input to the diffraction modulation structure, the diffraction modulation structure is used to output each wavelength component of the input mixed beam after parallel processing, the optical detection component detects the light intensity of the output plane of the diffraction modulation structure, the output plane includes M types of detection regions, each detection region includes N sub-regions, and the N sub-regions correspond to the N wavelengths one to one, based on which, the processing unit can determine the inference result of the N tasks corresponding to the N wavelengths according to the light intensity of each sub-region in each detection region, and implement parallel processing of multiple tasks, thereby implementing multiple-wavelength D2NN, so that compared with the prior art, the input of different tasks can be encoded to different wavelengths by multiple-wavelength D2NN, and implement parallel processing of different tasks, which not only has strong versatility, but also greatly improves the computation throughput.

In an exemplary embodiment, the input unit includes N-1 beam splitters sequentially arranged in an optical path direction; when the beam splitter is the first beam splitter in the optical path direction, the input of the beam splitter comprises the input of the task corresponding to two wavelengths; when the beam splitter is not the first beam splitter in the direction of the optical path, the inputs of the beam splitter comprise the output of the previous beam splitter and the input of the task corresponding to one wavelength; when the beam splitter is the last beam splitter in the optical path direction, the output of the beam splitter is the mixed beam.

Wherein, the beam splitter is a half-reflecting and half-transmitting mirror. As shown in fig. 3, N wavelengths λ ₁ ，λ ₂ ，λ ₃ ，…,λ _N The N-1 beam splitters are arranged in sequence along the light path direction, corresponding to the N-1 beam splitters, which is indicated by the thin arrows below in the figure. Along the optical path, for the first beam splitter, its input includes the input of two wavelength-specific tasks, one of which is transmitted light and the other of which is reflected light. Starting with the second beam splitter, the input includes the output of the previous beam splitter as transmitted light and the input of a wavelength-specific task as reflected light. Therefore, the input of tasks corresponding to all the wavelengths can be mixed by the last beam splitter to obtain a mixed beam, and the mixed beam is output to the diffraction modulation structure. In the embodiment, the modulation of the input of different tasks can be realized through a plurality of beam splitters, and the optical fiber coupler is simple in structure and easy to realize.

In an exemplary embodiment, the processing unit is specifically configured to:

and for each task, selecting the sub-region with the maximum light intensity from the sub-regions corresponding to the wavelengths corresponding to the input of the task in the M detection regions, and taking the class of the detection region in which the sub-region with the maximum light intensity is positioned as an inference result of the task.

Wherein the light detection assembly comprises a light detector for detecting the light intensity of the entire output plane; alternatively, the light detection assembly comprises a light detector corresponding to each of the sub-regions. The light detector may be a grayscale camera, or may be a light intensity sensor, or the like. In implementation, appropriate settings can be flexibly selected according to actual needs.

In particular, the wavelength λ is determined _i Corresponding sub-region D ¹ _i ，D ² _i ，……,D ^M _i Obtaining the light intensity of each sub-region, and the type of the detection region in which the sub-region with the maximum light intensity is positioned, i.e. the wavelength lambda _i The category of the corresponding task.

In this embodiment, for a certain task, the higher the light intensity of the sub-region of each detection region is, the higher the probability of the class belonging to the detection region is, and therefore, the more accurate classification can be achieved by using the class of the detection region in which the sub-region with the highest light intensity is located as the class of the task.

In an exemplary embodiment, the phase modulation factor of each of the diffractive optical elements is obtained by:

determining a first loss function based on an error between a detected value and a real value of the light intensity of the sub-region corresponding to the wavelength corresponding to the input of the task;

determining a second loss function based on the sum of the light intensities outside each sub-region corresponding to the wavelength corresponding to the input of the task;

determining a target loss function based on the first loss function and the second loss function;

determining a phase modulation factor for each of the diffractive optical elements based on the target loss function.

In practical application, the multi-wavelength D2NN can be trained in advance, and optical multi-task learning is achieved. Illustratively, a joint optimization method may be employed to train the multi-wavelength D2NN. The joint optimization method is a training method adopting the target loss function.

The target loss function includes two parts, one part is the first loss function, the first loss function is obtained based on an error between a detection value and a real value of the light intensity of the sub-region corresponding to the wavelength corresponding to the task, the target loss function takes a minimum error between the detection value and the real value as a target, and can improve the classification accuracy of the D2NN, the other part is the second loss function, the second loss function is obtained based on a sum of the light intensities outside the sub-regions corresponding to the wavelength corresponding to the task, the target loss function also takes a sum of the light intensities outside the minimum detection region as a target, so that the energy transmission efficiency of the multi-wavelength D2NN is maximized, and the classification accuracy of the D2NN can be further improved.

Illustratively, the target loss function includes:

wherein L (Gi, pi) represents the execution wavelength λ _i And in the corresponding task, a softmax cross entropy loss function generated by the error of the detected value Pi and the real value Gi of the light intensity of the sub-area, namely a first loss function is called a softmax cross entropy item. Where Gi is a one-hot vector of length M.

Representing the sum of the light intensities outside the sub-region in the detection area evaluated by the mean square error, i.e. the above-mentioned second loss function, referred to as the energy efficiency constraint, where I _i 。I _i I.e. wavelength lambda _i The total light intensity of the light source (c),

i.e. wavelength lambda _i The light intensity of the sub-region of each corresponding detection region

Sum, MSE () represents the mean square error.

During the training process, different wavelengths share the same phase modulation index at each diffraction layer. The optical multitasking function can be realized by solving the optimal solution of the formula (4) and performing iterative updating.

In an implementation, a multi-wavelength D2NN may be trained using a random gradient descent method, with inputs of training data sets for different tasks being encoded into light field amplitudes at different wavelengths into the D2NN. Error back-propagation is performed according to the objective loss function to optimize the network structure of the D2NN and the phase modulation coefficients of the diffractive optical elements.

In the field of manufacturing of multi-wavelength diffractive optical elements, the same phase modulation characteristics at different wavelengths can be achieved. Illustratively, the geometry of each diffractive optical element may be determined such that the optical path length of each diffractive optical element has the same phase value for each wavelength. In particular, this can be accomplished by adding an integral poly-phase retardation (e.g., 2 π) at one wavelength until the other wavelength reaches the appropriate phase retardation. The overall physical height will be determined according to actual accuracy. Alternatively, the optical path length at each wavelength can be controlled by using the refractive index change of the dispersive material at different wavelengths. It can also be designed by combining several aligned diffractive optical elements made of different materials, similar to polarization selective diffractive optical elements. The flexibility of wavefront manipulation in different physical dimensions (e.g., phase, amplitude, wavelength, and polarization) makes it possible for diffractive optical elements to encode multiple wavelengths. For example, a sub-surface composed of different types of nano-patches whose spatially varying rotation angles are multiplexed in one sub-wavelength unit can cause the diffractive optical element to resonate with different wavelengths.

The scheme provided by the invention is explained in more detail through specific scenes.

In the present embodiment, a task of classifying images will be described. The application of multi-wavelength D2NN in a highly parallel classification task is verified firstly, and multiple inputs can be classified simultaneously when a single classification task is executed.

Specifically, a three-wavelength D2NN with five diffraction layers was constructed using the PyTorch deep learning framework to classify MNIST datasets that can recognize three handwritten digits at each time. Consider a visible wavelength of 400nm to 700nm, where the input light source is set to a combination of three wavelengths of 400nm, 550nm and 700nm, encoding three handwritten numbers, respectively. Thus, the detection area of each class on the output plane is divided into three sub-areas accordingly.

Adam optimizers may be used for D2NN training to optimize the phase modulation coefficients of the diffractive optical element. The size of each diffractive optical element is set to 4um × 4um.

First, the performance of the multi-wavelength D2NN was evaluated, with the number of diffractive optical elements of each diffractive layer set to 200 × 200, and the corresponding diffractive layer size 0.8mm × 0.8mm (see fig. 4). The network performance at different numbers of diffractive optical elements per layer, i.e. K × K, K =200, 400, 600, 800 (see fig. 4), was further evaluated and compared. The number of diffraction layers was set to 5 and the distance between successive layers was optimized according to diffraction theory. The training batch size (batch size) is set to 32, the initial learning rate is set to 0.01, and halved, i.e., multiplied by 0.5, after each epoch during training. The D2NN training converges after five cycles to achieve an ideal mapping function for multi-wavelength input and output. D2NN received 60000 handwritten digit trains and performed blind tests with 10000 handwritten digits. For the diffractive layer of the diffractive optical element number K × K, each number of the pixel number 28 × 28 is first adjusted to K/2 × K/2, and refilled to K × K.

The numerical evaluation results are shown in fig. 4 as parts b and c, in which the performance of the multi-wavelength D2NN is valid on each class of detection area regardless of the use or non-use of the wavelength selective filter. Sections a-c of fig. 4 show exemplary results of classifying three handwritten input numbers at the same time, namely, "7", "2", and "5", encoded at wavelengths of 700nm, 550nm, and 400nm, respectively, with the number of diffractive optical elements per layer being 200 × 200. The classification result for each wavelength is determined by the maximum light intensity between the sub-regions corresponding to the detection regions of the class, where the three input numbers are indicated by three white arrows, as shown on the left side of the portions b and c in fig. 4. The energy distribution of the classification results for the three inputs at different wavelengths in sections b and c of fig. 4 shows that the D2NN can significantly identify the sub-region with the largest light intensity for proper classification.

Since the wavelength selective filter can eliminate the wavelength crosstalk in the detection process, for wavelengths of 700nm, 550nm and 400nm, the classification accuracy of the multi-wavelength D2NN using the wavelength selective filter is respectively 95.9%, 96.4% and 96.9%, which is slightly higher than that of the broadband wavelength detection without the wavelength selective filter, the latter is respectively 95.0%, 95.7% and 96.4%.

For both of these settings, as the number of diffractive optical elements per diffraction layer increases, the classification accuracy of the multi-wavelength D2NN at each wavelength further increases, as shown in part D in fig. 4. In the case where the number of diffractive optical elements per layer is 800 × 800, the classification accuracy of the multi-wavelength D2NN with the wavelength selective filter reaches 98.2%, 98.1%, and 98.1% for wavelengths of 700nm, 550nm, and 400nm, respectively, which is equivalent to training three single wavelengths D2NN with serial input, i.e., digital sequential input. The result shows that the multi-wavelength D2NN can remarkably improve the parallel computing capability. By encoding different classification tasks to different wavelengths and using multi-wavelength D2NN for multi-task learning, different machine learning tasks can be performed in parallel in D2NN.

Regarding the ability of the multi-wavelength D2NN for optical multitask learning, a multitask classifier is firstly constructed for classifying an MNIST data set and a fast-MNIST (FMNIST) data set, a classification task corresponding to the MNIST data set is recorded as a task I, and a classification task corresponding to the FMNIST data set is recorded as a task II. Both data sets included 60000 training samples and 10000 test samples, with 10 class numbers and, correspondingly, 10 detection regions. Thus, dual wavelength D2NN is constructed by dividing each of the ten detection regions into two sub-regions, see fig. 5,a, below which the classification results for tasks I and II are represented, respectively. The wavelength of the handwritten digit codes of the task I is 700nm, and the wavelength of the fashion product codes of the task II is 400nm. In the case where the other network arrangement is the same as that of fig. 4, first, two wavelengths D2NN are set to have five diffraction layers each having a number of diffractive optical elements of 200 × 200 without a wavelength selective filter on the photodetector. Part a of fig. 5 shows an example result of classifying the handwritten digit "7" with class number 7 in the MNIST data set and the fashion product "pullover" with class number 2 in the FMNIST data set at the same time. In part a of fig. 5, the energy distributions of the classification results of the two tasks show that the multi-wavelength D2NN can significantly identify the sub-region with the maximum light intensity, thereby achieving the correct classification. The maximum light intensity output of the task I and the task II is respectively concentrated in the upper sub-area of the No. 7 detection area and the lower sub-area of the No. 2 detection area.

The trained dual-wavelength D2NN is subjected to blind test on MNIST and FMNIST test data sets, and the classification accuracy rate respectively reaches 95.6% and 86.8%. The corresponding confusion matrix and energy distribution matrix are shown in the b and c parts of fig. 5, respectively, and the classification results of all samples and the energy distribution percentages of the two tasks are statistically summarized. The average energy percentages for the correct categories for the two tasks were 20.8% and 21.8%, respectively.

Further, the performance of the dual wavelength D2NN for parallel execution of two tasks was compared, and the single wavelength D2NN for parallel execution of two tasks by multiplexing two images from two data sets, respectively, as inputs by overlapping, as shown in table 1. The single wavelength D2NN performs 92.4% and 83.1% of the two tasks in parallel, respectively, which is much lower than the dual wavelength D2NN.

In addition, two single-wavelength D2NNs were also trained, one for each of the two tasks, with task I and task II being 97.1% and 87.5%, respectively. In order to improve the performance of the two-wavelength D2NN, the number of diffractive optical elements per diffraction layer may be increased, and the classification accuracy reaches 97.5% and 88.0% for two tasks with 400 × 400 diffractive optical elements per diffraction layer, respectively. The performance can be further improved by using the wavelength selective filter in the class detection area, and the classification accuracy rates of the two tasks of 200 multiplied by 200 diffraction optical elements of each diffraction layer respectively reach 95.9 percent and 87.0 percent; the classification accuracy reaches 97.6% and 88.9% for each diffraction layer of 400 × 400 diffractive optical elements, and shows a considerably higher classification accuracy than when two tasks are respectively performed by separately training two single-wavelength D2NN. Table 1 summarizes the experimental results, verifying that the designed dual wavelength D2NN with joint optimization method can successfully classify targets in parallel from two tasks without any adjustment of the diffraction layers of the two tasks.

TABLE 1 results of the experiment

Regarding the ability of multi-wavelength D2NN for optical multitask learning, a four-wavelength D2NN was also constructed for four task classes, which can be simultaneously classified from MNIST, FMNIST, kuzushiji MNIST (i.e., KMNIST) and Extended MNIST (i.e., EMNIST) datasets, and sequentially labeled as task I, task II, task III and task IV. KMNIST contains ancient text images with the same data set size and class number as the MNIST and FMNIST data sets. Ten classes of handwritten letters were randomly selected from the EMNIST dataset and kept the same dataset size as the other three tasks, i.e. 60000 training samples and 10000 test samples. The data sets for the four tasks from task I to task IV are encoded at wavelengths of 700nm, 600nm, 500nm, and 400nm, respectively. In this numerical experiment, the designed four-wavelength D2NN did not use a wavelength selective filter with low hardware complexity. In the case where other network settings are the same as those of fig. 4 and 5, the classification accuracy of the four-wavelength D2NN in parallel performing four tasks at different network scales was evaluated and compared with the classification accuracy of the single-wavelength D2NN, as shown in fig. 6. For four-wavelength D2NN with five diffractive layers, the number of diffractive optical elements per layer was 200 × 200, and the classification accuracy of the four tasks, from task I to task IV, was 92.8%, 83.0%, 81.0%, and 90.4%, respectively, which was significantly higher than 64.6%, 68.7%, 52.5%, and 55.3% for single-wavelength D2NNs at the same network scale. The four-wavelength D2NN for four-task classification always achieves higher accuracy than the single-wavelength D2NN at different network scales. As the number of tasks increases from 2 to 4, the multi-wavelength D2NN shows more advantages in implementing optical multi-task learning.

Further, the performance of the four-wavelength D2NN performing four tasks at different network scales, respectively, was evaluated and compared with the individual training performance of the four single-wavelength D2NN. Referring to fig. 6,a, the number of diffractive optical elements per diffraction layer is 200 × 200 by increasing the number of layers of the diffraction layer from 1 to 8, thereby increasing the network scale in part. Part b of fig. 6 increases the network size by increasing the number of diffractive optical elements per diffraction layer of the same number 5. Increasing the neural network size of multi-wavelength D2NN for optical multi-task learning can significantly improve its reasoning ability until the performance reaches a saturation state. The performance of the four-wavelength D2NN is continuously improved with the increase of the network scale, and is close to the performance of training four single-wavelength D2NN. Task I and task IV were 96.5%, 85.6%, 88.6%, and 93.8%, respectively, the number of diffraction layers was 5, and the number of diffractive optical elements per diffraction layer was 800 × 800, and in such a configuration, the four-wavelength D2NN exhibited comparable performance to four single-wavelength D2NN of the same network scale. The results show that our proposed method is effective for multitask learning of D2NN and enables lower hardware complexity. Encoding the input of the multi-classification tasks into multi-wavelength can effectively relieve competition among different classification tasks, and can relieve performance reduction of each classification task to the minimum.

The multi-wavelength D2NN in this embodiment is an all-optical computing processor, and can simultaneously execute a plurality of classification tasks with extremely low computation delay. The total calculated time delay for each instance of the multi-tasking input is the sum of the wavefront propagation time from the input plane to the photodetector plane and the response time of the photodetector, independent of the number of wavelength channels and the number of diffractive optical elements per layer. Taking the four-wavelength D2NN with five diffractive layers and 800 × 800 diffractive optical elements per layer in fig. 6 as an example, assuming a 30GHz detection rate, the total calculation time for each instance of the multi-sort task input is 1.23ns, and approximately 3.24 hundred million times diffractive optical calculation operations are performed. In addition, in comparison with spatial multiplexing of a plurality of D2NNs, optical signals of different wavelengths are independent of each other in multi-wavelength diffraction light calculation, and there is no crosstalk. Thus, increasing the number of wavelengths and diffractive optical elements in a D2NN system may increase computational throughput and simplify more sorting tasks.

For each new classification task, new wavelengths can be easily added to implement the new classification task. The classification task expansion process is shown as part a in fig. 6, and the cost of the whole process is extremely low.

In the above, the highly parallel nature of the three-wavelength D2NN was verified by parallel classification of three different inputs based on the MNIST dataset, where the precision of each wavelength is equivalent to training three single-wavelength D2NN for sequential inputs. To perform multiple classification tasks in parallel, inputs from different data sets are encoded to different wavelengths. Two task and four task classifications are performed using dual-wavelength and four-wavelength D2NN based on MNIST, FMNIST, KMNIST, and EMNIST datasets, respectively. With the increase of the number of tasks, the multi-wavelength D2NN achieves higher classification accuracy than the single-wavelength D2NN, and maintains the classification accuracy of the model of each task at a larger network scale, which is enough to show the great advantage of the multi-wavelength D2NN in achieving optical multi-task learning.

Note that the classification accuracy in fig. 4 is illustrated by dots for single-task multiple wavelengths, boxes for single-task multiple wavelengths (with wavelength selective filters), and triangles for single-task multiple wavelengths (without wavelength selective filters).

The classification accuracy in fig. 6 indicates the single-task single-wavelength by dots, the multi-task multi-wavelength by boxes, and the multi-task single-wavelength by triangles.

In this way, by encoding the multi-classification task into multiple wavelengths to utilize the wavelength dimension of the diffracted light field, different tasks can be realized in parallel at the speed of light by the optical multi-task learning method. The optical multitasking function is implemented in D2NN without mechanical movement of the diffraction layer, thereby significantly reducing the complexity of the system. Analysis shows that the method can obviously relieve competition among multiple tasks and maintain the performance of each classification task. As the number of classification tasks increases, the multi-wavelength D2NN shows greater advantages in implementing optical multitask learning. By performing optical multitask learning using wavelength division multiplexing techniques, the method can be extended to other photonic neural network architectures while achieving high parallelism, high precision, and high versatility.

The following describes a processing method provided by the present invention and applied to the multi-wavelength parallel based multitask diffraction neural network device provided in any of the above embodiments, and the method and the multi-wavelength parallel based multitask diffraction neural network device described above can be referred to each other correspondingly.

The present embodiment provides a processing method applied to the multi-wavelength parallel based multitask diffraction neural network device provided as any of the above embodiments, as shown in fig. 7, including:

step 701, modulating the input of the N tasks to N wavelengths by the input unit, forming a mixed light beam by superposing light fields, inputting the mixed light beam to the diffraction modulation structure, and detecting the light intensity of an output plane of the diffraction modulation structure by the light detection assembly;

step 702, the processing unit determines inference results of N tasks corresponding to N wavelengths according to the light intensity of each sub-region in each detection region of the output plane.

In an exemplary embodiment, the processing unit determines inference results of N tasks according to light intensities of the sub-regions in the detection regions of the output plane, including:

and for each task, selecting the sub-region with the maximum light intensity from the sub-regions corresponding to the wavelengths corresponding to the input of the task in the M detection regions, and taking the class of the detection region in which the sub-region with the maximum light intensity is positioned as an inference result of the task.

In an exemplary embodiment, the phase modulation factor of each of the diffractive optical elements is obtained by:

determining a first loss function based on an error between a detected value and a real value of the light intensity of the sub-region corresponding to the wavelength corresponding to the input of the task;

determining a second loss function based on the sum of the light intensities outside each sub-region corresponding to the wavelength corresponding to the input of the task;

determining a target loss function based on the first and second loss functions;

determining a phase modulation factor for each of the diffractive optical elements based on the target loss function.

In an exemplary embodiment, the task is a classification task of images.

In an exemplary embodiment, the N images are from different data sets.

The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.

Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. With this understanding in mind, the above-described technical solutions may be embodied in the form of a software product, which can be stored in a computer-readable storage medium such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in the embodiments or some parts of the embodiments.

Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.

Claims (10)

1. A multi-wavelength parallel based multitask diffractive neural network device, comprising: the system comprises an input unit, a diffraction modulation structure, a light detection assembly and a processing unit; wherein the diffractive modulation structure comprises a plurality of diffractive layers, each of the diffractive layers comprising a plurality of diffractive optical elements;

the input unit is used for modulating the input of the N tasks to N wavelengths, and inputting the N wavelengths into the diffraction modulation structure after forming a mixed light beam through light field superposition, wherein the N wavelengths correspond to the input of the N tasks one by one;

the diffraction modulation structure is used for parallelly processing and outputting each wavelength component of the input mixed light beam;

the light detection component is used for detecting the light intensity of the output plane of the diffraction modulation structure; the output plane comprises M categories of detection regions, each detection region comprises N sub-regions, and the N sub-regions correspond to the N wavelengths one by one; wherein M and N are both positive integers;

the processing unit is used for determining inference results of N tasks corresponding to N wavelengths according to the light intensity distribution of each sub-region in each detection region.

2. The multi-wavelength parallel based multitask diffractive neural network device according to claim 1, wherein said processing unit is specifically configured to:

and for each task, selecting the sub-region with the maximum light intensity from the sub-regions corresponding to the wavelengths corresponding to the input of the task in the M detection regions, and taking the class of the detection region in which the sub-region with the maximum light intensity is positioned as an inference result of the task.

3. The multi-wavelength parallel based multitask diffractive neural network device according to claim 1, wherein a phase modulation factor of each of said diffractive optical elements is obtained by:

determining a first loss function based on an error between a detected value and a real value of the light intensity of the sub-region corresponding to the wavelength corresponding to the input of the task;

determining a second loss function based on the sum of the light intensities outside each sub-region corresponding to the wavelength corresponding to the input of the task;

determining a target loss function based on the first loss function and the second loss function;

determining a phase modulation factor for each of the diffractive optical elements based on the target loss function.

4. The multi-wavelength parallel based multitasking diffractive neural network device according to any one of claims 1 through 3, wherein said N tasks are from different data sets.

5. The multi-wavelength parallel based multitask diffractive neural network device according to any one of claims 1-3, wherein said light detecting component comprises a light detector for detecting light intensity throughout said output plane; alternatively, the light detection assembly comprises a light detector corresponding to each of the sub-regions.

6. The multi-wavelength parallel based multitask diffraction neural network device according to any one of claims 1 through 3, wherein said input unit includes N-1 beam splitters arranged in order in an optical path direction; when the beam splitter is the first beam splitter in the optical path direction, the input of the beam splitter comprises the input of the task corresponding to two wavelengths; when the beam splitter is not the first beam splitter in the direction of the optical path, the inputs of the beam splitter comprise the output of the previous beam splitter and the input of the task corresponding to one wavelength; when the beam splitter is the last beam splitter in the optical path direction, the output of the beam splitter is the mixed beam.

7. A processing method applied to the multi-wavelength parallel based multitask diffraction neural network device according to any one of claims 1 to 6, characterized by comprising:

after the input unit modulates the input of N tasks to N wavelengths and forms mixed light beams through light field superposition to be input to the diffraction modulation structure, the light detection assembly detects the light intensity of an output plane of the diffraction modulation structure;

and the processing unit determines inference results of N tasks corresponding to N wavelengths according to the light intensity of each sub-region in each detection region of the output plane.

8. The processing method according to claim 7, wherein the processing unit determines inference results of the N tasks according to light intensities of the sub-regions in the detection regions of the output plane, including:

and for each task, selecting the sub-region with the maximum light intensity from the sub-regions corresponding to the wavelengths corresponding to the input of the task in the M detection regions, and taking the class of the detection region in which the sub-region with the maximum light intensity is positioned as an inference result of the task.

9. The processing method according to claim 7, wherein the phase modulation factor of each of the diffractive optical elements is obtained by:

determining a first loss function based on an error between a detected value and a real value of the light intensity of the sub-region corresponding to the wavelength corresponding to the input of the task;

determining a second loss function based on the sum of the light intensities outside each sub-region corresponding to the wavelength corresponding to the input of the task;

determining a target loss function based on the first loss function and the second loss function;

determining a phase modulation factor for each of the diffractive optical elements based on the target loss function.

10. The processing method according to claim 7, characterized in that said task is a task of classification of images.

Images