CN116739065A - Photon tensor convolution calculation method and system for multichannel data processing - Google Patents
Photon tensor convolution calculation method and system for multichannel data processing Download PDFInfo
- Publication number
- CN116739065A CN116739065A CN202310789577.4A CN202310789577A CN116739065A CN 116739065 A CN116739065 A CN 116739065A CN 202310789577 A CN202310789577 A CN 202310789577A CN 116739065 A CN116739065 A CN 116739065A
- Authority
- CN
- China
- Prior art keywords
- tensor
- wavelength
- convolution
- signal
- photon
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012545 processing Methods 0.000 title claims abstract description 41
- 238000004364 calculation method Methods 0.000 title claims abstract description 40
- 230000003287 optical effect Effects 0.000 claims abstract description 58
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 33
- 239000013598 vector Substances 0.000 claims abstract description 13
- 230000007274 generation of a signal involved in cell-cell signaling Effects 0.000 claims abstract description 4
- 239000004065 semiconductor Substances 0.000 claims description 17
- 239000006185 dispersion Substances 0.000 claims description 14
- 238000000034 method Methods 0.000 claims description 12
- 239000011159 matrix material Substances 0.000 claims description 11
- 238000003491 array Methods 0.000 claims description 10
- 238000001514 detection method Methods 0.000 claims description 8
- 230000008569 process Effects 0.000 claims description 7
- 238000009825 accumulation Methods 0.000 claims description 6
- 238000013461 design Methods 0.000 claims description 5
- 229910052581 Si3N4 Inorganic materials 0.000 claims description 4
- 238000006243 chemical reaction Methods 0.000 claims description 4
- 230000008878 coupling Effects 0.000 claims description 4
- 238000010168 coupling process Methods 0.000 claims description 4
- 238000005859 coupling reaction Methods 0.000 claims description 4
- 239000000835 fiber Substances 0.000 claims description 4
- 239000004038 photonic crystal Substances 0.000 claims description 4
- 229910052710 silicon Inorganic materials 0.000 claims description 4
- 239000010703 silicon Substances 0.000 claims description 4
- HQVNEWCFYHHQES-UHFFFAOYSA-N silicon nitride Chemical compound N12[Si]34N5[Si]62N3[Si]51N64 HQVNEWCFYHHQES-UHFFFAOYSA-N 0.000 claims description 4
- 210000001520 comb Anatomy 0.000 claims description 3
- 239000004973 liquid crystal related substance Substances 0.000 claims description 3
- 230000003321 amplification Effects 0.000 claims description 2
- 238000001914 filtration Methods 0.000 claims description 2
- 230000031700 light absorption Effects 0.000 claims description 2
- 238000003199 nucleic acid amplification method Methods 0.000 claims description 2
- 239000002245 particle Substances 0.000 claims description 2
- 230000001276 controlling effect Effects 0.000 claims 3
- 230000001105 regulatory effect Effects 0.000 claims 2
- 238000005516 engineering process Methods 0.000 abstract description 11
- 230000010354 integration Effects 0.000 abstract description 3
- 238000010586 diagram Methods 0.000 description 11
- 238000013527 convolutional neural network Methods 0.000 description 6
- 238000011773 genetically engineered mouse model Methods 0.000 description 6
- 238000002591 computed tomography Methods 0.000 description 5
- 230000000694 effects Effects 0.000 description 5
- 238000005265 energy consumption Methods 0.000 description 5
- 230000008901 benefit Effects 0.000 description 4
- 238000000605 extraction Methods 0.000 description 4
- 238000000926 separation method Methods 0.000 description 3
- 238000013473 artificial intelligence Methods 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 238000005070 sampling Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000002059 diagnostic imaging Methods 0.000 description 1
- 239000002360 explosive Substances 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000012067 mathematical method Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000010223 real-time analysis Methods 0.000 description 1
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The invention discloses a photon tensor convolution calculation method and a photon tensor convolution calculation system for multichannel data processing. The system comprises a multi-wavelength light source, a high-speed modulator, a signal generation unit, a weighting unit, a time delay unit, a balanced photoelectric detector, a signal processing unit and a multi-channel tensor reconstruction algorithm of an input signal. According to the invention, elements of different channels in a high-order tensor are rearranged into one-dimensional vectors by improving a tensor data vectorization rule, and based on a technology of time, space and frequency three-dimensional multiplexing, convolution calculation of single optical link and arbitrary multi-channel tensor data can be realized only by one modulator and one set of signal generating unit, and sliding convolution of multi-channel data can be directly completed in one calculation, more physical hardware resources can be released to execute multi-tensor parallel operation, so that more concise and efficient photon tensor convolution operation is completed, and the integration and expandability of the scheme are greatly increased.
Description
Technical Field
The invention belongs to the fields of photon calculation technology, multidimensional image processing technology, convolutional neural network technology and the like, and particularly relates to an artificial intelligence-oriented photon tensor convolutional calculation method and system for multichannel data processing.
Background
Over the past decades, the tremendous growth in Artificial Intelligence (AI) and internet of things (IoT) has driven an increasing demand for High Performance Computing (HPC) and real-time analysis of mass data. Tensor convolution calculation is an effective mathematical method for extracting inherent structural characteristics of multidimensional data, and plays a fundamental role in various fields such as automatic driving, computer vision, biomedicine, machine learning and the like. Mainstream electronic processors convert multi-channel Tensor convolution into generic matrix multiplication (GeMM) to improve throughput and performance computation parallelism (i.e., channel separation convolution), such as the Tensor Core of the inflight am apere architecture. With the rise of AI and the relaxation of moore's law, it is increasingly difficult to meet the computational power required for explosive data flows using traditional electronic processors based on von neumann architecture. Photon computing technology has proven to be a promising candidate for next-generation neuromorphic computing in recent years due to its inherent parallelism, ultra-high bandwidth, low latency, and low energy consumption, and is expected to solve the problems of increased computational power and power consumption due to memory/computation separation and data dimension conversion in traditional electronic computing architectures. The prior photon convolution technology converts standard tensor convolution operations (SC, standard convolution) into general matrix multiplication operations (GEMM, general Matrix to Matrix Multiplication) according to the experience of electronic computation when processing multi-channel tensors (see [ Feldmann, j., youngbloom, n., karpev, m.et al.parallel convolutional processing using an integrated photonic tensor core.nature 589,52-58 (2021) ].
However, when the GEMM algorithm is used in a photonic convolution computing architecture, each data patch requires a wavelength and a set of high-speed devices (including electro-optic modulators, radio frequency amplifiers, and signal generators). Even after optical delay line patching, each convolution channel of the tensor requires a wavelength and a set of high-speed devices, which are expensive and not easily integrated. Therefore, when tensor data with large depth are processed, the disadvantages of the photon computing system using the GEMM algorithm gradually appear, and the advantage of low energy consumption of optical computing can not be reflected. To this end, we propose a method and system for photon tensor convolution computation for arbitrary multi-channel data processing.
Disclosure of Invention
The technical problems to be solved by the invention are as follows: the complex optical architecture of the existing photon convolution technology in the process of multi-channel tensor data is overcome, and each convolution channel and even each patch need a wavelength and a set of high-speed devices (comprising an electro-optical modulator, a radio frequency amplifier and a signal generator). According to the invention, elements of different channels in a high-order tensor are rearranged into one-dimensional vectors by improving tensor data vectorization rules, and based on a technology of three-dimensional multiplexing of time, space and frequency, convolution calculation of single optical link and arbitrary multi-channel tensor data can be realized by only one modulator and one set of signal generating unit, and sliding convolution of multi-channel data can be directly completed in one calculation without clock synchronization among a plurality of channels. Therefore, more physical hardware resources can be released to execute multi-tensor parallel operation instead of parallel operation among channels, and more concise and efficient photon tensor convolution operation is completed. The integration and scalability of the solution is greatly increased due to the reduced need for high-speed equipment.
In order to achieve the above purpose, the present invention provides the following technical solutions:
a photon tensor convolution computing system for multi-channel data processing, comprising a multi-wavelength light source (an optical frequency comb or a multi-wavelength laser array), a high-speed modulator, a signal generating unit, a weighting unit, a delay unit (a dispersion delay or a true delay line), a balanced photodetector, and a signal processing unit, and a multi-channel tensor reconstruction algorithm of an input signal:
the multi-wavelength light source may generally be selected from an optical kerr frequency comb, a semiconductor mode-locked laser, or a semiconductor multi-wavelength laser array. For generating a stable multi-wavelength optical signal, the information of the tensor convolution kernel and the tensor data information to be processed need to be loaded on the frequency domain and the time domain respectively. The larger the number of wavelengths, the larger the single calculation amount;
the high-speed modulator is used for modulating the multi-wavelength signals simultaneously, and a wide-band Mach-Zehnder modulator (MZM) is generally used for loading high-speed waveforms generated by tensor data onto each wavelength simultaneously, so that the loading of time domain information is completed;
the signal generation unit is used for converting the original three-dimensional tensor data to be convolved into one-dimensional vector data according to a proposed algorithm, so that the three-dimensional tensor data can be conveniently loaded on an optical signal through a high-speed radio frequency source, and the calculation on an optical system is carried out;
the weighting unit is used for loading convolution kernel information on the frequency domain of each wavelength and adjusting the intensity of the output optical signal of each wavelength;
the delay unit is used for generating a bit time delay between the time domain waveforms of each wavelength;
the balanced photoelectric detector is used for loading negative number weights. And loading the weight value of the convolution kernel onto the corresponding wavelength, wherein the weight value corresponds to the filter power, the positive and negative of the weight value corresponds to the corresponding port output by the wavelength selection switch, the wavelength corresponding to the positive value is connected to the positive port of the balance detector (BPD), and the wavelength corresponding to the negative value is connected to the negative port of the BPD. Finally, the multi-wavelength modulated optical signal with the product accumulation operation is converted into an electric signal, and the electric signal is sent to a signal processing unit;
the signal processing unit is used for acquiring an electric signal output by the system after O/E conversion, and reconstructing the acquired analog signal into a convolution characteristic image of the original picture through weighted average filtering;
the multi-channel tensor reconstruction algorithm is used for reconstructing the original input multi-channel tensor into a one-dimensional vector array suitable for photon convolution operation according to the improved im2col algorithm.
Further, the multi-wavelength light source used in the system can generally use Kerr optical frequency comb, semiconductor mode-locked laser, semiconductor laser array, etc. The first two kinds of frequency combs are used, so that a single-chip light source can obtain more wavelength numbers, but a wavelength selective switch is used at the rear end of the system to independently regulate and control each wavelength. With the use of semiconductor laser arrays, since the intensity value of each wavelength is individually adjustable, no wavelength selective switch is required,
further, the delay value (τ) of the multi-wavelength light source used in the system needs to be matched with the modulation rate (B) of the system, that is, the dispersion value is one bit length (τ=1/B) of the time domain modulation waveform. In order to produce the required delay value, the wavelength interval (Δλ) of the multi-wavelength light source needs to be matched to the dispersive medium parameters (dispersion coefficient D and length L) or the true delay line length, i.e., τ=dlΔλ.
Furthermore, the MZ modulator in the system is designed with wide optical bandwidth, so that more wavelength modulation can be performed simultaneously, larger calculated amount is completed, and the integrity of the optical signal is ensured. Meanwhile, the design and selection of low driving voltage reduce the use of a radio frequency amplifier, reduce the power consumption of a system and ensure the integrity of low-frequency electric signals.
Further, the weighting unit in the system can control the intensity and the route of multiple wavelengths through the liquid crystal Lcos panel by using the wavelength selective switch for a discrete system. For an on-chip integrated system, the resonant wavelength of each micro-ring resonator can be adjusted by using a micro-ring resonator array through thermo-optical modulation, and the customized intensity output of each wavelength is completed, which corresponds to the loading of a convolution kernel.
Furthermore, the delay unit in the system is generally loaded by adopting a dispersion delay device or a true delay line, wherein the delay unit is generally realized by a single-mode fiber with a certain dispersion coefficient, a chirped FBG grating and a photonic crystal layer device. The latter is typically achieved by on-chip silicon or low loss silicon nitride delay lines.
Further, the multi-channel tensor reconstruction algorithm includes the following steps:
s1, slicing the original tensor according to depth, and converting the original tensor into d in A two-dimensional m x m matrix.
S2, referring to the size of tensor convolution kernel (N multiplied by N), using the size of N multiplied by N two-dimensional convolution kernel, d in On individual channelsThe corresponding patching is sequentially unfolded into a one-dimensional array according to an im2col algorithm. The arrays obtained on the same patch and different channels are connected into a longer one-dimensional array.
S3, according to stride size, generating a one-dimensional array which is sequentially connected among different channels by patching of each corresponding convolution kernel in the input tensor.
S4, connecting one-dimensional arrays generated at the patch positions corresponding to each convolution kernel sliding in the input tensor sequentially in an end-to-end mode according to the front-to-end sequence of the convolution kernels sliding to form a final one-dimensional array to be convolved on light, and broadcasting the final one-dimensional array to each wavelength as a high-speed time domain modulation signal.
Further, depth d of tensor to be processed in But may be of any size, the tensor reconstruction algorithm is still valid. I.e., tensor standard convolution of arbitrary depth can be achieved through a single device link.
Further, in the reconstruction process of the tensor to be processed, column priority or row priority does not affect the calculation result, efficiency and storage mode.
A photon tensor convolution computing method for multi-channel data processing, comprising the steps of:
s1, firstly, reconstructing the original input multi-channel tensor into a one-dimensional vector array suitable for photon convolution operation according to a proposed multi-channel tensor reconstruction algorithm.
S2, generating optical signals by the integrated multi-wavelength light source, and then adding N 3 The individual wavelength signals are directly fed into a broadband MZ modulator by means of end-face coupling.
S2, MZ modulator receives N 3 After the high-speed signals generated by the signal generator are amplified by the radio frequency amplifier, the wavelengths are uniformly modulated by the MZ modulator to obtain N 3 And sending the optical signals with the same high-speed time domain waveforms to a weighting unit.
S4, the weighting unit receives N 3 And carrying out power regulation and control on each wavelength according to the absolute value of the tensor convolution kernel by the optical signals with the time domains modulated at high speed, so that the optical signals meet the corresponding value on the tensor convolution kernel. Through addingThe wavelength after the weight is sent to the time delay unit.
S5, the time delay unit receives N after time domain modulation and frequency domain weighting 3 Each wavelength of the dispersive unit generates a fixed time delay between the time domains of each wavelength, then N 3 The wavelengths are divided into two paths according to the signs of the values on the convolution kernel and are respectively sent into two detection ports of the balanced photoelectric detector.
S6, after two detection ports of the balanced photoelectric detector respectively receive two paths of signals, the optical signals are converted into electric signals. The optical signals of the same path are subjected to intensity accumulation in the photoelectric detector, and the accumulated result is subjected to differential processing by the balance detector to obtain a final electric output signal, namely a characteristic signal obtained after tensor convolution operation of the tensor signal to be convolved.
Further, generating a multi-wavelength signal by an integrated kerr frequency comb, a semiconductor mode-locked laser, or an integrated semiconductor laser array can provide more wavelengths for a customized tensor convolution kernel.
Further, the MZ modulator is designed to have as large an optical bandwidth as possible for modulating more wavelengths simultaneously, so as to ensure the calculation amount and the integration performance.
Furthermore, since a general rf amplifier is designed to protect a high-frequency circuit, a low-frequency design is designed to pass a high-frequency impedance, i.e., a signal of about 0kHz to 100kHz is generally filtered out. Therefore, when we modulate complex tensor signals, waveform distortion often occurs. We therefore add a high frequency carrier to the signal or use a low drive voltage modulator to ensure signal integrity.
Furthermore, the weighting unit uses a micro-ring resonator array or a semiconductor optical amplifier array, the former realizes the loading of the weight by controlling the resonance state of the micro-ring resonator through heat adjustment, and the latter realizes the loading of the weight by controlling the amplification and absorption of light through the regulation of the particle number inversion state through a semiconductor PN junction.
Further, the delay unit is loaded by a dispersive delay device or a true delay line, the former is generally implemented by a single mode fiber with a certain dispersion coefficient, a chirped FBG grating and a photonic crystal layer device. The latter is typically implemented by on-chip silicon or silicon nitride delay lines.
The invention has the beneficial effects that:
1) The invention realizes the standard tensor calculation of the multidimensional data based on the characteristics of high bandwidth, low energy consumption, parallelism and the like of photons and the wavelength, time and space dimension of multiplexed light, thereby effectively avoiding the problems of increased calculation complexity and power consumption caused by storage/calculation separation and multidimensional data conversion in electronic calculation.
2) The invention rearranges the elements of different channels in the higher-order tensor into one-dimensional vectors based on a reconstruction algorithm of the multi-channel tensor, and can realize the convolution calculation of single optical link and any multi-channel tensor data at the same time by only needing one modulator and one set of signal generating unit based on the technology of three-dimensional multiplexing of time, space and frequency, and the sliding convolution of the multi-channel data can be directly completed in one calculation. The high-speed photoelectric function device is not required to be additionally arranged, so that the system is simplified, the stability of the system is improved, and the expandability of the scheme can be greatly improved.
3) The invention is based on a mode of realizing multi-channel tensor convolution by a single link, and clock synchronization among a plurality of convolution channels is not needed. Therefore, more physical hardware resources can be released to execute multi-tensor parallel operation instead of parallel operation among single tensor channels, and more concise and efficient photon tensor convolution operation is completed.
The features and advantages of the present invention will be described in detail by way of example with reference to the accompanying drawings.
Drawings
FIG. 1 is a schematic diagram of a multi-channel tensor reconstruction algorithm according to the present invention;
FIG. 2 is a schematic diagram showing the steps of a multi-channel tensor reconstruction algorithm according to the present invention;
FIG. 3 is a schematic diagram of a multi-wavelength time domain sequence before tensor weighting time delay according to the present invention;
FIG. 4 is a schematic diagram of a multi-wavelength time domain sequence with tensor weighted delay according to the present invention;
FIG. 5 is a schematic diagram of a tensor convolution process and its corresponding time-domain misalignment accumulation according to the present invention;
FIG. 6 is a schematic diagram of a discrete system of photon tensor convolution based on multi-channel data processing in accordance with the present invention;
FIG. 7 is a schematic diagram of an integrated system on chip based on photon tensor convolution for multi-channel data processing in accordance with the present invention;
FIG. 8 is a schematic diagram of an embodiment of a 64-channel CT scan tensor data convolution process using a multi-layer convolutional neural network;
FIG. 9 is a data diagram illustrating an embodiment of a 64-channel CT scan tensor data convolution process using a multi-layer convolutional neural network;
FIG. 10 is a schematic diagram showing the effect of using a multi-layer convolutional neural network to perform 64-channel CT scan tensor data feature extraction in an embodiment;
FIG. 11 is a schematic diagram showing the effect of using a multi-layer convolutional neural network to perform two classifications of 64-channel CT scan tensor data;
Detailed Description
The invention provides a photon tensor convolution calculation method and a photon tensor convolution calculation system for multichannel data processing, which solve the problems of overhigh energy consumption, poor expansibility and the like of a system caused by dependence on a large number of high-speed radio frequency devices (a high-speed modulator, a high-speed signal generator and a high-bandwidth radio frequency amplifier) in a traditional photon tensor processing system. In the proposed photon tensor convolution (PTPU) scheme, as a standard convolution calculation, a multi-channel tensor convolution can be directly implemented, with the tensor check being utilized to verify a joint mapping of feature space dimensions and channel dimensions. Only one modulator is required for loading any multi-channel tensor data to be processed.
The invention firstly provides a photon tensor convolution computing system for multichannel data processing, which consists of a multi-wavelength light source, a high-speed modulator, a signal generating unit, a weighting unit, a time delay unit, a balance photoelectric detector and a signal processing unit. The method is a multi-channel tensor reconstruction algorithm;
firstly, according to a proposed multi-channel tensor reconstruction algorithm, an original input multi-channel tensor is reconstructed into a one-dimensional vector array suitable for photon convolution operation.
The integrated multi-wavelength light source then generates an optical signal, which is then used to convert N 3 The individual wavelength signals are directly fed into a broadband MZ modulator by means of end-face coupling. MZ modulator receives N 3 After the high-speed signals generated by the signal generator are amplified by the radio frequency amplifier, the wavelengths are uniformly modulated by the MZ modulator to obtain N 3 And sending the optical signals with the same high-speed time domain waveforms to a weighting unit.
The weighting unit receives N 3 And carrying out power regulation and control on each wavelength according to the absolute value of the tensor convolution kernel by the optical signals with the time domains modulated at high speed, so that the optical signals meet the corresponding value on the tensor convolution kernel. The weighted wavelengths are fed into a delay unit.
The time delay unit receives N after time domain modulation and frequency domain weighting 3 Each wavelength of the dispersive unit generates a fixed time delay between the time domains of each wavelength, then N 3 The wavelengths are divided into two paths according to the signs of the values on the convolution kernel and are respectively sent into two detection ports of the balanced photoelectric detector.
And finally, after two detection ports of the balanced photoelectric detector respectively receive two paths of signals, converting the optical signals into electric signals. The optical signals of the same path are accumulated in intensity in the photoelectric detector, and the accumulated results are subtracted by the balance detector to obtain a final electric output signal, namely a characteristic signal obtained after tensor convolution operation of the tensor signal to be convolved is completed.
In this embodiment, the multi-wavelength light source used in the system may generally use a kerr optical frequency comb, a semiconductor mode-locked laser, a semiconductor laser array, or the like. The first two kinds of frequency combs are used, so that a single-chip light source can obtain more wavelength numbers, but a wavelength selective switch is used at the rear end of the system to independently regulate and control each wavelength. With the use of semiconductor laser arrays, since the intensity value of each wavelength is individually adjustable, no wavelength selective switch is required,
in this embodiment, the delay value (τ) of the multi-wavelength light source used in the system needs to be matched with the modulation rate (B) of the system, that is, the dispersion value is one bit of the length (τ=1/B) of the time domain modulation waveform. In order to produce the required delay value, the wavelength interval (Δλ) of the multi-wavelength light source needs to be matched to the dispersive medium parameters (dispersion coefficient D and length L) or the true delay line length, i.e., τ=dlΔλ.
In this embodiment, the MZ modulator in the system performs design of wide optical bandwidth, so that more wavelength modulation can be performed simultaneously, so as to complete larger calculation amount and ensure the integrity of the optical signal. Meanwhile, the design and selection of low driving voltage are carried out, the use of a radio frequency amplifier is reduced, the power consumption of the system is reduced, and the integrity of low-frequency electric signals is ensured.
In this embodiment, the weighting unit in the system may use a wavelength selective switch to control the intensity and routing of multiple wavelengths through the liquid crystal Lcos panel for a discrete system. For an on-chip integrated system, the resonant wavelength of each micro-ring resonator can be adjusted by using a micro-ring resonator array through thermal tuning, and the customized intensity output of each wavelength is completed, which corresponds to the loading of a convolution kernel.
In this embodiment, the delay unit in the system is generally loaded by a dispersive delay device or a true delay line by using the delay unit, where the delay unit is generally implemented by a single mode fiber with a certain dispersion coefficient, a chirped FBG grating, and a photonic crystal layer device. The latter is typically achieved by on-chip silicon or low loss silicon nitride delay lines.
In this embodiment, the multi-channel tensor reconstruction algorithm includes the following steps:
s1, slicing the original tensor according to depth, and converting the original tensor into d in A two-dimensional m x m matrix.
S2, referring to the size of tensor convolution kernel (N multiplied by N), using the size of N multiplied by N two-dimensional convolution kernel, d in The corresponding patch on each channel is sequentially expanded into a one-dimensional array according to an im2col algorithm. The arrays obtained on the same patch and different channels are connected into a longer one-dimensional array.
S3, according to stride size, generating a one-dimensional array which is sequentially connected among different channels by patching of each corresponding convolution kernel in the input tensor.
S4, connecting one-dimensional arrays generated at the patch positions corresponding to each convolution kernel sliding in the input tensor sequentially in an end-to-end mode according to the front-to-end sequence of the convolution kernels sliding to form a final one-dimensional array to be convolved on light, and broadcasting the final one-dimensional array to each wavelength as a high-speed time domain modulation signal.
In this embodiment, the depth d of the tensor to be processed in But may be of any size, the tensor reconstruction algorithm is still valid. I.e., tensor standard convolution of arbitrary depth can be achieved through a single device link.
In this embodiment, in the reconstruction process of the tensor to be processed, the column priority or the row priority does not affect the result, efficiency and storage mode of the calculation.
The principle of a multi-channel tensor reconstruction algorithm proposed by the invention is shown in figures 1 and 2,
firstly, slicing an original tensor to be processed according to depth, and converting the original tensor into d in A two-dimensional m x m matrix.
Then, referring to the size of the tensor convolution kernel (N), d is determined by using the size of the N two-dimensional convolution kernel in The corresponding patch on each channel is sequentially expanded into a one-dimensional array according to an im2col algorithm. The arrays obtained on the same patch and different channels are connected into a longer one-dimensional array.
Then, according to stride size, each patch of the corresponding convolution kernel in the input tensor is generated into a one-dimensional array which is connected among different channels in turn.
And finally, connecting the one-dimensional arrays generated at the patch positions corresponding to each convolution kernel sliding in the input tensor sequentially in an end-to-end mode according to the front-to-end sequence of the convolution kernel sliding to form a final one-dimensional array to be convolved on light, and broadcasting the final one-dimensional array to be convolved on light as a high-speed time domain modulation signal to each wavelength.
For standard tensor convolution operations, the formula can be expressed as:
for the proposed multi-channel tensor reconstruction algorithm, its vectorized representation is:
X d，k ＝T d，Mod(k，N)，Fix(k/N) ，k∈(0，N 2 -1) (2)
after passing through the photon convolution system, the product accumulation operation result is obtained as follows:
removing some redundant information, wherein effective data are as follows:
as shown in FIG. 3, the multi-wavelength time domain sequence before tensor weighting time delay has no dislocation between time domain waveforms after all wavelengths are uniformly loaded with tensor vectorized signals, and is completely aligned. Each intensity value in the frequency domain is filtered to a consistent level.
The tensor weighted time-delayed multi-wavelength time-domain sequence is shown in fig. 4, and after all wavelengths pass through the customized time-delay unit, uniform time delay dislocation is generated between each wavelength time domain and is matched with the modulation rate (τ=1/b=dl Δλ). And on the frequency domain, according to the absolute value of the convolution kernel, adjusting the power of each wavelength to a corresponding value to finish the mapping of the tensor convolution kernel. The multi-wavelength signals with frequency domain weighting and time domain time delay respectively finish multiply-accumulate operation of each bit through the photoelectric detector. And as shown, the convolution of multiple channels may be done directly in the operation again. The mark (1) is the convolution result of the RG two channels, and the mark (2) is the convolution result of the GB two channels. As shown in fig. 5, the corresponding value of each bit is shown in detail.
The invention provides a photon tensor convolution calculation method and a photon tensor convolution calculation system for multichannel data processing, wherein a discrete system based on photon tensor convolution of multichannel data processing is shown in fig. 6. The operation steps are as follows:
firstly, according to the multi-channel tensor reconstruction algorithm, the original input multi-channel tensor is reconstructed into a one-dimensional vector array suitable for photon convolution operation.
The integrated multi-wavelength light source then generates an optical signal, which is then used to convert N 3 The individual wavelength signals are directly fed into a broadband MZ modulator by means of end-face coupling. MZ modulator receives N 3 After the high-speed signals generated by the signal generator are amplified by the radio frequency amplifier, the wavelengths are uniformly modulated by the MZ modulator to obtain N 3 And sending the optical signals with the same high-speed time domain waveforms to a weighting unit.
The weighting unit receives N 3 And carrying out power regulation and control on each wavelength according to the absolute value of the tensor convolution kernel by the optical signals with the time domains modulated at high speed, so that the optical signals meet the corresponding value on the tensor convolution kernel. The weighted wavelengths are fed into a delay unit.
The time delay unit receives N after time domain modulation and frequency domain weighting 3 Each wavelength of the dispersive unit generates a fixed time delay between the time domains of each wavelength, then N 3 The wavelengths are divided into two paths according to the signs of the values on the convolution kernel and are respectively sent into two detection ports of the balanced photoelectric detector.
And finally, after two detection ports of the balanced photoelectric detector respectively receive two paths of signals, converting the optical signals into electric signals. The optical signals of the same path are accumulated in intensity in the photoelectric detector, and the accumulated results are subtracted by the balance detector to obtain a final electric output signal, namely a characteristic signal obtained after tensor convolution operation of the tensor signal to be convolved is completed.
The invention provides a photon tensor convolution calculation method and a photon tensor convolution calculation system for multichannel data processing, wherein an integrated system on a chip based on photon tensor convolution of multichannel data processing is shown in fig. 7. Unlike the former, the weighted delay unit is implemented by a micro-ring array and a true delay line, respectively. And the resonant state of the micro-ring resonator is controlled through heat regulation to realize the modulation of wavelength through power, so that the loading of required weight is completed.
As shown in fig. 8, a multi-layer convolutional neural network for performing multi-channel tensor convolution was established for verifying our proposed photon tensor convolution calculation method and system. Because the used tensor is a CT scanning image of 64 channels, the tensor is vectorized and then input into a photon convolution system, the data obtained by the first layer convolution is shown in fig. 9, the enlarged view of the data shows 63 different sampling points, and the data respectively correspond to 63 convolution characteristic channels obtained by directly carrying out 2 x 2 convolution kernels on the tensor of 64 channels, so that the greatest advantage of the scheme is also presented, namely the direct convolution of any multi-channel tensor is realized through a single optical link.
As shown in fig. 10, the feature extraction effect graph of the photon tensor convolution system is recovered by the sampling points in the above data (only the first thirty features are demonstrated). The feature extraction effect shows complete lower edge information, and the extraction effect is good.
In summary, the invention discloses a photon tensor convolution calculation method and a photon tensor convolution calculation system suitable for any multi-channel high-dimensional data processing, which are suitable for application scenes such as machine vision, medical imaging, automatic driving and the like, and all deep learning networks comprising tensor convolution operation. The technical problems to be solved by the invention are as follows: the prior photon convolution technology converts standard tensor convolution operation (SC, standard convolution) into general matrix multiplication operation (GEMM, general Matrix to Matrix Multiplication) according to the experience of electronic calculation and based on the idea of depth separable convolution when processing multi-channel tensors. However, when the GEMM algorithm is used in a photonic convolution computing architecture, each data patch requires a wavelength and a set of high-speed devices (including electro-optic modulators, radio frequency amplifiers, and signal generators). Even after optical delay line patching, each convolution channel of the tensor requires a wavelength and a set of high-speed devices, which are expensive and not easily integrated. Therefore, when tensor data with large depth are processed, the disadvantages of the photon computing system using the GEMM algorithm gradually appear, and the advantage of low energy consumption of optical computing can not be reflected. According to the invention, elements of different channels in a high-order tensor are rearranged into one-dimensional vectors by improving a tensor data vectorization rule, and standard convolution calculation of single optical link and any multi-channel tensor data can be realized by only one modulator based on a time, space and frequency three-dimensional multiplexing technology. The sliding convolution of multi-channel data can be accomplished directly in one computation without the need for clock synchronization between multiple channels. Therefore, more physical hardware resources can be released to execute multi-tensor parallel operation instead of parallel operation among channels, and photon tensor convolution operation with more conciseness, high efficiency and low power consumption is completed. The system architecture consists of a multi-wavelength light source (an optical frequency comb or a multi-wavelength laser array), a Mach-Zehnder modulator (MZM), a signal generating unit, a weighting unit, a delay unit (a dispersion delay or a true delay line), a balanced photoelectric detector and a signal processing unit.
The foregoing description of the preferred embodiments of the invention is not intended to be limiting, but rather is intended to cover all modifications, equivalents, or alternatives falling within the spirit and principles of the invention.
Claims (9)
1. A photon tensor convolution computing system for multi-channel data processing, characterized by: the multi-channel tensor reconstruction method comprises a multi-wavelength light source, a high-speed modulator, a signal generation unit, a weighting unit, a time delay unit, a balanced photoelectric detector, a signal processing unit and a multi-channel tensor reconstruction algorithm of an input signal:
the multi-wavelength light source is used for generating stable multi-wavelength light signals, and the information of tensor convolution kernels and tensor data information to be processed are loaded on the frequency domain and the time domain of the stable multi-wavelength light signals respectively; the larger the number of wavelengths, the larger the single calculation amount;
the high-speed modulator is used for loading the high-speed waveform generated by tensor data onto each wavelength simultaneously to finish the loading of time domain information;
the signal generation unit is used for converting the original three-dimensional tensor data to be convolved into one-dimensional vector data according to a proposed algorithm so as to load the one-dimensional vector data onto an optical signal through a high-speed radio frequency source and perform calculation on an optical system;
the weighting unit is used for loading convolution kernel information on the frequency domain of each wavelength, and loading the convolution kernel information by adjusting the intensity of the output optical signal of each wavelength;
the delay unit is used for generating a bit time delay between the time domain waveforms of each wavelength;
the balanced photoelectric detector is used for loading negative number weights; loading the weight value of the convolution kernel onto the corresponding wavelength, wherein the weight value corresponds to the filter power, the positive and negative of the weight value corresponds to the corresponding port output by the wavelength selection switch, the wavelength corresponding to the positive value is connected to the positive port of the balance detector, and the wavelength corresponding to the negative value is connected to the negative port of the BPD; finally, the multi-wavelength modulated optical signal with the product accumulation operation is converted into an electric signal, and the electric signal is sent to a signal processing unit;
the signal processing unit is used for acquiring an electric signal output by the system after O/E conversion, and reconstructing the acquired analog signal into a convolution characteristic image of the original picture through weighted average filtering;
the multi-channel tensor reconstruction algorithm is used for reconstructing the original input multi-channel tensor into a one-dimensional vector array suitable for photon convolution operation according to the improved im2col algorithm.
2. A photon tensor convolution computing system for multi-channel data processing as defined in claim 1 wherein:
the multi-wavelength light source used by the system comprises a Kerr optical frequency comb, a semiconductor mode-locking laser and a semiconductor laser array; the first two kinds of frequency combs are used, a single-chip light source is used for obtaining more wavelength numbers, and a wavelength selection switch is used at the rear end of the system for independently regulating and controlling each wavelength; with a semiconductor laser array, no wavelength selective switch is required as the intensity value of each wavelength is individually adjustable.
The delay value tau of the multi-wavelength light source used by the system is matched with the modulation rate B of the system, namely, the dispersion value is the length of one bit of the time domain modulation waveform, and tau=1/B; to produce the desired delay value, the wavelength interval Δλ of the multi-wavelength light source is matched to the dispersive medium parameter or true delay line length, i.e., τ=dl Δλ.
3. A photon tensor convolution computing system for multi-channel data processing as defined in claim 1 wherein: the high-speed modulator in the system is designed with wide optical bandwidth so as to modulate more wavelengths at the same time, so that larger calculated amount is completed, and the integrity of the optical signal is ensured; meanwhile, the design and selection of low driving voltage reduce the use of a radio frequency amplifier, reduce the power consumption of a system and ensure the integrity of low-frequency electric signals.
4. A photon tensor convolution computing system for multi-channel data processing as defined in claim 1 wherein: the weighting unit in the system adopts a wavelength selection switch for a discrete system, and controls the intensity and the route of multiple wavelengths through the liquid crystal Lcos panel; for an on-chip integrated system, a micro-ring resonator array is adopted, the resonant wavelength of each micro-ring resonator is adjusted through thermo-optical modulation, and customized intensity output of each wavelength is completed, and the customized intensity output corresponds to loading of a convolution kernel.
5. A photon tensor convolution computing system for multi-channel data processing as defined in claim 1 wherein: the time delay unit in the system is loaded through a dispersion time delay device or a true time delay line, wherein the dispersion time delay device or the true time delay line is realized through a single-mode fiber with a preset dispersion coefficient, a chirped FBG grating and a photonic crystal layer device; the latter is realized by on-chip silicon or low-loss silicon nitride delay lines.
6. A photon tensor convolution computing system for multi-channel data processing as defined in claim 1 wherein: the multi-channel tensor reconstruction algorithm comprises the following steps:
s1, slicing the original tensor according to depth, and converting the original tensor into d in A two-dimensional m x m matrix; wherein the depth d of the tensor to be processed in For any size, the tensor reconstruction algorithm is still effective, i.e. tensors of any depth are realized through a single device linkStandard convolution;
s2, referring to the size of tensor convolution kernel (N multiplied by N), using the size of N multiplied by N two-dimensional convolution kernel, d in The corresponding patching on each channel is sequentially unfolded into a one-dimensional array according to an im2col algorithm; the arrays obtained on different channels of the same patch are connected into a longer one-dimensional array; in the reconstruction process of the tensor to be processed, the column priority or the row priority does not influence the calculation result, efficiency and storage mode;
s3, according to stride size, generating a one-dimensional array which is sequentially connected among different channels by patching of each corresponding convolution kernel in the input tensor;
s4, connecting one-dimensional arrays generated at the patch positions corresponding to each convolution kernel sliding in the input tensor sequentially in an end-to-end mode according to the front-to-end sequence of the convolution kernels sliding to form a final one-dimensional array to be convolved on light, and broadcasting the final one-dimensional array to each wavelength as a high-speed time domain modulation signal.
7. A photon tensor convolution calculation method for multi-channel data processing based on the system of any one of claims 1-6, comprising the steps of:
s1, firstly, reconstructing an original input multi-channel tensor into a one-dimensional vector array suitable for photon convolution operation according to a proposed multi-channel tensor reconstruction algorithm;
s2, generating optical signals by the integrated multi-wavelength light source, and then adding N 3 The individual wavelength signals are directly input into a broadband MZ modulator in an end-face coupling mode;
s3, MZ Modulator receives N 3 After the high-speed signals generated by the signal generator are amplified by the radio frequency amplifier, the wavelengths are uniformly modulated by the MZ modulator to obtain N 3 The optical signals with the same high-speed time domain waveforms are sent to a weighting unit;
s4, the weighting unit receives N 3 And carrying out power regulation and control on each wavelength according to the absolute value of the tensor convolution kernel by the optical signals with the time domains modulated at high speed, so that the optical signals meet the corresponding value on the tensor convolution kernel. WeightedThe wavelength is sent into a time delay unit;
s5, the time delay unit receives N after time domain modulation and frequency domain weighting 3 Each wavelength of the dispersive unit generates a fixed time delay between the time domains of each wavelength, then N 3 The wavelengths are divided into two paths according to the signs of the values on the convolution kernel and are respectively sent into two detection ports of the balanced photoelectric detector;
s6, after two detection ports of the balanced photoelectric detector respectively receive two paths of signals, the optical signals are converted into electric signals. The optical signals of the same path are subjected to intensity accumulation in the photoelectric detector, and the accumulated result is subjected to differential processing by the balance detector to obtain a final electric output signal, namely a characteristic signal obtained after tensor convolution operation of the tensor signal to be convolved.
8. A method of photon tensor convolution computation for multi-channel data processing as defined in claim 7 wherein: the signal integrity is ensured by adding a high frequency carrier to the signal or using a low drive voltage modulator.
9. A method of photon tensor convolution computation for multi-channel data processing as defined in claim 7 wherein: the weighting unit uses a micro-ring resonator array or a semiconductor optical amplifier array, the former realizes the loading of the weight by controlling the resonance state of the micro-ring resonator through heat adjustment, and the latter realizes the loading of the weight by controlling the amplification and absorption of light through regulating the particle number inversion state through a semiconductor PN junction.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310789577.4A CN116739065A (en) | 2023-06-30 | 2023-06-30 | Photon tensor convolution calculation method and system for multichannel data processing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310789577.4A CN116739065A (en) | 2023-06-30 | 2023-06-30 | Photon tensor convolution calculation method and system for multichannel data processing |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116739065A true CN116739065A (en) | 2023-09-12 |
Family
ID=87913267
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310789577.4A Pending CN116739065A (en) | 2023-06-30 | 2023-06-30 | Photon tensor convolution calculation method and system for multichannel data processing |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116739065A (en) |
-
2023
- 2023-06-30 CN CN202310789577.4A patent/CN116739065A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114815959B (en) | Photon tensor calculation acceleration method and device based on wavelength division multiplexing | |
| CA3101026A1 (en) | Optoelectronic computing systems | |
| CN114819132B (en) | Photon two-dimensional convolution acceleration method and system based on time-wavelength interleaving | |
| Tan et al. | Photonic signal processor based on a Kerr microcomb for real-time video image processing | |
| CN115169542B (en) | Two-dimensional photon convolution acceleration system and device for convolution neural network | |
| CN115222035B (en) | Photon neural network convolution acceleration chip | |
| CN115167610B (en) | Two-dimensional photon convolution acceleration chip and application system and method | |
| CN112001487A (en) | Photon neural network | |
| CN117077750B (en) | Parallel photon convolution operation chip and system | |
| US11934943B1 (en) | Two-dimensional photonic neural network convolutional acceleration chip based on series connection structure | |
| CN113325917A (en) | Light computing device, system and computing method | |
| CN116739065A (en) | Photon tensor convolution calculation method and system for multichannel data processing | |
| CN115130666B (en) | Two-dimensional photon convolution acceleration method and system | |
| Shinya et al. | High-Speed Optical Convolutional Neural Network Accelerator with 100 Gbaud EO-polymer/Si Hybrid Optical Modulator | |
| Zhang et al. | Time-stretch optical neural network with time-division multiplexing | |
| CN116484931B (en) | Photon matrix multiplication operation device and operation method for neural network | |
| CN117709423A (en) | Deep neural network photon acceleration chip and operation system thereof | |
| CN117436486A (en) | Optical convolution neural network based on thin film lithium niobate and silicon mixture | |
| Moss | Ultrahigh bandwidth applications of optical microcombs | |
| US20230281437A1 (en) | Radio-Frequency Photonic Architecture for Deep Neural Networks, Signal Processing, and Computing | |
| Nakajima | Parallel neuromorphic computing on space and wavelength division multiplexed photonic processor | |
| Tan et al. | Photonic real time video image signal processor at 17Tb/s based on a Kerr microcomb | |
| Tan et al. | Ultrahigh bandwidth applications of optical microcombs | |
| Moss | Applications of Optical Frequency Microcombs | |
| CN117784313A (en) | Two-dimensional photon convolution operation chip and system based on cyclic array waveguide grating |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |