WO2022031982A1 - Système de traitement optique à modulation à amplitude seule massivement parallèle et procédés d'apprentissage automatique - Google Patents

Système de traitement optique à modulation à amplitude seule massivement parallèle et procédés d'apprentissage automatique Download PDF

Info

Publication number
WO2022031982A1
WO2022031982A1 PCT/US2021/044767 US2021044767W WO2022031982A1 WO 2022031982 A1 WO2022031982 A1 WO 2022031982A1 US 2021044767 W US2021044767 W US 2021044767W WO 2022031982 A1 WO2022031982 A1 WO 2022031982A1
Authority
WO
WIPO (PCT)
Prior art keywords
analog
optical
input
images
image
Prior art date
Application number
PCT/US2021/044767
Other languages
English (en)
Inventor
Mario Miscuglio
Volker Sorger
Original Assignee
The George Washington University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by The George Washington University filed Critical The George Washington University
Priority to US18/019,723 priority Critical patent/US20230298145A1/en
Publication of WO2022031982A1 publication Critical patent/WO2022031982A1/fr

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00Image enhancement or restoration
    • G06T5/20Image enhancement or restoration using local operators
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Neural networks
    • G06N3/08Learning methods
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Neural networks
    • G06N3/04Architecture, e.g. interconnection topology
    • G06N3/045Combinations of networks
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Neural networks
    • G06N3/06Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
    • G06N3/063Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using electronic means
    • G06N3/065Analogue means
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Neural networks
    • G06N3/06Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
    • G06N3/067Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using optical means
    • G06N3/0675Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using optical means using electro-optical, acousto-optical or opto-electronic means
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00Image enhancement or restoration
    • G06T5/10Image enhancement or restoration using non-spatial domain filtering
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00Image enhancement or restoration
    • G06T5/50Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00Arrangements for image or video recognition or understanding
    • G06V10/10Image acquisition
    • G06V10/12Details of acquisition arrangements; Constructional details thereof
    • G06V10/14Optical characteristics of the device performing the acquisition or on the illumination arrangements
    • G06V10/143Sensing or illuminating at different wavelengths
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00Arrangements for image or video recognition or understanding
    • G06V10/70Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/18Eye characteristics, e.g. of the iris
    • G06V40/193Preprocessing; Feature extraction
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/20Special algorithmic details
    • G06T2207/20081Training; Learning
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/20Special algorithmic details
    • G06T2207/20084Artificial neural networks [ANN]
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/20Special algorithmic details
    • G06T2207/20212Image combination
    • G06T2207/20221Image fusion; Image merging

Definitions

  • the present invention relates to the field of optical processing for artificial intelligence and machine learning applications.
  • Machine-intelligence has become a driving factor in modern society.
  • its demand outpaces the underlying electronic technology due to limitations given by fundamental physics such as capacitive charging of wires and by system architectures for storing and handling data, both driving recent trends towards processor heterogeneity.
  • CNN Convolution neural networks
  • NN neural network
  • Convolutions combine two pieces of information, namely a feature map and a kernel, to form a third function, such as a transformed feature map.
  • the feature maps, the kernels and the convolution operations between two matrixes are used herein according to their use and meaning in the neural networks field.
  • Convolution layers are responsible for consuming the majority ( ⁇ 80%) of the compute resources during inference tasks. This results in a significant latency and computational power consumption, especially for datasets comprising appreciably large feature maps, or requiring deep CNNs for achieving high accuracy, even when the network has been trained and the memory initialized.
  • data-parallel specialized architectures such as Graphic Processing Units (GPUs) and Tensor Processing Units (TPUs), providing a high-degree of programmability, deliver dramatic performance gains compared to general-propose processors.
  • TPUs and GPUs When used to implement deep NN performing inference on large two-dimensional data sets such as images, TPUs and GPUs are rather power-hungry and require a long computation time (> tens of ms), which is function of the complexity of the task and accuracy required. This translates into manifold operations with complex kernel and larger feature map.
  • Exemplary embodiments of the present invention provide analog amplitude-only Fourier optical processors and systems capable of processing large-scale matrices (e.g. larger than 1,000 x 1,000 elements) in a single time-step and microsecond-short latency (e.g. 100 microseconds).
  • the analog amplitude-only Fourier optical processors may be referred hereinafter as Analog-Optical-Processors.
  • Exemplary embodiments of the present invention provide methods for obtaining amplitude-only (AO) electro-optical convolutions between large matrices (e.g. feature maps corresponding to images and matrices used as kernels in neural networks) displayed by reprogrammable high-resolution amplitude-only spatial modulators (e.g. Digital Micromirror Devices, kHz-fast reprogrammable) based on two stages of Fourier Transforms (FT), without the support of any interferometric scheme.
  • the large matrices on which convolution is performed may be feature maps corresponding to images and kernel matrices used in neural networks classification systems.
  • the methods may be implemented via the Analog-Optical-Processors.
  • Exemplary embodiments of the present invention provide optoelectronic systems, such as the Analog-Optical-Processor, in which low-power laser light is actively patterned by electronically configured DMDs in both the object and Fourier plane of a 4f system, encoding information only in the amplitude of the wave-front.
  • the DMDs may include programmable micromirrors which can be individually controlled.
  • the DMDs may include a large number of programmable micromirrors (e.g. 2 million or more), with a certain resolution depth (e.g. lbit, 8 bit, etc.) and a high speed (e.g. about 1,031 Hz for 8bit resolution and about 20 kHz for 1 bit resolution).
  • the systems may achieve reprogrammable operations for near real-time, and may have about lOOx lower system latency than current GPU accelerators image processing.
  • the systems may achieve a throughput of up to 4-Peta operations per second at 8 bit resolution.
  • the systems may emulate on the same platform multiple convolutional layers of a NN.
  • Exemplary embodiments of the present invention provide analog optical convolutional neural networks (hereinafter may be referred as Analog-Optics-CNNs) performing accurate classification tasks on large matrices.
  • Analog-Optics-CNNs analog optical convolutional neural networks
  • the optical neural networks herein outperform current GPU based NNs and the NNs based on phase-based display technology in terms of latency, by one and two orders of magnitude, respectively.
  • the analog optical convolutional neural networks may be referred hereinafter as Analog-Optics-CNNs.
  • Exemplary embodiments of the present invention provide methods for training the Analog-Optics-CNNs off-chip, using a detailed physical model which describes the optical system and its nonidealities (e.g. optical aberrations and misalignments).
  • the optical processing system in a context of a neural network Analog- Optics-CNN obtained a classification accuracy of 98% and 54% for MNIST and CIFAR-10, respectively, with a throughput up to 1,000 convolutions per seconds between two 2MP images, which is one order of magnitude faster than the state-of-the-art GPU.
  • the optical processing systems herein may be employed as an accelerator for performing artificial intelligent tasks, such as convolution neural networks, to provide real-time, massively parallel throughput compared to current fully electronic systems.
  • artificial intelligent tasks such as convolution neural networks
  • An example of the system and its alternative forms are provided.
  • Results and simulations for the Neural network inference are also provided and prove the validity of the approach.
  • the results indicate that these intelligent information processing schemes (i.e. the systems, devices and methods disclosed herein) open new perspectives of a flexible and compact platforms which may be transformative for diverse applications.
  • the applications may range from image analysis to image classification and super-resolution imaging on unmanned aerial vehicles.
  • the systems and methods herein may also enable high bandwidth free-space communication in data centers, intelligently pre-processing data locally at the edge of the network.
  • the inventions herein enable the calculation of massively parallel amplitude-only convolutions in the Fourier domain, and enable accurate inference with processing time which is orders of magnitude smaller (e.g. one order of magnitude) than state-of
  • FIG 1. shows a principle diagram for an Analog-Optical-Processor based on 4F Fourier optical systems.
  • FIG 2. shows an Analog-Optical-Processor employing Digital Micromirror Devices for providing amplitude-only spatial modulation.
  • FIG. 3 shows a photograph of a prototype for the Analog-Optical-Processor employing
  • FIG. 4 shows an Analog-Optical-Processor which can be used as convolutional layer in a CNN and includes a computing and control system.
  • FIG. 5 shows a diagram of the factors considered in the physical model and in simulations for the Analog-Optical-Processor.
  • FIGS. 6a, b show diagrams for the data processing performed by the Analog-Optical- Processor and the Simulated-Optical-Processor.
  • FIGS. 7a, b, c show a comparison with respect to classification performance between experiments performed via the Analog-Optical-Processor and simulations based on the physical model.
  • FIG. 8 shows an optics based convolutional neural network (i.e. Analog-Optics-CNN).
  • FIGS. 9a, b shows a diagram comparing the architectures of an Analog-Optics-CNN and its corresponding Simulated-Optics-CNN.
  • FIG. 10 shows a flow-chart of a training process for the Analog-Optics-CNN.
  • FIG. 11 shows a diagram for a method of using an Analog-Optics-CNN system to perform classification of images by employing pre-trained kernels.
  • FIGS. 12a, b show a comparison with respect to classification performance between experiments performed via the Analog-Optical-CNN and simulations based on the physical model.
  • FIGS. 13a, 13b show a comparison between experiments and simulations with respect to the Structural Similarity Index Measure of the output images.
  • FIG. 14 shows a diagram for a fine-tuned method of using an Analog-Optics-CNN system to perform classification of images by employing pre-trained kernels and fine-tuned kernels.
  • FIG. 15 shows an embodiment of an Analog-Optical-Processor employing a parallelization scheme performing simultaneous batch processing on multiple input images.
  • FIGS. 16a, b show an Analog-Optical-Processor employing a parallelization scheme where the same input image is simultaneously filtered by multiple kernels.
  • FIG. 17 shows an Analog-Optical-Processor employing a diffraction based parallelization scheme where the same input image is simultaneously filtered by multiple kernels.
  • FIG. 18a shows an image classification system for performing real time classification in photogrammetry applications and for target recognition.
  • FIG. 18b is a flow diagram to isolate and track targets.
  • FIG. 19 shows a high-speed iris recognition and classification system.
  • Analog-Optical-Processor capable of processing large-scale matrices (e.g. 1,000 x 1,000) is described with reference to FIGS. 1-4.
  • the Analog-Optical-Processor is configured to process large-scale matrices in a single time-step and with a short latency (e.g. 100 microsecond-short).
  • the Analog-Optical-Processor is based on 4F systems such as shown in FIG. 1.
  • the 4F system may include an object-plane (where the input-image of the object is formed via, e.g., a first spatial-amplitude-filter), a first focal-lens, a Fourier plane where a second-spatial- amplitude-filter (hereinafter kernel filter) is disposed; a second-lens; and a detector array (camera) disposed in the image plane.
  • the object plane, the first-lens, the Fourier plane, the second lens, and the image plane / detector array are equally spaced by one focal-length.
  • the functioning of the 4F system is as follows: an input-image (i.e.
  • the letter "F" is generated at the object plane (e.g. via a plane wave which is spatially modulated in the object plane by an optical component, such as: a mask, micromirror array, metasurfaces, etc.); the input-image is transmitted to a first-lens which Fourier transforms the image onto the Fourier plane; the second spatial-amplitude-filter kernel spatially modulates the image in the Fourier plane (the Fourier transform of the input image is pixel-wise multiplied with the kernel); the light from the kernel is received by the second lens which performs an inverse Fourier transform projecting it on the image plane as output-image; the output-image is converted into electronic domain by the camera.
  • a plane wave which is spatially modulated in the object plane by an optical component, such as: a mask, micromirror array, metasurfaces, etc.
  • the input-image is transmitted to a first-lens which Fourier transforms the image onto the Fourier plane
  • the second spatial-amplitude-filter kernel spatially modulates the image
  • FIG. 2 shows an exemplary embodiment of an Analog-Optical-Processor which is a 4F optical processing system where the spatial-amplitude-filters are implemented via Digital Micromirror Devices (D M Ds) and which can be used a convolutional layer of a neural network.
  • the Analog-Optical-Processor may include the following components: an LED / laser source (1) emitting a beam, a collimator and a beam expander (2), a first DMD (DMD-1) (3), a first lens (Lens-1) (4), a second DMD (DMD-2) (5), a second lens (Lens-2) (6), and a camera (e.g. highspeed CCD) (7).
  • the DMD-1, Lens-1, DMD-2, Lens-2 and the camera may be spaced by one focal length "f" so as to form a 4F system.
  • the beam (e.g. low-power laser beam at 633 nm, HeNe Laser) may be collimated and expanded to uniformly intersect the entire active area of DMD-1 in the object plane.
  • DMD-1 provides spatial modulation defining the input-image (feature map), by independently tilting each micromirror of its array according to a pre-loaded pattern or input image / data.
  • the DMD-1 in the object plane may be oriented with a tilting angle (e.g. 22.5°) with respect to the normal incidence and may be rotated in-plane by an angle (e.g. 45°).
  • Light reflected from DMD- 1 is Fourier-transformed passing through the first Fourier lens disposed one focal length "f" from the first DMD-1 in the object plane.
  • the pattern in DMD-2 acts as a spatial mask / filter in the Fourier plane, selecting the spatial frequency components of the input image.
  • the Fourier transform of the input image is pixel-wise multiplied (amplitude only) with the kernel pattern (displayed by the DMD-2) in the Fourier plane of the 4-f system.
  • the frequency filtered image (result of the product) is inverse Fourier transformed into the real space by the second Fourier lens and imaged by a camera (e.g. high-speed camera) disposed in the image-plane.
  • the camera may image a square of the product result.
  • An all-optical or electronic nonlinearity may be applied to each camera pixel. Both Fourier transformation steps are performed entirely passively, i.e. zero-static power consumption, which is in stark contrast to performing convolutions as dot product multiplications in electronics.
  • the Analog-Optical-Processor is a 4F Fourier optical system, which acts as an Amplitude Only Fourier filter (AO-FF).
  • the invention is not limited by the particular geometries (e.g. angles, distances) and configurations disclosed herein. Various modification of geometrical parameters (e.g. angles, distances) can be made without limiting the scope of the invention.
  • the invention is not limited by the particular optical components used such as lenses and spatial-filters.
  • the functionality of the lenses may be implemented by equivalent components, such as: metalenses, diffractive optics tools, curved mirrors, and metareflectors.
  • the functionality of the spatial-amplitude-filters / DMDs may be implemented by equivalent components, such as: amplitude only diffractive masks, absorptive films, prepatterned films, phase change materials, two dimensional materials.
  • the input-image may be provided by the imaged object (e.g. input-image may be disposed on the object plan) itself and the DMD-1 is not needed.
  • DMD-1 and/or DMD-2 may be replaced by other types of spatial-amplitude-filters, such as: amplitude only diffractive masks, absorptive films, prepatterned films, phase change materials, 2 dimensional materials.
  • FIG. 3 shows an exemplary embodiment of an Analog-Optical-Processor as described in FIG. 2 (photograph of a working prototype).
  • both input images and kernels are updated with a frequency of 20 kHz at lbit and 1kHz at 8bit.
  • the integration time of the high-speed Charge Coupled Device camera is function of its resolution.
  • High-speed cameras with frame rate equal to update speed of the digital micromirror devices may be used.
  • the resolution of the DMDs is 2 Megapixel, but higher-resolution may be used.
  • High-speed analog micro-opto-electro-mechanical system (MOEMS) may be used.
  • the two DMDs are in specular orientation.
  • the DMDs are rotated out of plane (zy) by 24° degrees, which is parallel to the mirror on-state plane, and by 45° in plane (xy).
  • the resolution of the DMDs and the focal length of Lens-1 are selected to resize the Fourier Transform to the active area of DMD-1.
  • the focal length of the second Fourier lens is selected to generate an inverse Fourier transform of the same dimension as the CCD camera sensor.
  • FIG. 4 shows an exemplary embodiment of an Analog-Optical-Processor and a computing and control system configured to process data and to control the analog processor.
  • the Analog-Optical-Processor may include: data input/output interfacing and electronic architecture which performs data synchronization, data storage and the rest of the neural network tasks.
  • the Analog-Optical-Processor may include: a computing device including memories, a unified system interface; and a serial interface. The computing device stores input images and kernels.
  • the computing device (13) loads the input image as well as the kernel (e.g. 1920x1080 image, 8 bit deep, at a rate of 1000 Hz) to the DMDs by means of a HDMI cable or directly generated through an FPGA (e.g. Virtex 7) (14), which may connect to the Digital Light Processing (DLP) boards (15) (e.g. made by Texas Instrument) of the 2 DMDs 3 and 5 through a serial connection (16).
  • DLP Digital Light Processing
  • the serial connection may be configured to reduce the latency in providing the signals and allowing for processing while streaming data.
  • the amplitude only Fourier filtered images are detected with a high-speed charge-coupled-device (CCD) camera (e.g.
  • the unified system interface is configured to store the data or process it implementing other NN tasks, such as max pooling, activation function and fully connected layer.
  • the Analog-Optical-Processor may be used as convolution-layer in various NN system.
  • the Analog-Optical-Processor may be used to implement multiple convolution layers.
  • the resulting images could be potentially loaded into the 1st DMD. Images collected by the camera may be processed and loaded back into the 1 st DMD, and thereby further processed by the convolution layer.
  • the physical model is used for training the neural network off-chip and obtaining physically meaningful trained kernels.
  • the factors considered in the physical model and simulations are described with reference to FIG. 5 in the following: i). the input pattern is generated considering the DMDs' Magnitude Transfer Function (MTF), including contrast (reflectivity) and effective pixel size (factor 8); ii) the slanted angle at which the DMDs are oriented, by assigning a non-uniform phase delay (function of the inclination of the plane) and the resulting propagation mismatch (factor 9); iii) the lens aperture and aberration using Seidel Polynomial (factor 10); iv).
  • MTF Magnitude Transfer Function
  • factor 8 the slanted angle at which the DMDs are oriented
  • the first step is modelling the magnitude transfer function (MTF) of the DMDs or amplitude-only mask/film/metasurface which represents the capability of transferring the modulation depth of the input to the output signals at a specific spatial frequency, which represents the degree of fidelity of the digital signal.
  • MTF magnitude transfer function
  • each pixel is considered a square mirror with a hole etched at the center, which does not reflect light (pin on which the micromirror is hinged), and each mirror is separated from the neighboring mirrors by a small distance.
  • a driving electrical signal is applied to the DMD, an electro-static force is created between the electrodes of the selected mirror, so that it is tilted to deliver the illuminating light into the optical system (24°). In state “OFF" the mirror will reflect the illuminating light out of the optical system.
  • the DMD module used in the system is constituted of an array of up to 1920X1080 micromirrors with full addressing circuitry.
  • each mirror can be individually driven, such as by a motor or solenoid or the like, to rotate by applying a potential difference between the mirror and the addressing electrode.
  • the response time of each mirror is 10 ps, and the addressing voltage is 5 V.
  • the pixel pitch of the micromirror array of the DMD is about 17 pm.
  • Each pixel is a square micromirror of dimensions 16pmxl6pm, and with an etched hole of 1pm diameter at the center. Therefore, the fill factor r is approximately equal to 16/17, and the normalized radius of the hole, rc, is 0.5/17. For this reason, the algorithm performs a 17x17 pixel expansion of the input image associating the pattern of the mirror for modelling the optical image fidelity.
  • the MTF also takes into account imperfect contrast of the ON-OFF ratio which can be altered for each pixel in an 8bit resolution depth (factor 8).
  • the model follows in characterizing the non- uniform phase induced by the orientation and tilted angle of the micromirror array with respect to the propagating beam direction (factor 9).
  • the electric field which accounts for the tilting angle of the DMD and its orientation in space is obtained by elementwise multiplication of the field patterned by the 1st DMD and phase term proportional to the distance from the center and tilting angle 0.
  • djj is the physical distance between the centre of the DMD, considering the 45o in-plane rotation.
  • H T4(U, v) e -ikW , Eq. 4 being uo normalized image height, defined along the u axis in the imaging plane, A is the circular function which defines the circular aperture, given in terms of exit pupil size and pupil distance.
  • the aberrated wave-front of the Fourier transform is obtained by multiplying it with the H functions.
  • the lens is selected with respect to its focal length which dimension the Fourier transform.
  • the interaction with the second DMD is obtained by performing a pixel-wise multiplication between the 2nd DMD pattern and the impinging wave front, according to Huygens' Principle.
  • the resulting beam is inverse Fourier transformed obtaining the convolution in the real space (with flipped axis).
  • This step considers the aberration and f number of the 2nd Fourier lens with the rationale of having an image in the image plane of the same size of the CCD sensor (factor 11).
  • the CCD camera accomplishes the dimensionality reduction integrating the optical power (square optical intensity) mapping each expanded 17x17 super-pixel to a single pixel (factor 12).
  • the algorithm used for modelling the system can be used for similar 4f system which uses miniaturized reprogrammable metasurfaces and flat diffractive metalenses. In that case, the characterization of the optical tools and their inaccuracies would have even a greater impact to the results provided by the optical engine.
  • a Simulated-Optical-Processor which may be used as convolution layer in a neural network (NN) is described with reference to FIG. 6.
  • the Simulated-Optical-Processor is implemented via a software-routine run on a computing system.
  • the Simulated-Optical- Processor is configured to receive an input-image and a kernel and to generate an outputimage as shown in FIG. 6(b).
  • the output-image of the Simulated-Optical-Processor is obtained by emulating the Analog-Optical-Processor, i.e., performing computer simulations predicting the output-image (as function of the input-image and the kernel) which would be generated by the actual Analog- Optical-Processor (see FIGS. 6(a) and (b)).
  • the computer simulations may be conducted according to the physical model described above and may include all relevant parameters of the hardware Analog-Optical-Processor.
  • the output-image may be calculated as described with reference to equations 1-7 and will be a function of the input-image, the kernel, and the physical model parameters (e.g. dimensions, materials, and configurations of the optical components).
  • the emulated-convolution-layer simulated Analog-Optical-Processor
  • the Simulated-Optical-Processor should bring outcomes similar to the Analog-Optical-Processor. In other words, for the same pair input-image & kernel, the output-images generated by the Simulated-Optical-Processor and by the Analog-Optical-Processor should be very similar (if the simulation is perfect then the output of the Simulated-Optical-Processor and the Analog-Optical-Processor should be identical).
  • the Simulated-Optical-Processor may be implemented as a convolution-layer in a NN and could be used to perform classification of images.
  • the image-output of an Analog-Optical-Processor and the Simulated-Optical-Processor are compared, as explained with reference to FIG. 7.
  • the embodiment uses an 8bit (grey scale) high-resolution logo (head of George Washington) as input-image and six kernels (as seen in FIG. 7b).
  • the output-images generated by the Analog-Optical-Processor have been obtained experimentally for each of the six kernels.
  • the output-images generated by the Simulated- Optical-Processor have been obtained by simulations for each of the six kernels.
  • the images output by the Simulated-Optical-Processor are quite similar to the images output by the Analog-Optical-Processor.
  • the comparison between numerical simulation and experimental results for several kernels proves there is a very good agreement between experiment and model, which is also expressed in terms of Structural Similarity (SSI M) and Root mean square error (RMSE).
  • SSI M Structural Similarity
  • Analog optics based convolutional neural network (hereinafter referred as Analog- Optics-CNN) are described hereinafter with reference to FIG. 8.
  • the Analog-Optics-CNN is a convolutional neural network systems including a Analog-Optical-Processor.
  • an Analog-Optics-CNN system may include a DMD based Analog-Optical-Processor, as in FIG. 1-3, configured to perform a convolution between inputimages and kernels, and to capture output-images via a CCD camera.
  • the Analog-Optics-CNN system may be used to perform convolutions between any combinations of input-images (out of a set of input images) and kernels (e.g. the 16 kernels shown in FIG. 8).
  • the Analog-Optics- CNN may further include a computer system configured to implement one or more CNN layers, such as: an extraction layer; a batch normalization layer; a max pooling layer; a flattening layer; a Fully Connected Layer (FC); and a Rectifier linear unit (ReLu) layer.
  • the structure and functioning of these computer implemented CNN layers are well known by the artisan in computer sciences fields such as machine learning and AL
  • the Analog-Optics-CNN may be configured to perform the classification of the input-images into a set of classes (e.g. cat, dog, tiger, bear, etc.).
  • the Analog-Optical-Processor convolution layer is implemented in optical domain whereas the other layers are implemented in the Electrical Domain via one or more computing systems.
  • a method for performing image classification via the Analog-Optics-CNN system is described hereinafter with reference to FIGS. 2, 8.
  • An input-image (18) e.g. image to be analyzed, such as the cat image in FIG. 8) is loaded into the DMD-1 (3) and a first kernel (out of a set of kernels) is loaded into DMD-2 (5).
  • the Analog-Optical-Processor performs the Fourier convolution between the input-image and the kernel and the camera (7) captures the convoluted image transferring it into electrical domain.
  • the 4f convolution operation is repeated for the input-image with each of the kernels in a set of kernels (e.g.
  • the input-image (18) loaded in the DMD- 1 (3) may be a 208 x 208 pixels image from CIFAR-10 dataset (i.e. CIFAR-10 dataset is a collection of images that are commonly used to train machine learning and computer vision algorithms) so as to match the optical system (e.g. size of lenses and the DMDs).
  • CIFAR-10 dataset is a collection of images that are commonly used to train machine learning and computer vision algorithms
  • the input DMD-1 image is Fourier Transformed by FL1 lens (4) on a first pre-trained kernel (out of 16 pretrained kernels) implemented via DMD-2 (5).
  • the pre-trained kernel is also a 208x208 pixels kernel.
  • the output-image of the multiplication is transferred back to spatial domain by FL2 and is captured by the CCD/Camera (7) and converted into electrical domain.
  • the Analog-Optical-Processor system performs a matrix multiplication / convolution in optical domain (19) between the image (18) and the kernel.
  • the convolution operation is repeated for each of the 16 pre-trained kernels, thereby obtaining 16 output-images (see 16@208x208 images in FIG. 8).
  • the 16 output-images are run through an extraction layer, wherein images are reduced to 16 matrixes of 32x32 pixels.
  • the output of the extraction layer is run through a batch normalization and pooling layer wherein the images are further reduced to 16 matrixes of 16x16 pixels.
  • the output of the max pooling layer is run through a flattening layer thereby outputting a 1x256 matrix.
  • the output of the flattening layer is run through a Fully Connected and a ReLu function layer thereby outputting a 1x10 matrix (21) wherein each of the 10 elements of the matrix encodes 10 different classes.
  • the Simulated-Optics-CNN system is a CNN system including all layers of the Analog- Optics-CNN described with reference to FIG. 8 except that the convolution layer (i.e. the Analog-Optical-Processor) is replaced by a convolution layer implemented via the software routine of the Simulated-Optical-Processor (as seen in FIG. 9).
  • the Simulated-Optical-Processor convolutional layer is implemented in electronics domain only via a software-routine running on a computing system, as explained above with reference to FIG. 6.
  • FIG. 9 shows two convolutional neural networks (CNNs): an Analog-Optics-CNN (Fig. 9(a)) and its corresponding Simulated-Optics-CNN (Fig. 9(b)). Except for the convolution layer, all layers of the Simulated-Optics-CNN are identical with the layers of the Analog-Optics-CNN. Moreover, if the simulations of the physical model are good, the Simulated-Optical-Processor (i.e. the convolution layer in the Simulated-Optics-CNN) should bring outcomes similar to the Analog-Optical-Processor (i.e. the convolution layer in the Analog-Optics-CNN).
  • the Simulated-Optical-Processor i.e. the convolution layer in the Simulated-Optics-CNN
  • Analog-Optics-CNN and its corresponding Simulated-Optics-CNN should have similar classification performance. If the simulation is perfect then the output of the Simulated-Optics- CNN and the Analog-Optics-CNN should be identical.
  • FIG. 10 shows a Flow-chart of the training process for benchmarking dataset (22).
  • the physical model (23) of the amplitude only convolutional layer is used for training the entire CNN and for obtaining the weights for the kernel to be loaded in the 2nd DMD (5) of the convolution layer (24).
  • Experimentally obtained results (25) of the Amplitude Only Fourier filtering (performed by the Analog-Optical-Processor) are fed to the Fully Connected layer (FC) for performing the final prediction on unseen data.
  • FC Fully Connected layer
  • An ulterior fine-tuning process compensating for hardware inaccuracy which the physical model does not consider, can be implemented using the hardware-obtained convolution results in order to re-train the fully connected weights of the layer using a limited number of training samples.
  • the Simulated-Optics-CNN system is a proper CNN system and consequently can be trained, via known training methods in the machine learning field, so as to obtain the layer weights most suitable (or improved) for performing image classification for various classes of images (e.g. images in CIFAR-10, images in MNIST).
  • a training algorithm may be performed, via a computer, on the Simulated-Optics-CNN system (shown in FIG. 9(b)) thereby obtaining improved weights for a set of kernel layers that optimize the classification process.
  • the training algorithm may include a step where input-images are received from a Data Set (e.g.
  • the obtained improved kernel weights are used to form a set of trained kernels optimizing the classification process.
  • training may include optimization of weights of other CNN layers, different from the kernels.
  • the training algorithm may be performed via commercially available software, such as PyTorch (see e.g. https://pytorch.org/)
  • the training process (and finding the optimal kernels and weights) is performed on the Simulated-Optics-CNN instead of directly on the Analog-Optics-CNN (online learning / training) because of advantages of offered by the current computer systems, such as: speed ( ⁇ 500MHz for the GPU compared with 10kHz update rate of the DMDs) and ease of manipulating lots of data.
  • the Simulated-Optics-CNN system should perform similarly to the Analog-Optics-CNN system (if the simulation were perfect then the output of the Simulated-Optics-CNN and the Analog-Optics-CNN would be identical). Thus, if the trained kernels and trained FC-weights are optimizing the Simulated-Optics-CNN system then it is very likely they will optimize the Analog-Optics-CNN system.
  • An Analog-Optics-CNN system is disclosed (such as described with reference to FIGS. 9- 10) receiving at its DMD-2 the pre-trained kernels obtained via the Simulated-Optics-CNN.
  • the Analog-Optics-CNN system can be used to perform classification of the images as explained in the following.
  • a method for using an Analog-Optics-CNN system in conjunction with its corresponding Simulation-Optics-CNN system to perform classification on a set of input-images is described with reference to FIG. 11.
  • the method may include one or more of the following steps:
  • the training algorithm may include a step where input-images are received from a Data Set (e.g. MNIST); a step where a set of initial-kernels are received from a Data Set; steps implementing other layers of the Simulated-Optics-CNN system, such as performing a prediction or classification of the input-image and minimizing network's Loss Function (FIG. 10).
  • the training algorithm may be performed via a commercially available software such as PyTorch (see https://pytorch.org/);
  • step (c). run the Analog-Optics-CNN to perform classification on a set of input-images while using the pre-trained kernels obtained at step (b) in the second DMD (i.e. perform actual convolution in optics between input-images and the pre-trained kernels, capture the output-images at the CCD camera, and process the output-images through the other CNN layers);
  • FIG. 12(a) shows the 16 kernels obtained during training for the classification of handwritten digits (MNIST dataset).
  • the Analog-Optics-CNN system was blind tested, adopting the obtained kernels and using unseen images from the MNIST test dataset (not used as part of the training / validation) and achieved a 98% classification accuracy.
  • the Analog-Optics-CNN system was blind tested, adopting the obtained kernels and using unseen images from the MNIST test dataset (not used as part of the training / validation) and achieved a 98% classification accuracy.
  • the hardware implementation we perform convolutions between the kernels and unseen feature maps using the optical engine.
  • the results of the emulated and experimental convolution layer are compared in terms of transformed feature maps and classification accuracy. Since our simulation model already considers some nonidealities of the optical hardware, the convolution results of the hardware implementation match the simulation result quite well qualitatively and their shapes are almost identical as can be seen in FIG. 12(b). In FIG. 12(b) can be seen the output results of the emulated and experimental implementation of the convolution layer for different kernels (x- axis) and input images (y-axis). Although, the match is not perfect quantitatively, as highlighted by a lower SSIM (Structural Similarity Index Measure) as seen in FIG. 13(a). This is due to several concurring factors including a) small misalignment in the optical setup, b) model which takes into account unphysical reflection of grid boundaries, c) non-ideal camera dynamic range.
  • SSIM Structuretural Similarity Index Measure
  • the table in FIG. 13(b) shows testing results on the MNIST and CIFAR datasets with respect to the performance of a normal space domain convolution CNN (full precision), the Fourier convolution CNN using the simulation model, Analog-Optics CNN without fine tuning, and the Analog-Optics CNN with fine tuning. More details regarding the tests and the results are found in the Optica article included in the list of references herein (Miscuglio et. al, Optica 7, 1812-1819 (2020)).
  • the physical model cannot take into account all the features of the Analog-Optical- Processor (e.g. lens aberrations, imperfections of the DMDs, misalignments and imperfect distances and angles of components).
  • optical systems as the 4F optical processor often change in time because of changes in environmental conditions, such as thermal drift, and others. Consequently, the pre-trained kernels obtained for the CNN using simulations do not match perfectly the hardware Analog-Optics-CNN (since there are performance differences between the Analog-Optics-CNN and the Simulated-Optics-CNN). Therefore, there is a need to fine tune the Analog-Optics-CNN such as to account for the discrepancies between the simulated CNN and the hardware optics CNN. This can be achieved by finding fine-tuned- kernels (and other weights if available) better matching the hardware optics CNN.
  • one embodiment implements an ulterior fine-tuning process which uses the hardware convolution results to re-train the fully connected weights of the layer with a reduced number of training samples.
  • Fine tuning utilizes the knowledge learned via the simulation model from a full training set by determining a mapping from experimental results towards simulation results. Then, the mapping was used to compensate for the hardware-to-model discrepancies. This approach proved to be particularly useful and the tuned hardware results accuracy shows a significant improvement (98%) compared with the one without fine-tuning (92%).
  • this fine-tuning approach which compensates for hardware-to-model discrepancies can be used if the optical engine is processing data in harsh environment conditions, for application such as super-resolution on object detection performance in satellite imagery, which can cause random misalignments.
  • a method for using an Analog-Optics-CNN system in conjunction with its corresponding Simulation-Optics-CNN system to perform classification on a set of input-images is described with reference to FIG. 14.
  • the method may include one or more of the following steps:
  • the training algorithm may include a step where input-images are received from a Data Set (e.g. MNIST); a step where a set of initial-kernels are received from a Data Set; steps implementing other layers of the Simulated-Optics-CNN system, such as performing a prediction or classification of the input-image and minimizing network's Loss Function (FIG. 10)
  • the training algorithm may be performed via a commercially available software such as PyTorch (see https://pytorch.org/);
  • step (d). provide the preliminary-output-images to the Simulated-Optics-CNN and use them as improved training data; run training algorithm on the Simulated-Optics-CNN while using the preliminary-output-images to optimize / fine tune the pre-trained kernels (input-images and the pre-trained kernels obtained at step (b) may be used as start point). Determine the fine-tuned-kernels
  • step (e) run the Analog-Optics-CNN to perform classification on a set of input-images while using the fine-tuned-kernels obtained at step (d) in the second DMD (i.e. perform actual convolution in optics between input-images and the fine-tuned kernels, capture the output-images at the camera, and process the output-images through the other CNN layers); (f). for each input-image, generate classification information such as: class, accuracy, loss, etc.
  • Improved fine-tuned kernels may be further obtained by repeating steps (c) and (d) wherein the kernels of step (c) are replaced with the fine-tuned kernels.
  • the fine-tuned kernels can be further improved by iteratively repeating steps (c) and (d) a number of "n" times, wherein the kernels used at step (c) of an iteration are the fine-tuned kernels determined at step (d) during the previous iteration.
  • the Analog-Optics-CNN may use a set of kernels obtained by a process including performing training directly on the Analog-Optics-CNN (i.e. online training) or by a process including a combination between performing training on the Simulated-Optics-CNN and performing training on the Analog-Optics-CNN.
  • FIGS. 15-17 show different parallelization schemes for the optical processing system and methods.
  • FIG. 15 shows a first optical processing system implementing a batch processing scheme leveraging on the vast parallelism provided by the optical processing system (e.g. 2 Mega Pixels for the DMDs and the camera).
  • the all-optical processing system AO Fourier based convolutional layer
  • FIG. 15 shows an embodiment where multiple images (e.g. 9 images) may be tiled into the input plane (e.g. DMD-1 in the 4f system) (26) and batch-processed using the same kernel in the Fourier plane (27).
  • the image showing number "7" is provided on a corner portion of the input plane
  • the image "7” is Fourier Transformed by a lens on the kernel thereby performing a matrix multiplication / convolution between the image and kernel.
  • the result of the multiplication / convolution is transferred back to spatial domain on the output plane (e.g. CCD camera) as letter "7" (dimmer) displayed on the corner plane.
  • the output plane e.g. CCD camera
  • FIGS. 16 and 17 optical processing systems and architectures employ another parallelization scheme for the 4f system.
  • this architecture the same input (28) is simultaneously filtered by multiple kernels.
  • the Fourier transformed image generated by a first focal lens of a 4F system is multiplied (e.g. via a plurality of beam splitters) into a plurality of copy-images and each of the copy-images is directed (e.g. via a set of mirrors) to different kernels.
  • the Fourier transformed copy-images may be directed to the different kernels by using opportune beam splitters, array of mirrors and well-dimensioned lenslet array, as known in the art.
  • Each resulting product is inverse Fourier transformed (using a second lenslet array) and imaged by different sensors or different (non-overlapping) portions of one CCD array.
  • the filtered images can be integrated by the same sensor, performing dimensionality reduction.
  • An exemplary embodiment discloses an optical processor 300 employing multi-kernel parallelization architecture as described with respect to FIG. 16.
  • the processor 300 may include an image-input incident as a wavefront signal onto a first lens. After emerging from the first lens, the image signal is processed by a signal splitter (FIG. 16(b)) configured to create a set of signal copies (copy-1, copy-2 and copy-3 in this case). Each of the image copies is incident upon a separate kernel (i.e. kernel-1, kernel-2, and kernel-3) thereby being processed by the kernel.
  • the three signals emerging from the kernels, representing a convolution between the input signal and the kernel are further redirected through the second focal lens.
  • the second focal lens Fourier transforms the three signals and directs each of them on separate non-overlapping areas of the detector array.
  • the three signals detected on the detector array are converted into electric domain by the camera and each will represent the convolution between the input image and the corresponding kernel out of the three kernels.
  • Delay elements and geometry of the system may be implemented so as to ensure a 4F spacing between the elements.
  • FIG. 16(b) shows an exemplary embodiment of a signal splitter configured to receive one input-image and to generate three separate copies of the inputimage.
  • FIG. 17 shows another exemplary embodiment (i.e. processor 400) of the multi-kernel based Analog-Optical-Processor.
  • the Analog-Optical-Processors employs a diffractive component configured to receive an input image and to create a plurality of output images directed at different angles with respect to the optical axis (as seen in FIG. 17).
  • Each of the plurality of images corresponds to an interference / diffraction order.
  • Each of the plurality of images may be redirected through a separate kernel (as shown in FIG. 17).
  • Lens-1 acts as diffractive component and therefore the system does not include a separate diffractive component.
  • the kernels are shown as transmissive elements for simplicity, the skilled artisan would understand that the geometry of the system can be configured so as to use DMDs (reflective components) to implement the kernels. It is understood that the same system may be implemented for a number of kernels which is much larger than 3 and that the optical processor 300 is not limited by the number of kernels.
  • the kernels may be provided by separate DMDs (one DMD for one kernel) or by separate non-overlapping portions of one DMD (all kernels provided by on DMD simultaneously).
  • FIG. 18 (a) shows an Analog-Optical-Processor used as convolutional layer in an image classification system used for real time classification in (36) photogrammetry applications (37) target recognition.
  • the image classification system may be used to perform real-time target and image recognition on images collected by an airplane or a satellite.
  • FIG. 18(b) shows a flow-chart of an exemplary method which can be used to isolate and track targets in maritime environments and to perform potential track prediction.
  • FIG. 19 shows an exemplary embodiment of an image classification system 500 using an analog optical processor as amplitude-only convolutional layer.
  • the system 500 may be used to perform high-speed iris recognition and classification.
  • the system 500 includes an analog optical processor having a modified 4F system configuration but functioning similarly to the analog optical processor in FIGS. 1-3.
  • the system may include a non-invasive low power infrared laser (38) whose beam is collimated via a lens or other optical system so as to illuminate the iris (39) of the eye of a user.
  • the collimated light reflected and spatially modulated by the iris's features acts as the input-image in the 4F optical processing system (the iris is in the input plane of the 4F system).
  • the light reflected by the iris is spatially modulated by the iris' features, thereby forming an iris-image.
  • the light reflected from the user's iris is directed to the First Fourier lens and transformed into its spatial frequency component passing through a Fourier lens (1st Fourier Lens) disposed at a first focal distance "f", then the light is directed (e.g. via a beam splitter allowing the light to pass through) to a DMD system disposed two focal distances from the iris in the Fourier plane of the 4F system.
  • the Fourier transform of the iris interacts with the pattern generated by a DMD, loaded with different signatures updating at a speed such as 20kHz (40).
  • the light reflected / modulated by the DMD is directed (e.g. via a beam splitter reflecting the light) towards the 2nd Fourier Lens, inverse transformed and imaged by a camera for subsequent post processing.
  • the iris-images received at the camera may be further processed by other layers of a neural network, such as Fully Connected Layer, thereby being classified in a set of classes.
  • the system may further perform image recognition on the iris-images by determining if one or more of the iris-images corresponds to a certain person (which person may be granted access to a certain facility or service).
  • the input-image (displayed at the input / object plane) is not provided via DMDs but it comes directly from the object to be classified, i.e., the iris.
  • the iris itself provides space modulation of the input image.
  • an optical system for performing tensor operations wherein the optical system includes or employs an Analog- Optical-Processor.
  • the system for performing tensor operations may be any suitable system, such as the one disclosed in the International Patent Application PCT/US2020/028516 titled "Photonic Tensor Core Matrix Multiplier" invented by the inventors herein, and incorporated herein by reference.
  • the system includes or uses the Analog-Optical-Processor.
  • the inventions herein are not limited by DMD's configurations and parameters, such as number of micromirrors, dimension of micromirrors. It is understood that various types of DMDs may be used without limiting the scope of the invention.
  • the DMDs may include 2 million individually controlled and programmable micromirrors (higher resolution is also achievable), with a resolution depth of 8 bit and a speed of 1,031 Hz ( ⁇ 20 kHz with 1 bit resolution).
  • the DMDs may enable the achievement of reprogrammable operations for (near) real-time, which is about lOOx lower system latency with respect to current GPU accelerators (SLM-based systems) image processing, with a maximum throughput of 4-Peta operations per second at 8 bit resolution, emulating on the same platform multiple convolutional layers of a neural network.
  • SLM-based systems GPU accelerators
  • the inventions are not limited by the optical devices used to generate the spatial amplitude filters, the spatial light modulators and the amplitude-only patterns (e.g. the input images, the kernel patterns, the input matrix, the kernel matrix). It is understood that various types of optical devices may be used to generate the amplitude-only patterns without limiting the scope of the invention.
  • the spatial light modulators may be implemented via devices such as DMDs, high-speed analog micro-opto-electro-mechanical systems (e.g. with large resolution and fast switching rate), patterned diffractive masks or materials, such as phase change materials (e.g.
  • the input images and the amplitude-only filters may be any of two-dimensional matrixes / images, mono-dimensional, gray-scale, binary, multichannel (e.g. colored) images.
  • Amplitude- only fixed filters, films or selectively patterned materials may be utilized, both in the object and Fourier plane.
  • Arrangements of the optical processor may use both transmissive and reflective amplitude-only spatial modulators in both object and Fourier plane.
  • the inventions are not limited by the optical components used to perform the Fourier transforms (e.g. the Fourier lenses). It is understood that various types of optical components may be used to perform the Fourier transforms without limiting the scope of the invention.
  • the Fourier lenses may be replaced by short distance diffractive optical elements, curved mirrors or metalenses or meta-reflector.
  • the entire system and operation is conducted automatically, and without any manual interaction.
  • the process occurs substantially in real-time without any delay or manual action.
  • the system operates dynamically; for example, the various components continually receive signals for training and operation.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Software Systems (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Computing Systems (AREA)
  • Multimedia (AREA)
  • Molecular Biology (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Data Mining & Analysis (AREA)
  • Computational Linguistics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Neurology (AREA)
  • Ophthalmology & Optometry (AREA)
  • Human Computer Interaction (AREA)
  • Databases & Information Systems (AREA)
  • Medical Informatics (AREA)
  • Image Analysis (AREA)

Abstract

Des processeurs optiques de Fourier à amplitude seule sont capables de traiter des matrices à grande échelle en une seule étape temporelle avec une latence d'une microseconde. Les processeurs peuvent avoir une architecture de système optique 4f et peuvent utiliser des modulateurs spatiaux à amplitude seule à haute résolution reprogrammables, tels que des dispositifs à micromiroirs numériques (DMD). De plus, l'invention concerne des procédés permettant d'obtenir des convolutions électro-optiques à amplitude seule entre de grandes matrices affichées par les DMD. Les grandes matrices sur lesquelles une convolution est réalisée peuvent être des cartes de caractéristiques correspondant à des images et des matrices de noyau utilisées dans des systèmes de classification de réseaux neuronaux. L'invention concerne également des réseaux neuronaux de convolution optiques analogiques qui effectuent des tâches de classification précises sur de grandes matrices. De plus, l'invention concerne des procédés pour l'apprentissage hors puce des réseaux neuronaux de convolution optiques analogiques. L'apprentissage comprend la construction d'un modèle physique précis pour le processeur optique analogique et la réalisation de simulations informatiques du processeur optique selon le modèle physique. Les procédés n'ont besoin d'employer aucun schéma interférométrique.
PCT/US2021/044767 2020-08-05 2021-08-05 Système de traitement optique à modulation à amplitude seule massivement parallèle et procédés d'apprentissage automatique WO2022031982A1 (fr)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US18/019,723 US20230298145A1 (en) 2020-08-05 2021-08-05 Massively parallel amplitude-only optical processing system and methods for machine learning

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US202063061487P 2020-08-05 2020-08-05
US63/061,487 2020-08-05

Publications (1)

Publication Number Publication Date
WO2022031982A1 true WO2022031982A1 (fr) 2022-02-10

Family

ID=80117665

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2021/044767 WO2022031982A1 (fr) 2020-08-05 2021-08-05 Système de traitement optique à modulation à amplitude seule massivement parallèle et procédés d'apprentissage automatique

Country Status (2)

Country Link
US (1) US20230298145A1 (fr)
WO (1) WO2022031982A1 (fr)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115392138A (zh) * 2022-10-27 2022-11-25 中国航天三江集团有限公司 基于机器学习的光-机-热耦合分析模型

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
GB2561238A (en) * 2017-04-07 2018-10-10 Univ Bath Apparatus and method for monitoring objects in space
US20220147799A1 (en) * 2020-11-12 2022-05-12 Samsung Electronics Co., Ltd. Neural computer including image sensor capable of controlling photocurrent

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2001004687A1 (fr) * 1999-07-09 2001-01-18 Opts, Inc. Reseau adaptatif a compression
US20100074478A1 (en) * 2005-06-03 2010-03-25 Hoyos Hector T Method and apparatus for iris biometric systems for use in an entryway
US20140085713A1 (en) * 2012-09-25 2014-03-27 The Board Of Trustees Of The University Of Illinois Phase Derivative Microscopy
US20200202590A1 (en) * 2018-11-15 2020-06-25 InstaRecon, Inc. Method and system for fast reprojection

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2001004687A1 (fr) * 1999-07-09 2001-01-18 Opts, Inc. Reseau adaptatif a compression
US20100074478A1 (en) * 2005-06-03 2010-03-25 Hoyos Hector T Method and apparatus for iris biometric systems for use in an entryway
US20140085713A1 (en) * 2012-09-25 2014-03-27 The Board Of Trustees Of The University Of Illinois Phase Derivative Microscopy
US20200202590A1 (en) * 2018-11-15 2020-06-25 InstaRecon, Inc. Method and system for fast reprojection

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115392138A (zh) * 2022-10-27 2022-11-25 中国航天三江集团有限公司 基于机器学习的光-机-热耦合分析模型

Also Published As

Publication number Publication date
US20230298145A1 (en) 2023-09-21

Similar Documents

Publication Publication Date Title
US20230298145A1 (en) Massively parallel amplitude-only optical processing system and methods for machine learning
Miscuglio et al. Massively parallel amplitude-only Fourier neural network
Wetzstein et al. Inference in artificial intelligence with deep optics and photonics
Mait et al. Computational imaging
CN104508586B (zh) 可重新配置的光学处理系统
EP2328007B1 (fr) Systèmes d'imagerie basés sur des tâches
US8760516B2 (en) Task-based imaging systems
JPH0868967A (ja) 画像処理方法及び画像処理装置
CA2345261A1 (fr) Ensembles lentilles programmables et systemes optiques comprenant ces derniers
JP2007513427A (ja) 光学システムおよびデジタルシステムの設計を最適化するシステムおよび方法
Bai et al. All-optical image classification through unknown random diffusers using a single-pixel diffractive network
Arguello et al. Deep optical coding design in computational imaging: a data-driven framework
Hu et al. Highly-parallel optical fourier intensity convolution filter for image classification
Wittpahl et al. Realistic image degradation with measured PSF
Hazineh et al. Polarization multi-image synthesis with birefringent metasurfaces
CN114415369A (zh) 成像方法、成像装置、光学成像系统及车辆
Chao et al. Optical implementation of a feature-based neural network with application to automatic target recognition
Yildirim et al. Nonlinear processing with linear optics
Shamir Adaptive pattern recognition correlators
Gamboa et al. High-speed Opto-electronic Pre-processing of Polar Mellin Transform for Shift, Scale and Rotation Invariant Image Recognition at Record-Breaking Speeds
US20230401436A1 (en) Scale-, shift-, and rotation-invariant diffractive optical networks
Shevkunov et al. Deep convolutional neural network-based lensless quantitative phase retrieval
Miscuglio et al. Fourier Optics Coprocessor for Image Processing and Convolutional Neural Network
Redman et al. Performance evaluation of two optical architectures for task-specific compressive classification
CN117454949A (zh) 全光学卷积神经网络装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21852663

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21852663

Country of ref document: EP

Kind code of ref document: A1