WO2022166189A1 - 光人工神经网络智能芯片及制备方法 - Google Patents

光人工神经网络智能芯片及制备方法 Download PDF

Info

Publication number
WO2022166189A1
WO2022166189A1 PCT/CN2021/115966 CN2021115966W WO2022166189A1 WO 2022166189 A1 WO2022166189 A1 WO 2022166189A1 CN 2021115966 W CN2021115966 W CN 2021115966W WO 2022166189 A1 WO2022166189 A1 WO 2022166189A1
Authority
WO
WIPO (PCT)
Prior art keywords
neural network
artificial neural
optical
different
layer
Prior art date
Application number
PCT/CN2021/115966
Other languages
English (en)
French (fr)
Inventor
崔开宇
熊健
杨家伟
黄翊东
张巍
冯雪
刘仿
Original Assignee
清华大学
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 清华大学 filed Critical 清华大学
Publication of WO2022166189A1 publication Critical patent/WO2022166189A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/06—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
    • G06N3/067—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using optical means
    • G06N3/0675—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using optical means using electro-optical, acousto-optical or opto-electronic means
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/044—Recurrent networks, e.g. Hopfield networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/06—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
    • G06N3/067—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using optical means
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/10—Interfaces, programming languages or software development kits, e.g. for simulating neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/10—Terrestrial scenes

Definitions

  • the present application relates to the technical field of artificial intelligence, and in particular, to an optical artificial neural network smart chip and a preparation method.
  • the existing intelligent recognition technology usually needs to image a person or object first, and then go through the steps of image preprocessing, feature extraction, and feature matching to realize the recognition of the person or object.
  • image preprocessing image preprocessing
  • feature extraction feature matching
  • feature matching feature matching
  • the embodiments of the present application provide an optical artificial neural network smart chip and a preparation method.
  • an embodiment of the present application provides an optical artificial neural network smart chip, including: an optical filter layer, an image sensor and a processor; the optical filter layer corresponds to the input layer, linear layer and all The connection weight from the input layer to the linear layer, the square detection response of the image sensor corresponds to the first nonlinear activation function in the nonlinear layer of the artificial neural network; the processor corresponds to the full connection of the artificial neural network and The output layer, or the processor corresponds to the full connection of the artificial neural network, the second nonlinear activation function in the nonlinear layer, and the output layer;
  • the optical filter layer is disposed on the surface of the image sensor, and the optical filter layer includes a light modulation structure, and the light modulation structure is used for the incident light entering into different positions of the light modulation structure.
  • different spectral modulations to obtain incident light-carrying information corresponding to different positions on the surface of the image sensor;
  • the incident light-carrying information includes light intensity distribution information, spectral information, angle information of the incident light, and the The phase information of the incident light;
  • the image sensor converts the information carried by the incident light corresponding to the different position points after being modulated by the optical filter layer to the electrical signals corresponding to the different position points after the first nonlinear activation processing through the square detection response, and will be different from the different position points. sending the electrical signal corresponding to the location point to the processor;
  • the processor performs full connection processing on the electrical signals corresponding to different position points, or the processor performs full connection processing and second nonlinear activation processing on the electrical signals corresponding to different position points to obtain an artificial neural network. output signal.
  • optical artificial neural network intelligent chip is used for intelligent processing tasks of the target object;
  • the intelligent processing tasks include at least one or more of intelligent perception, intelligent recognition and intelligent decision-making tasks;
  • the reflected light, transmitted light and/or radiated light of the target object enters the trained optical artificial neural network smart chip, and the intelligent processing result of the target object is obtained; the intelligent processing result at least includes the intelligent perception result and the intelligent recognition result and/or one or more of the Smart Decision Results;
  • the trained optical artificial neural network intelligent chip refers to the optical artificial neural network intelligent chip including the trained optical modulation structure, image sensor and processor;
  • the trained light modulation structures, image sensors and processors refer to using the input training samples and output training samples corresponding to the intelligent processing tasks to perform data analysis on data including different light modulation structures, image sensors and different fully connected parameters.
  • the input training samples and output training samples corresponding to the intelligent processing tasks described above are intelligent for optical artificial neural networks containing different light modulation structures, image sensors, and processors with different fully connected parameters and different second nonlinear activation parameters.
  • the light modulation structure, image sensor and processor that meet the training convergence conditions are obtained by the chip training.
  • the different light modulation structures are designed and realized by adopting computer optical simulation design.
  • the light modulation structures in the optical filter layer include regular structures and/or irregular structures; and/or, the light modulation structures in the optical filter layer include discrete structures and/or continuous structures.
  • the optical filter layer has a single-layer structure or a multi-layer structure.
  • the light modulation structure in the optical filter layer includes a unit array composed of a plurality of micro/nano units, each micro/nano unit corresponds to one or more pixels on the image sensor; the structures of each micro/nano unit are the same or different.
  • micro-nano unit includes a regular structure and/or an irregular structure; and/or, the micro-nano unit includes a discrete structure and/or a continuous structure.
  • micro-nano unit includes a plurality of groups of micro-nano structure arrays, and the structures of each group of micro-nano structure arrays are the same or different.
  • each group of micro-nano structure arrays has the function of broadband filtering or narrow-band filtering.
  • each group of micro-nano structure arrays is a periodic structure array or a non-periodic structure array.
  • micro-nano unit has polarization independent properties.
  • micro-nano unit has quadruple rotational symmetry.
  • the filter layer is made of one or more of semiconductor materials, metal materials, liquid crystals, quantum dot materials, and perovskite materials; and/or, the filter layer is made of photonic crystal, ultra A filter layer prepared by one or more of surface, random structure, nanostructure, metal surface plasmon SPP micro-nano structure, and tunable Fabry-Perot resonant cavity.
  • the semiconductor material includes one or more of silicon, silicon oxide, silicon nitride, titanium oxide, composite materials mixed in a preset ratio, and direct bandgap compound semiconductor materials; and/or, the nanometer
  • the structure includes one or more of nanodot 2D materials, nanopillar 2D materials, and nanowire 2D materials.
  • the thickness of the optical filter layer is 0.1 ⁇ ⁇ 10 ⁇ , where ⁇ represents the center wavelength of the incident light.
  • the image sensor is any one or more of the following:
  • CMOS image sensor CIS charge coupled element CCD, single photon avalanche diode SPAD array and focal plane photodetector array.
  • an embodiment of the present application further provides an intelligent device, including: the optical artificial neural network intelligent chip as described in the first aspect.
  • the smart devices include one or more of smart phones, smart computers, smart identification devices, smart perception devices, and smart decision-making devices.
  • the embodiments of the present application also provide a method for preparing an optical artificial neural network smart chip as described in the first aspect, including:
  • the optical filter layer is used to perform different spectral modulation on the incident light entering different positions of the optical modulation structure through the optical modulation structure, so as to obtain different spectral modulations on the surface of the image sensor.
  • the image sensor converts the information carried by the incident light corresponding to the different position points after being modulated by the optical filter layer to the electrical signals corresponding to the different position points after the first nonlinear activation processing through the square detection response, and will be different from the different position points. sending the electrical signal corresponding to the location point to the processor;
  • the processor performs full connection processing on the electrical signals corresponding to different position points, or the processor performs full connection processing and second nonlinear activation processing on the electrical signals corresponding to different position points to obtain an artificial neural network. output signal.
  • preparing an optical filter layer containing a light modulation structure on the surface of the image sensor including:
  • an optical filter layer containing a light modulation structure is obtained;
  • the input training samples and the output training samples corresponding to the intelligent processing task are used to analyze the optical modulation structure, image sensor and
  • the optical artificial neural network smart chips with different fully connected parameters are trained to obtain light modulation structures, image sensors and processors that satisfy the training convergence conditions;
  • the optical artificial neural network intelligent chip of the processor with the fully connected parameters and different second nonlinear activation parameters is trained, and the optical modulation structure, image sensor and processor that meet the training convergence conditions are obtained.
  • the embodiment of the present application also provides an optical artificial neural network environmental protection monitoring chip, which is used for intelligent processing tasks of environmental protection monitoring, including: an optical filter layer, an image sensor and a processor; the optical filter layer corresponds to the input of the artificial neural network layer, linear layer and the connection weight from the input layer to the linear layer, the square detection response of the image sensor corresponds to the first nonlinear activation function in the nonlinear layer of the artificial neural network; the processor corresponds to the artificial neural network The full connection and the output layer of the neural network, or the processor corresponds to the full connection of the artificial neural network, the second nonlinear activation function in the nonlinear layer, and the output layer;
  • the optical filter layer is disposed on the surface of the image sensor, and the optical filter layer includes a light modulation structure, and the light modulation structure is used for the incident light entering into different positions of the light modulation structure.
  • different spectral modulations to obtain incident light-carrying information corresponding to different positions on the surface of the image sensor;
  • the incident light-carrying information includes light intensity distribution information, spectral information, angle information of the incident light, and the Phase information of incident light;
  • the incident light includes reflected light, transmitted light and/or radiated light of environmental pollutants;
  • the image sensor converts the information carried by the incident light corresponding to the different position points after being modulated by the optical filter layer to the electrical signals corresponding to the different position points after the first nonlinear activation processing through the square detection response, and will be different from the different position points. sending the electrical signal corresponding to the location point to the processor;
  • the processor performs full connection processing on electrical signals corresponding to different locations, or the processor performs full connection processing and second nonlinear activation processing on electrical signals corresponding to different locations, so as to obtain environmental monitoring intelligence. process result;
  • the environmental protection monitoring intelligent processing task includes identification and/or qualitative analysis of environmental pollutants; the environmental protection monitoring intelligent processing results include identification results of environmental pollutants and/or qualitative analysis results of environmental pollution.
  • the optical artificial neural network environmental protection monitoring chip includes a trained light modulation structure, an image sensor and a processor;
  • the trained light modulation structures, image sensors and processors refer to using the input training samples and output training samples corresponding to the environmental monitoring intelligent processing tasks, to perform data analysis on the light modulation structures, image sensors and processors with different light modulation structures and image sensors.
  • the optical modulation structure, image sensor and processor that meet the training convergence conditions obtained by training the optical artificial neural network environmental monitoring chip connected to the parameter processor; or, the trained optical modulation structure, image sensor and processor refer to Using the input training samples and output training samples corresponding to the environmental monitoring intelligent processing task, the processing of the processor including different light modulation structures, image sensors and processors with different fully connected parameters and different second nonlinear activation parameters An optical modulation structure, an image sensor and a processor that satisfy the training convergence conditions obtained by training the optical artificial neural network environmental protection monitoring chip;
  • the input training samples include incident light reflected, transmitted and/or radiated by different environmental pollutants; the output training samples include corresponding identification results of environmental pollutants; and/or, the input training samples include Incident light reflected, transmitted and/or radiated by environmental pollutants; the output training sample includes corresponding qualitative analysis results of environmental pollution.
  • the different light modulation structures are designed and realized by adopting computer optical simulation design.
  • the light modulation structures in the optical filter layer include regular structures and/or irregular structures; and/or, the light modulation structures in the optical filter layer include discrete structures and/or continuous structures.
  • the optical filter layer has a single-layer structure or a multi-layer structure.
  • the light modulation structure in the optical filter layer includes a unit array composed of a plurality of micro/nano units, each micro/nano unit corresponds to one or more pixels on the image sensor; the structures of each micro/nano unit are the same or different.
  • micro-nano unit includes a regular structure and/or an irregular structure; and/or, the micro-nano unit includes a discrete structure and/or a continuous structure.
  • micro-nano unit includes a plurality of groups of micro-nano structure arrays, and the structures of each group of micro-nano structure arrays are the same or different.
  • each group of micro-nano structure arrays has the function of broadband filtering or narrow-band filtering.
  • each group of micro-nano structure arrays is a periodic structure array or a non-periodic structure array.
  • micro-nano unit has polarization independent properties.
  • micro-nano unit has quadruple rotational symmetry.
  • the filter layer is made of one or more of semiconductor materials, metal materials, liquid crystals, quantum dot materials, and perovskite materials; and/or, the filter layer is made of photonic crystal, ultra A filter layer prepared by one or more of surface, random structure, nanostructure, metal surface plasmon SPP micro-nano structure, and tunable Fabry-Perot resonant cavity.
  • the semiconductor material includes one or more of silicon, silicon oxide, silicon nitride, titanium oxide, composite materials mixed in a preset ratio, and direct bandgap compound semiconductor materials; and/or, the nanometer
  • the structure includes one or more of nanodot 2D materials, nanopillar 2D materials, and nanowire 2D materials.
  • the thickness of the optical filter layer is 0.1 ⁇ ⁇ 10 ⁇ , where ⁇ represents the center wavelength of the incident light.
  • the embodiment of the present application also provides an environmental protection monitoring device, including the optical artificial neural network environmental protection monitoring chip as described above.
  • Embodiments of the present application also provide a method for preparing an optical artificial neural network environmental protection monitoring chip as described above, including:
  • the optical filter layer is used to perform different spectral modulation on the incident light entering different positions of the optical modulation structure through the optical modulation structure, so as to obtain different spectral modulations on the surface of the image sensor.
  • the image sensor converts the information carried by the incident light corresponding to the different position points after being modulated by the optical filter layer to the electrical signals corresponding to the different position points after the first nonlinear activation processing through the square detection response, and will be different from the different position points. sending the electrical signal corresponding to the location point to the processor;
  • the processor performs full connection processing on electrical signals corresponding to different locations, or the processor performs full connection processing and second nonlinear activation processing on electrical signals corresponding to different locations, so as to obtain environmental monitoring intelligence. process result.
  • the preparation method of the optical artificial neural network environmental protection monitoring chip also includes: a training process for the optical artificial neural network environmental protection monitoring chip, specifically including:
  • the optical artificial neural network environmental monitoring chips including different light modulation structures, image sensors and processors with different fully connected parameters are trained to obtain A light modulation structure, an image sensor, and a processor that satisfy the training convergence conditions, and the light modulation structure, the image sensor, and the processor that satisfy the training convergence conditions are used as the trained light modulation structure, image sensor, and processor;
  • the processing that includes different light modulation structures, image sensors, and different fully connected parameters and different second nonlinear activation parameters
  • the optical artificial neural network environmental monitoring chip of the device is trained to obtain a light modulation structure, an image sensor and a processor that meet the training convergence conditions, and the light modulation structure, image sensor and processor that meet the training convergence conditions are used as the trained light modulation structure. , image sensor and processor.
  • the different light modulation structures are designed and realized by adopting computer optical simulation design.
  • the embodiment of the present application also provides an optical artificial neural network fingerprint identification chip, which is used for fingerprint identification processing tasks, including: an optical filter layer, an image sensor and a processor; the optical filter layer corresponds to the input layer of the artificial neural network , the linear layer and the connection weight from the input layer to the linear layer, the square detection response of the image sensor corresponds to the first nonlinear activation function in the nonlinear layer of the artificial neural network; the processor corresponds to the artificial neural network The full connection and the output layer of the network, or the processor corresponds to the full connection of the artificial neural network, the second nonlinear activation function in the nonlinear layer, and the output layer;
  • the optical filter layer is disposed on the surface of the image sensor, and the optical filter layer includes a light modulation structure, and the light modulation structure is used for the incident light entering into different positions of the light modulation structure.
  • different spectral modulations to obtain incident light-carrying information corresponding to different positions on the surface of the image sensor;
  • the incident light-carrying information includes light intensity distribution information, spectral information, angle information of the incident light, and the Phase information of incident light;
  • the incident light includes reflected light, transmitted light and/or radiated light of the user's fingerprint;
  • the image sensor converts the information carried by the incident light corresponding to the different position points after being modulated by the optical filter layer to the electrical signals corresponding to the different position points after the first nonlinear activation processing through the square detection response, and will be different from the different position points. sending the electrical signal corresponding to the location point to the processor;
  • the processor performs full connection processing on the electrical signals corresponding to different position points, or the processor performs full connection processing and second nonlinear activation processing on the electrical signals corresponding to different position points to obtain fingerprint identification processing. result.
  • the optical artificial neural network fingerprint identification chip includes a trained light modulation structure, an image sensor and a processor;
  • the trained light modulation structures, image sensors and processors refer to using the input training samples and output training samples corresponding to the fingerprint recognition processing tasks to perform data analysis on different light modulation structures, image sensors, and fully connected
  • the input training samples include incident light reflected, transmitted and/or radiated by different human fingerprints; and the output training samples include corresponding fingerprint identification results.
  • the different light modulation structures are designed and realized by adopting computer optical simulation design.
  • the light modulation structures in the optical filter layer include regular structures and/or irregular structures; and/or, the light modulation structures in the optical filter layer include discrete structures and/or continuous structures.
  • the optical filter layer has a single-layer structure or a multi-layer structure.
  • the light modulation structure in the optical filter layer includes a unit array composed of a plurality of micro/nano units, each micro/nano unit corresponds to one or more pixels on the image sensor; the structures of each micro/nano unit are the same or different.
  • micro-nano unit includes a regular structure and/or an irregular structure; and/or, the micro-nano unit includes a discrete structure and/or a continuous structure.
  • micro-nano unit includes a plurality of groups of micro-nano structure arrays, and the structures of each group of micro-nano structure arrays are the same or different.
  • each group of micro-nano structure arrays has the function of broadband filtering or narrow-band filtering.
  • each group of micro-nano structure arrays is a periodic structure array or a non-periodic structure array.
  • micro-nano unit has polarization independent properties.
  • micro-nano unit has quadruple rotational symmetry.
  • the filter layer is made of one or more of semiconductor materials, metal materials, liquid crystals, quantum dot materials, and perovskite materials; and/or, the filter layer is made of photonic crystal, ultra A filter layer prepared by one or more of surface, random structure, nanostructure, metal surface plasmon SPP micro-nano structure, and tunable Fabry-Perot resonant cavity.
  • the semiconductor material includes one or more of silicon, silicon oxide, silicon nitride, titanium oxide, composite materials mixed in a preset ratio, and direct bandgap compound semiconductor materials; and/or, the nanometer
  • the structure includes one or more of nanodot 2D materials, nanopillar 2D materials, and nanowire 2D materials.
  • the thickness of the optical filter layer is 0.1 ⁇ ⁇ 10 ⁇ , where ⁇ represents the center wavelength of the incident light.
  • the embodiment of the present application also provides a fingerprint identification device, including the above-mentioned optical artificial neural network fingerprint identification chip.
  • the embodiment of the present application also provides a preparation method of an optical artificial neural network fingerprint identification chip as described in any of the above, including:
  • the optical filter layer is used to perform different spectral modulation on the incident light entering different positions of the optical modulation structure through the optical modulation structure, so as to obtain different spectral modulations on the surface of the image sensor.
  • the image sensor converts the information carried by the incident light corresponding to the different position points after being modulated by the optical filter layer to the electrical signals corresponding to the different position points after the first nonlinear activation processing through the square detection response, and will be different from the different position points. sending the electrical signal corresponding to the location point to the processor;
  • the processor performs full connection processing on the electrical signals corresponding to different position points, or the processor performs full connection processing and second nonlinear activation processing on the electrical signals corresponding to different position points to obtain fingerprint identification processing. result.
  • the preparation method of the optical artificial neural network fingerprint identification chip also includes: a training process for the optical artificial neural network fingerprint identification chip, specifically including:
  • training optical artificial neural network fingerprint recognition chips including different optical modulation structures, image sensors and processors with different fully connected parameters.
  • the light modulation structure, image sensor and processor that meet the training convergence condition are trained as the trained light modulation structure, image sensor and processor;
  • the optical artificial neural network fingerprint recognition chip is trained to obtain a light modulation structure, an image sensor and a processor that meet the training convergence conditions, and the light modulation structure, image sensor and processor that meet the training convergence conditions are used as the trained light modulation structure, image sensor and processor.
  • Image sensor and processor is
  • the embodiment of the present application also provides an optical artificial neural network face recognition chip, which is used for face recognition processing tasks, including: an optical filter layer, an image sensor and a processor; the optical filter layer corresponds to the artificial neural network.
  • the input layer, the linear layer, and the connection weight from the input layer to the linear layer, the square detection response of the image sensor corresponds to the first nonlinear activation function in the nonlinear layer of the artificial neural network;
  • the processor corresponds to The full connection and the output layer of the artificial neural network, or, the processor corresponds to the full connection of the artificial neural network, the second nonlinear activation function in the nonlinear layer, and the output layer;
  • the optical filter layer is disposed on the surface of the image sensor, and the optical filter layer includes a light modulation structure, and the light modulation structure is used for the incident light entering into different positions of the light modulation structure.
  • different spectral modulations to obtain incident light-carrying information corresponding to different positions on the surface of the image sensor;
  • the incident light-carrying information includes light intensity distribution information, spectral information, angle information of the incident light, and the Phase information of incident light;
  • the incident light includes reflected light, transmitted light and/or radiated light of the user's face;
  • the image sensor converts the information carried by the incident light corresponding to the different position points after being modulated by the optical filter layer to the electrical signals corresponding to the different position points after the first nonlinear activation processing through the square detection response, and will be different from the different position points. sending the electrical signal corresponding to the location point to the processor;
  • the processor performs full connection processing on electrical signals corresponding to different position points, or the processor performs full connection processing and second nonlinear activation processing on electrical signals corresponding to different position points to obtain face recognition. process result.
  • the optical artificial neural network face recognition chip includes a trained light modulation structure, an image sensor and a processor;
  • the trained light modulation structure, image sensor and processor refers to using the input training samples and output training samples corresponding to the face recognition processing task to perform the training of the light modulation structure, image sensor and the image sensor with different full range.
  • a light modulation structure, an image sensor and a processor that satisfy the training convergence conditions obtained by training an optical artificial neural network face recognition chip connected to a processor with parameters; or, the trained light modulation structure, image sensor and processor are Refers to using the input training samples and output training samples corresponding to the face recognition processing task, to the processor including different light modulation structures, image sensors, and different fully connected parameters and different second nonlinear activation parameters.
  • the optical modulation structure, image sensor and processor that meet the training convergence conditions are obtained by training the optical artificial neural network face recognition chip;
  • the input training samples include incident light reflected, transmitted and/or radiated by different faces; and the output training samples include corresponding face recognition results.
  • the different light modulation structures are designed and realized by adopting computer optical simulation design.
  • the light modulation structures in the optical filter layer include regular structures and/or irregular structures; and/or, the light modulation structures in the optical filter layer include discrete structures and/or continuous structures.
  • the optical filter layer has a single-layer structure or a multi-layer structure.
  • the light modulation structure in the optical filter layer includes a unit array composed of a plurality of micro/nano units, each micro/nano unit corresponds to one or more pixels on the image sensor; the structures of each micro/nano unit are the same or different.
  • micro-nano unit includes a regular structure and/or an irregular structure; and/or, the micro-nano unit includes a discrete structure and/or a continuous structure.
  • micro-nano unit includes a plurality of groups of micro-nano structure arrays, and the structures of each group of micro-nano structure arrays are the same or different.
  • each group of micro-nano structure arrays has the function of broadband filtering or narrow-band filtering.
  • each group of micro-nano structure arrays is a periodic structure array or a non-periodic structure array.
  • micro-nano unit has polarization-independent properties, and in particular, the micro-nano unit has quadruple rotational symmetry.
  • optical filter layer is composed of one or more layers
  • the filter layer is made of one or more of semiconductor materials, metal materials, liquid crystals, quantum dot materials, and perovskite materials; and/or, the filter layer is made of photonic crystals, metasurfaces, random A filter layer prepared by one or more of a structure, a nanostructure, a metal surface plasmon SPP micro-nano structure, and a tunable Fabry-Perot resonant cavity.
  • the semiconductor material includes one or more of silicon, silicon oxide, silicon nitride, titanium oxide, composite materials mixed in a preset ratio, and direct bandgap compound semiconductor materials; and/or, the nanometer
  • the structure includes one or more of nanodot 2D materials, nanopillar 2D materials, and nanowire 2D materials.
  • the thickness of the optical filter layer is 0.1 ⁇ ⁇ 10 ⁇ , where ⁇ represents the center wavelength of the incident light.
  • the embodiment of the present application also provides a face recognition device, including the optical artificial neural network face recognition chip as described above.
  • the embodiment of the present application also provides a preparation method of the optical artificial neural network face recognition chip as described above, including:
  • the optical filter layer is used to perform different spectral modulation on the incident light entering different positions of the optical modulation structure through the optical modulation structure, so as to obtain different spectral modulations on the surface of the image sensor.
  • the image sensor converts the information carried by the incident light corresponding to the different position points after being modulated by the optical filter layer to the electrical signals corresponding to the different position points after the first nonlinear activation processing through the square detection response, and will be different from the different position points. sending the electrical signal corresponding to the location point to the processor;
  • the processor performs full connection processing on electrical signals corresponding to different position points, or the processor performs full connection processing and second nonlinear activation processing on electrical signals corresponding to different position points to obtain face recognition. process result.
  • the preparation method of the optical artificial neural network face recognition chip also includes: a training process for the optical artificial neural network face recognition chip, specifically including:
  • the optical artificial neural network face recognition chips including different light modulation structures, image sensors and processors with different fully connected parameters are trained Obtain the light modulation structure, image sensor and processor that satisfy the training convergence condition, and use the light modulation structure, image sensor and processor that satisfy the training convergence condition as the trained light modulation structure, image sensor and processor;
  • the optical artificial neural network face recognition chip of the device is trained to obtain a light modulation structure, an image sensor and a processor that meet the training convergence conditions, and the light modulation structure, image sensor and processor that meet the training convergence conditions are used as the trained light modulation structure. Structures, Image Sensors and Processors.
  • the different light modulation structures are designed and realized by adopting computer optical simulation design.
  • the embodiment of the present application also provides an optical artificial neural network machine vision enhancement chip, which is used for machine vision intelligent processing tasks, including: an optical filter layer, an image sensor and a processor; the optical filter layer corresponds to the artificial neural network.
  • the input layer, the linear layer, and the connection weight from the input layer to the linear layer, the square detection response of the image sensor corresponds to the first nonlinear activation function in the nonlinear layer of the artificial neural network;
  • the processor corresponds to The full connection and the output layer of the artificial neural network, or, the processor corresponds to the full connection of the artificial neural network, the second nonlinear activation function in the nonlinear layer, and the output layer;
  • the optical filter layer is disposed on the surface of the image sensor, and the optical filter layer includes a light modulation structure, and the light modulation structure is used for the incident light entering into different positions of the light modulation structure.
  • different spectral modulations to obtain incident light-carrying information corresponding to different positions on the surface of the image sensor;
  • the incident light-carrying information includes light intensity distribution information, spectral information, angle information of the incident light, and the Phase information of incident light;
  • the incident light includes reflected light, transmitted light and/or radiated light of a target object in a machine vision scene;
  • the image sensor converts the information carried by the incident light corresponding to the different position points after being modulated by the optical filter layer to the electrical signals corresponding to the different position points after the first nonlinear activation processing through the square detection response, and will be different from the different position points. sending the electrical signal corresponding to the location point to the processor;
  • the processor performs full connection processing on the electrical signals corresponding to different position points, or the processor performs full connection processing and second nonlinear activation processing on the electrical signals corresponding to different position points to obtain machine vision intelligence. process result;
  • the machine vision intelligent processing task includes the recognition and/or qualitative analysis of the target object in the machine vision scene; the machine vision intelligent processing result includes the recognition result and/or the qualitative analysis result of the target object in the machine vision scene.
  • the optical artificial neural network machine vision enhancement chip includes a trained light modulation structure, an image sensor and a processor;
  • the trained light modulation structures, image sensors and processors refer to using the input training samples and output training samples corresponding to the machine vision intelligent processing tasks to perform data analysis on the light modulation structures, image sensors and different full-scale light modulation
  • a light modulation structure, an image sensor, and a processor that meet the training convergence conditions obtained by training an optical artificial neural network machine vision enhancement chip connected to a parameter processor; or, the trained light modulation structure, image sensor, and processor are Refers to the use of input training samples and output training samples corresponding to the machine vision intelligent processing task, to include different light modulation structures, image sensors and processors with different fully connected parameters and different second nonlinear activation parameters.
  • the optical modulation structure, image sensor and processor that meet the training convergence conditions are obtained by training the optical artificial neural network machine vision enhancement chip;
  • the input training sample includes incident light reflected, transmitted and/or radiated by a target object in a specific machine vision scene
  • the output training sample includes a target object recognition result in a specific machine vision scene
  • all The input training samples include incident light reflected, transmitted and/or radiated by a target object in a specific machine vision scene
  • the output training samples include qualitative analysis results of the target object in a specific machine vision scene.
  • the different light modulation structures are designed and realized by adopting computer optical simulation design.
  • the light modulation structures in the optical filter layer include regular structures and/or irregular structures; and/or, the light modulation structures in the optical filter layer include discrete structures and/or continuous structures.
  • the optical filter layer has a single-layer structure or a multi-layer structure.
  • the light modulation structure in the optical filter layer includes a unit array composed of a plurality of micro/nano units, each micro/nano unit corresponds to one or more pixels on the image sensor; the structures of each micro/nano unit are the same or different.
  • micro-nano unit includes a regular structure and/or an irregular structure; and/or, the micro-nano unit includes a discrete structure and/or a continuous structure.
  • micro-nano unit includes a plurality of groups of micro-nano structure arrays, and the structures of each group of micro-nano structure arrays are the same or different.
  • each group of micro-nano structure arrays has the function of broadband filtering or narrow-band filtering.
  • each group of micro-nano structure arrays is a periodic structure array or a non-periodic structure array.
  • micro-nano unit has polarization independent properties.
  • micro-nano unit has quadruple rotational symmetry.
  • the filter layer is made of one or more of semiconductor materials, metal materials, liquid crystals, quantum dot materials, and perovskite materials; and/or, the filter layer is made of photonic crystal, ultra A filter layer prepared by one or more of surface, random structure, nanostructure, metal surface plasmon SPP micro-nano structure, and tunable Fabry-Perot resonant cavity.
  • the semiconductor material includes one or more of silicon, silicon oxide, silicon nitride, titanium oxide, composite materials mixed in a preset ratio, and direct bandgap compound semiconductor materials; and/or, the nanometer
  • the structure includes one or more of nanodot 2D materials, nanopillar 2D materials, and nanowire 2D materials.
  • the thickness of the optical filter layer is 0.1 ⁇ ⁇ 10 ⁇ , where ⁇ represents the center wavelength of the incident light.
  • Embodiments of the present application also provide an enhanced machine vision system, including a control mechanism and the optical artificial neural network machine vision enhancement chip as described above.
  • the embodiment of the present application also provides a preparation method of the optical artificial neural network machine vision enhancement chip as described in any of the above, including:
  • the optical filter layer is used to perform different spectral modulation on the incident light entering different positions of the optical modulation structure through the optical modulation structure, so as to obtain different spectral modulations on the surface of the image sensor.
  • the image sensor converts the information carried by the incident light corresponding to the different position points after being modulated by the optical filter layer to the electrical signals corresponding to the different position points after the first nonlinear activation processing through the square detection response, and will be different from the different position points. sending the electrical signal corresponding to the location point to the processor;
  • the processor performs full connection processing on the electrical signals corresponding to different position points, or the processor performs full connection processing and second nonlinear activation processing on the electrical signals corresponding to different position points to obtain machine vision intelligence. process result.
  • the preparation method of the optical artificial neural network machine vision enhancement chip also includes: the training process of the optical artificial neural network machine vision enhancement chip, specifically including:
  • train the optical artificial neural network machine vision enhancement chip including different light modulation structures, image sensors and processors with different fully connected parameters Obtain the light modulation structure, image sensor and processor that satisfy the training convergence condition, and use the light modulation structure, image sensor and processor that satisfy the training convergence condition as the trained light modulation structure, image sensor and processor;
  • the optical artificial neural network machine vision enhancement chip of the device is trained to obtain a light modulation structure, an image sensor and a processor that meet the training convergence conditions, and the light modulation structure, image sensor and processor that meet the training convergence conditions are used as the trained light modulation structure. Structures, Image Sensors and Processors.
  • the different light modulation structures are designed and realized by adopting computer optical simulation design.
  • the optical artificial neural network smart chip and preparation method realize a brand-new smart chip capable of realizing the artificial neural network function.
  • the optical filter layer corresponds to the input layer and linearity of the artificial neural network.
  • the image sensor corresponds to a part of the nonlinear layer of the artificial neural network;
  • the processor corresponds to another part of the nonlinear layer of the artificial neural network and the output layer.
  • the optical filter layer is disposed on the surface of the image sensor, and the optical filter layer includes a light modulation structure, and the light modulation structure is used to perform different spectral modulations on the incident light entering different positions of the light modulation structure, so as to Information carried by the incident light corresponding to different positions is obtained on the surface of the image sensor.
  • the modulation effect of the light modulation structure on the optical filter layer on the incident light is equivalent to the connection between the input layer and the linear layer. Weights.
  • the image sensor converts the incident light-carrying information corresponding to the different position points after being modulated by the optical filter layer for the first time, and then converts it into the information corresponding to the different position points after the first nonlinear activation processing through the square detection response.
  • the electrical signals corresponding to different positions are sent to the processor, and the processor performs full connection processing on the electrical signals corresponding to different positions, or the processor sends the electrical signals corresponding to different positions.
  • the full connection processing and the second nonlinear activation processing are performed to obtain the output signal of the artificial neural network.
  • the optical filter layer corresponds to the input layer, linear layer and the artificial neural network.
  • the connection weight from the input layer to the linear layer, the square detection response of the image sensor corresponds to the first nonlinear activation function in the nonlinear layer of the artificial neural network;
  • the processor corresponds to the full connection of the artificial neural network and the output layer, or, the processor corresponds to the full connection of the artificial neural network, the second nonlinear activation function in the nonlinear layer, and the output layer, that is, the optical filter layer and the image sensor in the smart chip realize the artificial neural network.
  • the related functions of the input layer, the linear layer and the partial nonlinear activation function in the network that is, the input layer, the linear layer and some or all of the nonlinear activation functions in the artificial neural network implemented by software in the prior art in the embodiments of the present application
  • the function is stripped, and the input layer, linear layer and some or all of the nonlinear activation functions in the artificial neural network are realized by hardware, so that the subsequent use of the intelligent chip for artificial neural network intelligent processing does not require Then perform complex signal processing and algorithm processing corresponding to the input layer, linear layer, and some or all of the nonlinear activation functions. Only the processor in the smart chip needs to perform full connection processing or full connection with the electrical signal and the second non-linear activation function.
  • Linear activation processing is enough, which can greatly reduce the power consumption and delay of artificial neural network processing. It can be seen that, in the embodiment of the present application, the optical filter layer is used as the input layer, the linear layer, and the connection weight between the input layer and the linear layer, and the square detection response of the image sensor is used as the non-linear layer of the artificial neural network.
  • the first nonlinear activation function in the linear layer; the processor is used as the fully connected and output layer of the artificial neural network, or, the processor corresponds to the second nonlinear activation in the fully connected, nonlinear layer of the artificial neural network function and output layer, it can be seen that the embodiment of the present application can not only save the complex signal processing and algorithm processing corresponding to the input layer, the linear layer and some nonlinear activation functions in the prior art, but also the embodiment of the present application actually At the same time, the image information, spectral information, angle of incident light and phase information of incident light of the target object are used, that is, the information carried by the incident light at different points in the target object space.
  • the carrying information covers the image, composition, shape, three-dimensional depth, structure and other information of the target object, so that when the identification processing is carried out according to the information carried by the incident light at different points in the target object space, the image, composition, shape, Multi-dimensional information such as three-dimensional depth, structure, etc., can solve the problem that the two-dimensional image information of the target object mentioned in the background technology is difficult to ensure the accuracy of recognition, for example, it is difficult to distinguish whether it is a real person or a picture.
  • the optical artificial neural network chip provided by the application embodiment can not only achieve the effect of low power consumption and low delay, but also achieve the effect of high accuracy, so that it can be applied to intelligent processing tasks such as intelligent perception, recognition and/or decision-making. be ready.
  • FIG. 1 is a schematic structural diagram of an optical artificial neural network smart chip provided in a first embodiment of the present application
  • FIG. 2 is a schematic diagram of the identification principle of an optical artificial neural network intelligent chip provided by an embodiment of the present application
  • FIG. 3 is a schematic diagram of disassembly of an optical artificial neural network smart chip provided by an embodiment of the present application
  • FIG. 4 is a schematic diagram of a target object recognition process provided by an embodiment of the present application.
  • FIG. 5 is a top view of an optical filter layer provided by an embodiment of the present application.
  • FIG. 6 is a top view of another optical filter layer provided by an embodiment of the present application.
  • FIG. 7 is a top view of another optical filter layer provided by an embodiment of the present application.
  • FIG. 8 is a top view of another optical filter layer provided by an embodiment of the present application.
  • FIG. 9 is a top view of yet another optical filter layer provided by an embodiment of the present application.
  • FIG. 10 is a top view of yet another optical filter layer provided by an embodiment of the present application.
  • FIG. 11 is a schematic diagram of a broadband filtering effect of a micro-nano structure provided by an embodiment of the present application.
  • FIG. 12 is a schematic diagram of a narrow-band filtering effect of a micro-nano structure provided by an embodiment of the present application.
  • FIG. 13 is a schematic structural diagram of a front-illuminated image sensor provided by an embodiment of the present application.
  • FIG. 14 is a schematic structural diagram of a back-illuminated image sensor provided by an embodiment of the present application.
  • 15 is a schematic flowchart of a method for preparing an optical artificial neural network smart chip provided by the third embodiment of the present application.
  • FIG. 16 is a schematic diagram of a pollutant sample identification process provided by an embodiment of the present application.
  • FIG. 17 is a schematic diagram of a fingerprint identification process provided by an embodiment of the present application.
  • FIG. 18 is a schematic diagram of a face recognition process provided by an embodiment of the present application.
  • FIG. 19 is a schematic diagram of a machine vision enhanced recognition process provided by an embodiment of the present application.
  • the existing intelligent recognition technology usually needs to image a person or object first, and then go through the steps of image preprocessing, feature extraction, and feature matching to realize the recognition of the person or object.
  • image preprocessing image preprocessing
  • feature extraction feature matching
  • feature matching feature matching
  • an embodiment of the present application provides an optical artificial neural network smart chip, wherein the optical filter layer in the smart chip corresponds to the input layer, the linear layer of the artificial neural network, and the connection weight between the input layer and the linear layer,
  • the square detection response of the image sensor corresponds to the first nonlinear activation function in the nonlinear layer of the artificial neural network;
  • the processor corresponds to the full connection and the output layer of the artificial neural network, or the processor corresponds to the full connection of the artificial neural network , the second nonlinear activation function in the nonlinear layer and the output layer
  • the embodiment of the present application uses the optical filter layer and the image sensor to project the spatial spectral information of the target object into an electrical signal, and then realizes the electrical signal in the processor.
  • the embodiments of the present application can not only save the complex and complicated processes corresponding to the input layer, the linear layer and some or all of the nonlinear activation functions in the prior art Signal processing and algorithm processing, and the embodiment of the present application actually uses the image information, spectral information, angle of incident light and phase information of incident light of the target object simultaneously, that is, the incident light at different points in the target object space carries information, which is represented by It can be seen that since the information carried by the incident light at different points in the target object space covers the image, composition, shape, three-dimensional depth, structure and other information of the target object, the identification processing is performed according to the information carried by the incident light at different points in the target object space.
  • the network chip can not only achieve the effect of low power consumption and low latency, but also improve the accuracy of intelligent processing, so that it can be better applied in intelligent processing fields such as intelligent perception, recognition and/or decision-making.
  • intelligent processing such as intelligent recognition
  • the content provided by the present application will be explained and illustrated in detail below through specific embodiments.
  • the optical artificial neural network smart chip provided by the first embodiment of the present application includes: an optical filter layer 1, an image sensor 2 and a processor 3;
  • the optical filter layer 1 corresponds to the artificial neural network
  • the input layer, the linear layer and the connection weight from the input layer to the linear layer, the square detection response of the image sensor 2 corresponds to the first nonlinear activation function in the nonlinear layer of the artificial neural network;
  • the processor 3 corresponds to the full connection and the output layer of the artificial neural network, or, the processor corresponds to the full connection of the artificial neural network, the second nonlinear activation function in the nonlinear layer, and the output layer;
  • the optical filter layer 1 is disposed on the surface of the image sensor or the surface of the photosensitive area of the image sensor, the optical filter layer 1 includes a light modulation structure, and the optical filter layer 1 is used to pass the light
  • the modulation structure performs spectral modulation on the incident light entering different positions of the light modulation structure with intensity modulation as the wavelength changes respectively, that is, performs different intensity modulation on the incident light with different wavelengths, so as to obtain on the surface of the image sensor Incident light-carrying information corresponding to different position points;
  • the incident light-carrying information includes light intensity distribution information, spectral information, angle information of the incident light, and phase information of the incident light;
  • the square detection response of the image sensor 2 refers to the intensity information of the incident light field detected by the image sensor, and the intensity information of the incident light field is the square of the modulo of the light field signal, that is, the The image sensor 2 converts the information carried by the incident light corresponding to the different position points after being modulated by the optical filter layer 1 to the electrical signal corresponding to the different position points after the first nonlinear activation processing through the square detection response, and will be different from the different position points.
  • the electrical signal corresponding to the position point is sent to the processor 3; the electrical signal is an image signal modulated by the optical filter layer;
  • the processor 3 is used to perform full connection processing on the electrical signals corresponding to different position points, or the processor performs full connection processing and second nonlinear activation processing on the electrical signals corresponding to different position points to obtain artificial The output signal of the neural network.
  • the optical filter layer 1 is disposed on the surface of the image sensor, the optical filter layer 1 includes a light modulation structure, and the optical filter layer 1 is used to enter different positions of the light modulation structure through the pair of light modulation structures
  • the incident light of the image sensor is subjected to different spectral modulation respectively, so as to obtain the information carried by the modulated incident light corresponding to different position points on the surface of the image sensor.
  • the modulation effect of the light modulation structure on the optical filter layer on the incident light can be regarded as the connection weight between the input layer and the linear layer;
  • the image sensor 2 when the image sensor 2 performs photoelectric conversion on the information carried by the modulated incident light, since the image sensor 2 can detect the intensity information of the light, the electrical signal obtained by processing the light field distribution signal is proportional to the light field Therefore, the image sensor 2 has a square detection response, so the image sensor 2 can be regarded as a part of the nonlinear layer of the artificial neural network, that is, the square detection response of the image sensor 2 can be regarded as an artificial neural network.
  • the first nonlinear activation function of the network is the image sensor 2 performs photoelectric conversion on the information carried by the modulated incident light.
  • the image sensor 2 converts the incident light-carrying information corresponding to the different position points after being modulated by the optical filter layer 1 through the first nonlinear activation process into electrical signals corresponding to the different position points through the square detection response.
  • the signal that is, the image signal modulated by the optical filter layer
  • the processor 3 connected to the image sensor 2 is used to perform full connection processing or full connection and second Non-linear activation processing to obtain the output signal of the artificial neural network.
  • the optical filter layer 1 includes a light modulation structure, and the light modulation structure correlates the incident light (such as reflected light, transmitted light, radiated light, etc. of the target to be identified) entering different positions of the light modulation structure.
  • Acting light performs spectral modulation with different intensities to obtain information carried by incident light corresponding to different positions on the surface of the image sensor 2 .
  • the modulation intensity is related to the specific structural form of the light modulation structure.
  • different light modulation structures can be designed (eg, by changing the shape and/or size parameters of the light modulation structure) to achieve different modulation strength.
  • the light modulation structures at different positions on the optical filter layer 1 have different spectral modulation effects on the incident light, and the modulation intensity of the light modulation structures on different wavelength components of the incident light corresponds to the artificial nerve
  • the connection strength of the linear layer of the network that is, the connection weight corresponding to the input layer and the input layer to the linear layer.
  • the optical filter layer 1 is composed of a plurality of optical filter units, and the optical modulation structure at different positions in each optical filter unit is different, so it has different spectral modulation effects on incident light;
  • the light modulation structures at different positions between the optical filter units may be the same or different, and thus have the same or different spectral modulation effects on the incident light.
  • the image sensor 2 converts the incident light-carrying information corresponding to the different position points after being modulated by the optical filter layer 1 through the first nonlinear activation process into electrical signals corresponding to the different position points through the square detection response. signal, and send electrical signals corresponding to different position points to the processor 3, and the image sensor 2 corresponds to a part of the nonlinear layer of the neural network.
  • the processor 3 performs full connection processing on the electrical signals at different positions or, the processor 3 performs full connection processing and a second nonlinear activation processing on the electrical signals at different positions, thereby obtaining an artificial neural network. output signal.
  • the image sensor 2 corresponds to a part of the nonlinear layer of the neural network
  • the processor 3 corresponds to another part of the nonlinear layer and the output layer of the neural network.
  • the square detection response of the image sensor 2 corresponds to the first nonlinear activation function in the nonlinear layer of the neural network.
  • the processor may only perform full connection processing. , the second nonlinear activation processing is no longer performed, or both the full connection processing and the second nonlinear activation processing are performed in the processor. Specifically, it can be determined according to the actual application scenario of the chip, which is not limited in this embodiment.
  • the processor 3 may be disposed in the smart chip, that is, the processor 3 may be disposed in the smart chip together with the filter layer 1 and the image sensor 2, or may be separately It is arranged outside the smart chip, and is connected to the image sensor 2 in the smart chip through a data line or a connecting device, which is not limited in this embodiment.
  • the processor 3 can be implemented by a computer, or by an ARM or FPGA circuit board with a certain computing capability, or by a microprocessor, which is not limited in this embodiment.
  • the processor 3 may be integrated in the smart chip, or may be provided independently of the smart chip. When the processor 3 is set independently of the smart chip, the electrical signal in the image sensor 2 can be read out to the processor 3 through the signal readout circuit, and then the processor 3 can read out the electrical signal. Full connection processing and nonlinear activation processing.
  • a nonlinear activation function can be used to implement, for example, a Sigmoid function, a Tanh function, a ReLU function, etc. This is not limited.
  • the optical filter layer 1 corresponds to the input layer, the linear layer and the connection weight between the input layer and the linear layer of the artificial neural network
  • the image sensor 2 corresponds to a part of the nonlinear layer of the artificial neural network, that is, the image sensor
  • the square detection response of 2 corresponds to the first nonlinear activation function of the artificial neural network
  • the image sensor 2 is used to perform nonlinear activation processing on the information carried by the incident light at different positions in space through the square detection response, and then convert it into an electrical signal
  • the processor 3 Corresponding to the remaining layers of the artificial neural network, the electrical signals at different positions are fully connected, and the output signal of the artificial neural network can be obtained through the second nonlinear activation function, so as to realize the intelligent perception, recognition and/or decision making.
  • the optical artificial neural network smart chip includes an optical filter layer 1, an image sensor 2 and a processor 3.
  • the processor 3 is implemented by a signal readout circuit and a computer.
  • the optical filter layer 1 in the optical artificial neural network smart chip corresponds to the input layer and the linear layer of the artificial neural network
  • the image sensor 2 corresponds to a part of the nonlinear layer of the artificial neural network
  • the processor 3 corresponds to The other part of the nonlinear layer and the output layer of the artificial neural network
  • the filtering effect of the optical filter layer 1 on the incident light entering the optical filter layer 1 corresponds to the connection weight of the input layer to the linear layer
  • the square detection response of the image sensor 2 corresponds to The first nonlinear activation function of the artificial neural network
  • the related functions of the nonlinear activation function so that the complex signal processing and algorithm processing corresponding to the input layer and the linear layer do not need to be performed in the subsequent intelligent processing using the smart chip (for example, the connection between the input layer and the linear layer is omitted. weights and other calculations), which can greatly reduce the power consumption and delay of artificial neural network processing.
  • this embodiment simultaneously utilizes the image information, spectral information, angle of incident light and phase information of the incident light of the target object, the intelligent processing of the target object can be realized more accurately.
  • the optical filter layer 1 has different broadband spectrum modulation effects on the incident light, and projects/connects the incident light spectrum P_ ⁇ at the corresponding unit position to the outgoing light field E_N; the square of the image sensor 2
  • the detection response corresponds to the nonlinear activation function of the optical artificial neural network, which converts the outgoing light field E_N of the optical filter layer 1 to the photocurrent response I_N of the image sensor.
  • the processor 3 includes a signal readout circuit and a computer.
  • the signal readout circuit in the processor 3 reads out the photocurrent response I_N and transmits it to the computer, and the computer performs full connection processing of the electrical signal or non-linear activation processing again. The final output result.
  • the light modulation structure on the optical filter layer 1 is integrated on the surface of the image sensor 2, modulates the incident light, and projects/connects the spectral information of the incident light to different pixels of the image sensor 2 to obtain
  • the electrical signal that the incident light carries information that is, after the incident light passes through the optical filter layer 1, is nonlinearly activated by the square detection response of the image sensor 2 and then converted into an electrical signal to form an image containing the spectral information of the incident light.
  • the processor 3 connected to the image sensor 2 processes the electrical signal including the spectral information and image information of the incident light, and then obtains an output result.
  • the optical artificial neural network chip provided in this embodiment actually utilizes the image information, spectral information, angle of incident light and phase information of incident light of the target object at the same time, that is, incident light at different points in space carries information
  • the artificial neural network is embedded in the hardware, and the material composition, image shape, three-dimensional depth and other information can be further extracted from the spatial image, spectrum, angle, and phase information, so as to solve the second problem of using the target object mentioned in the background technology section.
  • Dimensional image information is difficult to ensure the accuracy of recognition. For example, it is difficult to distinguish whether it is a real person or a picture. It realizes intelligent perception, recognition and/or decision-making functions for different application fields, and achieves low power consumption, low latency and high accuracy.
  • Spectrum light artificial neural network smart chip is difficult to ensure the accuracy of recognition. For example, it is difficult to distinguish whether it is a real person or a picture. It realizes intelligent perception, recognition and/or decision-making functions for different application fields, and achieves low power consumption, low latency and high
  • the optical filter layer corresponds to the input layer and the linear layer of the artificial neural network
  • the image sensor corresponds to a part of the nonlinear layer of the artificial neural network
  • the processing The device corresponds to another part of the nonlinear layer of the artificial neural network and the output layer.
  • the optical filter layer is disposed on the surface of the image sensor, and the optical filter layer includes a light modulation structure, and the light modulation structure is used to perform different spectral modulations on the incident light entering different positions of the light modulation structure, so as to Information carried by the incident light corresponding to different positions is obtained on the surface of the image sensor.
  • the modulation effect of the light modulation structure on the optical filter layer on the incident light is equivalent to the connection between the input layer and the linear layer. Weights.
  • the image sensor converts the incident light-carrying information corresponding to the different position points after being modulated by the optical filter layer for the first time, and then converts it into the information corresponding to the different position points after the first nonlinear activation processing through the square detection response.
  • the electrical signals corresponding to different positions are sent to the processor, and the processor performs full connection processing on the electrical signals corresponding to different positions, or the processor sends the electrical signals corresponding to different positions.
  • the full connection processing and the second nonlinear activation processing are performed to obtain the output signal of the artificial neural network.
  • the optical filter layer corresponds to the input layer, linear layer and the artificial neural network.
  • the connection weight from the input layer to the linear layer, the square detection response of the image sensor corresponds to the first nonlinear activation function in the nonlinear layer of the artificial neural network;
  • the processor corresponds to the full connection of the artificial neural network and the output layer, or, the processor corresponds to the full connection of the artificial neural network, the second nonlinear activation function in the nonlinear layer, and the output layer, that is, the optical filter layer and the image sensor in the smart chip realize the artificial neural network.
  • the related functions of the input layer, the linear layer and the partial nonlinear activation function in the network that is, the input layer, the linear layer and some or all of the nonlinear activation functions in the artificial neural network implemented by software in the prior art in the embodiments of the present application
  • the function is stripped, and the input layer, linear layer and some or all of the nonlinear activation functions in the artificial neural network are realized by hardware, so that the subsequent use of the intelligent chip for artificial neural network intelligent processing does not require Then perform complex signal processing and algorithm processing corresponding to the input layer, linear layer, and some or all of the nonlinear activation functions. Only the processor in the smart chip needs to perform full connection processing or full connection with the electrical signal and the second non-linear activation function.
  • Linear activation processing is enough, which can greatly reduce the power consumption and delay of artificial neural network processing. It can be seen that, in the embodiment of the present application, the optical filter layer is used as the input layer, the linear layer, and the connection weight between the input layer and the linear layer, and the square detection response of the image sensor is used as the non-linear layer of the artificial neural network.
  • the first nonlinear activation function in the linear layer; the processor is used as the fully connected and output layer of the artificial neural network, or, the processor corresponds to the second nonlinear activation in the fully connected, nonlinear layer of the artificial neural network function and output layer, it can be seen that the embodiment of the present application can not only save the complex signal processing and algorithm processing corresponding to the input layer, the linear layer and some nonlinear activation functions in the prior art, but also the embodiment of the present application actually At the same time, the image information, spectral information, angle of incident light and phase information of incident light of the target object are used, that is, the information carried by the incident light at different points in the target object space.
  • the carrying information covers the image, composition, shape, three-dimensional depth, structure and other information of the target object, so that when the identification processing is carried out according to the incident light carrying information at different points in the target object space, the image, composition, shape, Multi-dimensional information such as three-dimensional depth, structure, etc., can solve the problem that the two-dimensional image information of the target object mentioned in the background technology is difficult to ensure the accuracy of recognition, for example, it is difficult to distinguish whether it is a real person or a picture.
  • the optical artificial neural network chip provided by the application embodiment can not only achieve the effect of low power consumption and low delay, but also achieve the effect of high accuracy, so that it can be applied to intelligent processing tasks such as intelligent perception, recognition and/or decision-making. be ready.
  • the optical artificial neural network intelligent chip is used for the intelligent processing task of the target object;
  • the intelligent processing task includes at least one of intelligent perception, intelligent recognition and intelligent decision-making tasks or more;
  • the reflected light, transmitted light and/or radiated light of the target object enters the trained optical artificial neural network smart chip, and the intelligent processing result of the target object is obtained;
  • the intelligent processing result at least includes the intelligent perception result and the intelligent recognition result and/or one or more of intelligent decision-making outcomes;
  • the trained optical artificial neural network intelligent chip refers to the optical artificial neural network intelligent chip including the trained optical modulation structure, image sensor and processor; the trained optical modulation structure, image sensor and processor refers to the use of The input training samples and output training samples corresponding to the intelligent processing task are obtained by training optical artificial neural network intelligent chips including different optical modulation structures, image sensors and processors with different fully connected parameters, and the training convergence conditions are satisfied. or, the trained light modulation structure, image sensor and processor refers to using the input training samples and output training samples corresponding to the intelligent processing tasks to Light modulation structure, image sensor, and optical artificial neural network smart chip with different fully connected parameters and different second nonlinear activation parameters. The light modulation structure, image sensor and processing that satisfy the training convergence conditions obtained by training device.
  • the input training sample includes incident light reflected, transmitted and/or radiated by the target object in the corresponding intelligent processing task;
  • the output training sample includes intelligent processing results (such as recognition results, perception results, decision results or qualitative analysis results, etc.).
  • the optical artificial neural network intelligent chip can be used for intelligent processing tasks of the target object, for example, including one or more tasks of intelligent perception, intelligent recognition and intelligent decision-making tasks.
  • intelligent perception refers to mapping the signals of the physical world to the digital world through hardware devices such as cameras, microphones or other sensors, and using cutting-edge technologies such as speech recognition and image recognition, and then converting these digital signals into the digital world.
  • Information is further advanced to the level of recognition, such as memory, understanding, planning, decision-making, and so on.
  • Intelligent recognition refers to the technology that uses computers to process, analyze and understand images to identify targets and objects in different modes.
  • intelligent recognition technology is generally divided into face recognition and commodity recognition. Face recognition is mainly used in security inspections. , identity verification and mobile payment, commodity identification is mainly used in the process of commodity circulation, especially in unmanned retail areas such as unmanned shelves and smart retail cabinets.
  • Intelligent decision-making refers to the solution of automatic organization and coordination of multi-model operation by computer, access and processing of data in a large number of databases, and corresponding data processing and numerical calculation.
  • the reflected light, transmitted light and/or radiated light of the target object enters into the trained optical artificial neural network intelligent chip, and the intelligent processing result of the target object is obtained.
  • the recognition task of the target object is used as an example for description. It can be understood that when using the smart chip to perform the recognition task, the optical artificial neural network smart chip needs to be trained first. Smart chip training refers to determining the light modulation structure suitable for the current recognition task through training, as well as the fully connected parameters and nonlinear activation parameters suitable for the current recognition task.
  • the filtering effect of the optical filter layer on the incident light entering the optical filter layer corresponds to the connection weight between the input layer of the artificial neural network and the linear layer
  • the light modulation in the optical filter layer is changed.
  • the structure is equivalent to changing the connection weight from the input layer of the artificial neural network to the linear layer.
  • the smart chip can be used to perform the recognition task. Specifically, after the incident light carrying the image information of the target object and the spatial spectrum information enters the optical filter layer 1 of the trained smart chip, the light modulation structure in the optical filter layer 1 modulates the incident light, and the modulated optical filter layer 1 modulates the incident light.
  • the optical signal intensity is detected by the image sensor 2 and converted into an electrical signal, and then the processor 3 performs full connection processing or simultaneous full connection and second nonlinear activation processing to obtain the recognition result of the target object.
  • the complete process for target object recognition is: the broad-spectrum light source 100 illuminates the target object 200, and then the reflected light or transmitted light of the target object is collected by the optical artificial neural network smart chip 300, or the target object directly
  • the externally radiated light is collected by the optical artificial neural network smart chip 300 and processed by the optical filter layer, the image sensor and the processor in the smart chip, and then the recognition result can be obtained.
  • the trained optical artificial neural network smart chip refers to the optical artificial neural network smart chip including the trained optical modulation structure, image sensor and processor; the trained optical modulation structure, image sensor and processor refers to Using the input training samples and output training samples corresponding to the intelligent processing tasks, the optical artificial A light modulation structure, an image sensor and a processor that meet the training convergence conditions are obtained by training a neural network smart chip.
  • the input training sample corresponding to the intelligent recognition task is the recognition object sample
  • the output training sample corresponding to the intelligent recognition task is the recognition result of the recognition object sample.
  • the advantage of the smart chip provided in this embodiment is that image information, spectral information, angle information of incident light, and phase information of incident light at different points in the space of the recognition object can be obtained. Therefore, in order to take full advantage of this advantage, the real recognition object is preferentially used for the recognition object sample as the input training sample, rather than the two-dimensional image of the recognition object. Of course, this does not mean that two-dimensional images cannot be used as recognition object samples.
  • optical artificial neural network intelligent chip provided in this embodiment can also be used for other intelligent processing tasks of the target object, such as tasks such as intelligent perception and intelligent decision-making.
  • the optical filter layer 1 is used as the input layer and the linear layer of the neural network
  • the image sensor 2 is used as a part of the nonlinear layer of the neural network (that is, the square detection response of the image sensor 2 is used as the first part of the neural network.
  • the modulation intensity of the light modulation structure in the optical filter layer to the different wavelength components in the incident light of the target object is used as the connection weight from the input layer of the neural network to the linear layer.
  • the light modulation structure in this embodiment is obtained based on neural network training, and the optical simulation of the training sample is performed by a computer to obtain the sample modulation intensity of the light modulation structure in the training sample on the different wavelength components of the incident light of the target object in the intelligent processing task , take the sample modulation intensity as the connection weight from the input layer of the neural network to the linear layer, perform nonlinear activation, and use the training samples corresponding to the intelligent processing tasks to train the neural network until the neural network converges.
  • the structure acts as an optical filter layer corresponding to intelligent processing tasks.
  • the input layer and the linear layer (optical filter layer) of the neural network and a part of the nonlinear layer (the squared detection response of the image sensor 2) are realized as the first nonlinear activation of the neural network in the physical layer. function), not only the complex signal processing and algorithm processing corresponding to the input layer, the linear layer and some or all of the nonlinear activation functions in the prior art can be omitted.
  • the embodiment of the present application actually utilizes image information, spectral information, angle information of incident light, and phase information of incident light at different points in the target object space at the same time, that is, the information carried by incident light at different points in the target object space.
  • the information carried by the incident light at different points in the target object space covers the image, composition, shape, three-dimensional depth, structure and other information of the target object
  • the identification processing is performed according to the information carried by the incident light at different points in the target object space , which can cover the image, composition, shape, three-dimensional depth, structure and other multi-dimensional information of the target object, so as to solve the problem that the two-dimensional image information of the target object mentioned in the background technology is difficult to ensure the accuracy of recognition.
  • the optical artificial neural network chip provided in the embodiment of the present application can not only achieve the effect of low power consumption and low delay, but also achieve the effect of high accuracy, so that it can be applied to Prepare for intelligent processing tasks such as intellisense, recognition and/or decision making.
  • the optical artificial neural network smart chips including different light modulation structures, image sensors and processors with different fully connected parameters are trained, or, Light modulation structures, image sensors and optical artificial neural network smart chips with different fully-connected parameters and different second nonlinear activation parameters processors, the different light modulation structures are designed by using computer optical simulations when trained way to design and implement.
  • optical simulation allows the user to experience the product through a digital environment before making a physical prototype.
  • a suitable optical simulation solution can not only effectively help users improve design efficiency, but also simulate the interaction of light and materials in order to understand how the product will display under real conditions. Therefore, in this embodiment, the optical modulation structure is designed through computer optical simulation, and the optical modulation structure is adjusted through optical simulation until the neural network converges and the corresponding optical modulation structure is determined to be the final size of the optical modulation structure to be fabricated, which saves prototyping time and cost. cost, improve product efficiency, and easily solve complex optical problems.
  • the FDTD software can be used to simulate the design of the light modulation structure, and the light modulation structure can be changed in the optical simulation, so that the modulation intensity of the light modulation structure for different incident light can be accurately predicted, and it can be used as the input layer of the neural network.
  • the connection weight of the layer is used to train the optical artificial neural network smart chip to accurately obtain the optical modulation structure.
  • the light modulation structure in the optical filter layer includes a regular structure and/or an irregular structure; and/or, the light modulation structure in the optical filter layer includes Discrete structures and/or continuous structures.
  • the light modulation structure in the optical filter layer may only include regular structures, may only include irregular structures, or may include both regular structures and irregular structures.
  • the light modulation structure includes a regular structure may refer to: the minimum modulation unit included in the light modulation structure is a regular structure, for example, the minimum modulation unit may have regular shapes such as a rectangle, a square, and a circle.
  • the light modulation structure includes a regular structure can also refer to: the arrangement of the minimum modulation units included in the light modulation structure is regular, for example, the arrangement can be a regular array form, a circular form, a trapezoidal form, a polygonal form, etc.
  • the light modulation structure includes a regular structure may also refer to: the minimum modulation unit included in the light modulation structure is a regular structure, and the arrangement of the minimum modulation units is also regular.
  • the light modulation structure here includes an irregular structure may refer to: the minimum modulation unit included in the light modulation structure is an irregular structure, for example, the minimum modulation unit may be irregular shapes such as irregular polygons and random shapes.
  • the light modulation structure includes an irregular structure may also refer to: the arrangement of the minimum modulation units included in the light modulation structure is irregular, for example, the arrangement may be in the form of an irregular polygon, a random arrangement, and the like.
  • the light modulation structure includes an irregular structure may also refer to: the minimum modulation unit included in the light modulation structure is an irregular structure, and the arrangement of the minimum modulation unit is also irregular.
  • the light modulation structures in the optical filter layer may include discrete structures, may also include continuous structures, or may include both discrete structures and continuous structures.
  • the continuous modulation pattern here may refer to a linear pattern, a wavy line pattern, a zigzag line pattern, and the like.
  • discrete modulation pattern herein may refer to a modulation pattern formed by discrete figures (eg, discrete dots, discrete triangles, discrete stars, etc.).
  • the light modulation structure has different modulation effects on light of different wavelengths, and specific modulation methods include but are not limited to scattering, absorption, interference, surface plasmon, and resonance enhancement.
  • specific modulation methods include but are not limited to scattering, absorption, interference, surface plasmon, and resonance enhancement.
  • the optical filter layer is a single-layer structure or a multi-layer structure.
  • the optical filter layer may be a single-layer filter structure or a multi-layer filter structure, such as a two-layer, three-layer, four-layer and other multi-layer structure.
  • the optical filter layer 1 has a single-layer structure, and the thickness of the optical filter layer 1 is related to the target wavelength range.
  • the thickness of the grating structure can be 50 nm. ⁇ 5 ⁇ m.
  • the optical filter layer 1 since the function of the optical filter layer 1 is to spectrally modulate the incident light, it is preferably prepared from materials with high refractive index and low loss, for example, silicon, germanium, germanium-silicon materials, silicon compounds, germanium can be selected. Compounds, III-V group materials, etc. are prepared, wherein silicon compounds include but are not limited to silicon nitride, silicon dioxide, silicon carbide, and the like.
  • the optical filter layer 1 in order to form more or more complex connection weights between the input layer and the linear layer, preferably, the optical filter layer 1 can be set as a multi-layer structure, and the corresponding optical modulation of each layer The structure can be set to different structures, thereby increasing the spectral modulation capability of the optical filter layer to the incident light, so that more or more complex connection weights can be formed between the input layer and the linear layer, thereby improving the processing intelligence of the smart chip. accuracy at the time of the task.
  • the materials of each layer structure may be the same or different.
  • the first layer may be is a silicon layer
  • the second layer may be a silicon nitride layer.
  • the thickness of the optical filter layer 1 is related to the target wavelength range. For wavelengths of 400 nm to 10 ⁇ m, the total thickness of the multilayer structure may be 50 nm to 5 ⁇ m.
  • the light modulation structure in the optical filter layer includes a unit array composed of a plurality of micro-nano units, and each micro-nano unit corresponds to one or more units on the image sensor. Pixel point; the structure of each micro-nano unit is the same or different.
  • the light modulation structure in the form of an array structure.
  • the light modulation structure includes a unit array composed of a plurality of micro-nano units, and each micro-nano unit corresponds to one or more pixels on the image sensor.
  • the structures of each micro-nano unit may be the same or different.
  • the structure of each micro-nano unit may be periodic or aperiodic.
  • each micro-nano unit may further include multiple groups of micro-nano structure arrays, and the structures of each group of micro-nano structure arrays are the same or different.
  • the optical filter layer 1 includes a plurality of repeating continuous or discrete micro-nano units, such as 11, 22, 33, 44, 55 , 66, each micro/nano unit has the same structure (and each micro/nano unit is aperiodic structure), and each micro/nano unit corresponds to one or more pixels on the image sensor 2; as shown in FIG.
  • the optical filter Layer 1 contains multiple repeating micro-nano units, such as 11, 22, 33, 44, 55, 66, and each micro-nano unit has the same structure (the difference from Figure 5 is that each micro-nano unit in Figure 6 has a periodic structure) , each micro-nano unit corresponds to one or more pixels on the image sensor 2; as shown in FIG. 7, the optical filter layer 1 includes a plurality of repeated micro-nano units, such as 11, 22, 33, 44, 55, 66. Each micro-nano unit has the same structure (and each micro-nano unit has a periodic structure), and each micro-nano unit corresponds to one or more pixels on the image sensor 2. The difference from FIG.
  • the unit shape of the periodic array in the micro/nano unit has quadruple rotational symmetry; as shown in FIG.
  • the optical filter layer 1 includes a plurality of mutually different micro-nano units. , that is, different areas on the smart chip have different modulation effects on the incident light, thereby improving the freedom of design and improving the accuracy of recognition.
  • the optical filter layer 1 includes a plurality of repeated micro-nano units, such as 11, 22, 33, 44, 55, 66, and each micro-nano unit has the same structure.
  • each micro-nano unit has the same structure.
  • the units are composed of discrete aperiodic array structures, and each micro-nano unit corresponds to one or more pixels on the image sensor 2 .
  • the micro-nano unit has different modulation effects on light of different wavelengths, and the specific modulation methods include but are not limited to scattering, absorption, interference, surface plasmon, resonance enhancement, and the like.
  • the specific modulation methods include but are not limited to scattering, absorption, interference, surface plasmon, resonance enhancement, and the like.
  • the micro-nano units include regular structures and/or irregular structures; and/or, the micro-nano units include discrete structures and/or continuous structures.
  • the micro-nano unit may only include regular structures, may only include irregular structures, or may include both regular structures and irregular structures.
  • the micro-nano unit includes a regular structure may refer to: the minimum modulation unit included in the micro-nano unit is a regular structure, for example, the minimum modulation unit may have regular patterns such as rectangle, square and circle.
  • the micro-nano unit includes a regular structure can also refer to: the arrangement of the smallest modulation units contained in the micro-nano unit is regular, for example, the arrangement can be a regular array form, a circular form, a trapezoidal form, a polygonal form, etc.
  • the fact that the micro-nano unit includes a regular structure here may also refer to: the minimum modulation unit included in the micro-nano unit is a regular structure, and the arrangement of the minimum modulation unit is also regular.
  • the micro-nano unit includes an irregular structure may refer to: the minimum modulation unit included in the micro-nano unit is an irregular structure, for example, the minimum modulation unit may be irregular polygons, random shapes and other irregular shapes.
  • the micro-nano unit includes irregular structure may also refer to: the arrangement of the smallest modulation units included in the micro-nano unit is irregular, for example, the arrangement may be in the form of irregular polygons, random arrangement and the like.
  • the micro-nano unit includes an irregular structure may also refer to: the minimum modulation unit included in the micro-nano unit is an irregular structure, and the arrangement of the minimum modulation unit is also irregular.
  • the micro-nano units in the optical filter layer may include discrete structures, may also include continuous structures, or may include both discrete and continuous structures.
  • the micro-nano unit includes a continuous structure may refer to: the micro-nano unit is composed of continuous modulation patterns; here the micro-nano unit includes a discrete structure may refer to: the micro-nano unit is composed of discrete modulation patterns of.
  • the continuous modulation pattern here may refer to a linear pattern, a wavy line pattern, a zigzag line pattern, and the like.
  • discrete modulation pattern herein may refer to a modulation pattern formed by discrete figures (eg, discrete dots, discrete triangles, discrete stars, etc.).
  • micro-nano units have different modulation effects on light of different wavelengths, and specific modulation methods include but are not limited to scattering, absorption, interference, surface plasmon, and resonance enhancement.
  • specific modulation methods include but are not limited to scattering, absorption, interference, surface plasmon, and resonance enhancement.
  • the micro-nano unit includes multiple groups of micro-nano structure arrays, and the structures of each group of micro-nano structure arrays are the same or different.
  • the optical filter layer 1 includes a plurality of repeated micro-nano units, such as 11, 22, 33, 44, 55, 66, and each micro-nano unit includes a plurality of groups of micro-nano units
  • the structure array for example, the micro/nano unit 11 includes four different micro/nano structure arrays 110 , 111 , 112 and 113
  • the filtering unit 44 includes four different micro/nano structure arrays 440 , 441 , 442 and 443 . As shown in FIG.
  • the optical filter layer 1 includes a plurality of micro-nano units, such as 11, 22, 33, 44, 55, 66, and each micro-nano unit includes a plurality of groups of micro-nano structure arrays, such as the micro-nano unit 11 Four identical micro-nano structure arrays 110 , 111 , 112 and 113 are included.
  • micro-nano unit including four groups of micro-nano structure arrays is used as an example for illustration, and it does not play a limiting role.
  • the micro-nano unit of the micro-nano structure array is used as an example for illustration, and it does not play a limiting role.
  • each group of micro-nano structure arrays in the micro-nano unit has different modulation effects on light of different wavelengths, and the modulation effects on input light between groups of filter structures are also different.
  • the specific modulation methods include: Not limited to scattering, absorption, interference, surface plasmon, resonance enhancement, etc.
  • each group of micro-nano structure arrays has the function of broadband filtering or narrow-band filtering.
  • the micro-nano structure array obtains the modulation intensity of different wavelength components of the incident light of the target object by performing broadband filtering or narrow-band filtering on the incident light of the target object.
  • each group of micro-nano structure arrays in the optical filter layer has the function of broadband filtering or narrow-band filtering.
  • each group of micro-nano structure arrays may have broadband filtering effects, and they may all have narrow-band filtering effects, and some may have broadband filtering effects, and some may have narrow-band filtering effects.
  • the broadband filtering range and the narrow-band filtering range of each group of micro-nano structure arrays may also be the same or different.
  • it by designing the period, duty cycle, radius, side length and other size parameters of each group of micro-nano structures in the micro-nano unit, it has a narrow-band filtering effect, that is, only one (or less) wavelength of light can be used. pass.
  • the period, duty cycle, radius, side length and other dimensional parameters of each group of micro-nano structures in the micro-nano unit it has a broadband filtering effect, that is, light with more or all wavelengths can pass through.
  • the filtering state of each group of micro-nano structure arrays can be determined by performing broadband filtering, narrow-band filtering or a combination thereof according to the application scenario.
  • each group of micro-nano structure arrays is a periodic structure array or a non-periodic structure array.
  • each group of micro-nano structure arrays may be all periodic structure arrays, may all be non-periodic structure arrays, or may be partially periodic structure arrays and partially non-periodic structure arrays.
  • the periodic structure array is easy to carry out optical simulation design, and the aperiodic structure array can realize more complex modulation effect.
  • the optical filter layer 1 includes a plurality of repeated micro-nano units, such as 11, 22, 33, 44, 55, 66, and each micro-nano unit is composed of multiple groups of micro-nano structures
  • the arrays are composed of different micro-nano structure arrays, and the micro-nano structure arrays are non-periodic structures.
  • the aperiodic structure means that the shape of the modulation holes on the micro-nano structure array is arranged in a non-periodic arrangement. As shown in FIG.
  • the micro-nano unit 11 includes 4 different aperiodic structure arrays 110 , 111 , 112 and 113
  • the micro-nano unit 44 includes 4 different aperiodic structure arrays 440 , 441 , 442 and 443
  • the micro-nano structure array of sexual structure is designed by neural network data training for intelligent processing tasks in the early stage, and it is usually an irregular-shaped structure.
  • the optical filter layer 1 includes a plurality of repeated micro-nano units, such as 11, 22, 33, 44, 55, 66, and each micro-nano unit is composed of multiple groups of micro-nano structure arrays. The structures of the structure arrays are different from each other. The difference from FIG.
  • the micro-nano structure array is a periodic structure.
  • the periodic structure means that the shape of the modulation holes on the micro-nano structure array is arranged in a periodic arrangement, and the size of the period is usually 20 nm to 50 ⁇ m.
  • the micro-nano unit 11 includes 4 different periodic structure arrays 110 , 111 , 112 and 113
  • the micro-nano unit 44 includes 4 different periodic structure arrays 440 , 441 , 442 and 443 .
  • the filter structure is designed by training the neural network data for intelligent processing tasks in the early stage, and it is usually an irregular-shaped structure. As shown in FIG.
  • the optical filter layer 1 includes a plurality of different micro-nano units, such as 11, 22, 33, 44, 55, 66, and each micro-nano unit is composed of multiple groups of micro-nano structure arrays.
  • the structures of the micro-nano structure arrays are different from each other, and the micro-nano structure arrays are periodic structures.
  • the periodic structure means that the shapes on the filter structure are arranged in a periodic arrangement, and the size of the period is usually 20 nm to 50 ⁇ m.
  • the micro-nano structure arrays of the micro-nano unit 11 and the micro-nano unit 12 are different from each other.
  • the micro-nano unit 11 includes four different periodic structure arrays 110 , 111 , 112 and 113
  • the micro-nano unit 44 includes four There are different periodic structure arrays 440 , 441 , 442 and 443
  • the periodic structure micro-nano structure arrays are designed by neural network data training for intelligent processing tasks in the early stage, and are usually irregular-shaped structures.
  • each micro-nano unit in Fig. 5 to Fig. 9 includes four groups of micro-nano structure arrays, and the four groups of micro-nano structure arrays are respectively formed with modulation holes of four different shapes.
  • the incident light has different modulation effects.
  • the micro-nano unit including four groups of micro-nano structure arrays is used as an example for illustration, and it does not play a limiting role.
  • the four different shapes may be a circle, a cross, a regular polygon and a rectangle (not limited thereto).
  • each group of micro-nano structure arrays in the micro-nano unit has different modulation effects on light of different wavelengths, and the modulation effects on the input light are also different between groups of micro-nano structure arrays.
  • the specific modulation method Including but not limited to scattering, absorption, interference, surface plasmon, resonance enhancement, etc.
  • the optical filter layer 1 includes a plurality of repeated micro-nano units, such as 11, 22, 33, 44, 55, 66, Each micro-nano unit is composed of multiple groups of micro-nano structure arrays. The structures corresponding to the multiple groups of micro-nano structure arrays are different from each other.
  • the micro-nano structure array is a periodic structure. The difference from the above embodiment is that for any micro-nano structure array A cell contains one or more sets of empty structures for passing through incident light.
  • each micro-nano unit includes a set of micro-nano structure arrays and three sets of empty structures
  • the micro-nano unit 11 includes an aperiodic structure array 111
  • the micro-nano unit 22 includes an aperiodic structure array 221
  • the micro-nano unit 33 includes an aperiodic structure array 331
  • the micro-nano unit 44 includes an aperiodic structure array 441
  • the micro-nano unit 55 includes an aperiodic structure array 55
  • the micro-nano unit 66 includes an aperiodic structure array Array 661, wherein the array of micro-nano structures is used for different modulation of incident light.
  • micro-nano structure arrays can be set to include a set of micro-nano structure arrays and five Groups of empty structures or other numbers of micro-nano units of micro-nano structure arrays.
  • the array of micro-nano structures can be made of circular, cross-shaped, regular polygon and rectangular modulation holes (not limited thereto).
  • the groups of micro-nano structure arrays included in the micro-nano unit may not include empty structures, that is, the groups of micro-nano structure arrays may be aperiodic structure arrays or periodic structure arrays.
  • the micro-nano unit has polarization independent characteristics.
  • the optical filter layer is insensitive to the polarization of incident light, thereby realizing an optical artificial neural network smart chip that is insensitive to incident angle and polarization.
  • the optical artificial neural network smart chip provided by the embodiment of the present application is not sensitive to the incident angle and polarization characteristics of incident light, that is, the measurement result will not be affected by the incident angle and polarization characteristics of incident light, thereby ensuring the stability of spectral measurement performance. , which can ensure the stability of intelligent processing, such as the stability of intelligent perception, the stability of intelligent recognition, the stability of intelligent decision-making and so on.
  • the micro-nano unit may also have polarization-dependent properties.
  • the micro-nano unit has quadruple rotational symmetry.
  • quadruple rotational symmetry is a specific case of polarization-independent characteristics.
  • the optical filter layer 1 includes a plurality of repeated micro-nano units, such as 11, 22, 33, 44, 55, 66, Each micro-nano unit is composed of multiple groups of micro-nano structure arrays, the corresponding structures of the multiple groups of micro-nano structure arrays are different from each other, and the micro-nano structure arrays are periodic structures.
  • the corresponding structure of the array can be a circle, a cross, a regular polygon, a rectangle, etc.
  • the optical filter layer is composed of one or more layers
  • the filter layer is made of one or more of semiconductor materials, metal materials, liquid crystals, quantum dot materials, and perovskite materials; and/or, the filter layer is made of photonic crystals, metasurfaces, random Structure, nanostructure, metal surface plasmon SPP (Surface Plasmon Polaritons, SPP) micro-nano structure, tunable Fabry-Perot cavity (Fabry-perot Cavity, FP cavity) one or more preparations filter layer.
  • SPP Surface Plasmon Polaritons
  • Fabry-perot Cavity, FP cavity tunable Fabry-Perot cavity
  • the semiconductor material includes one or more of silicon, silicon oxide, silicon nitride, titanium oxide, composite materials mixed in a preset ratio, and direct bandgap compound semiconductor materials; and/or, the nanostructure includes nanostructures.
  • photonic crystals and the combination of metasurfaces and random structures can be compatible with CMOS technology and can have better modulation effects.
  • the micropores of micro-nano modulation structures can also be filled with other materials for surface smoothing; Mine can make use of the spectral modulation characteristics of the material itself to minimize the volume of a single modulation structure; SPP is small in size and can realize polarization-dependent light modulation; liquid crystal can be dynamically regulated by voltage to improve spatial resolution; adjustable Fabry-Perot The resonant cavity can be dynamically adjusted to improve the spatial resolution.
  • the thickness of the optical filter layer is 0.1 ⁇ ⁇ 10 ⁇ , where ⁇ represents the center wavelength of the incident light.
  • the overall size (area) of each micro-nano unit in the optical filter layer 1 is usually ⁇ 2 -105 ⁇ 2 , and the thickness is usually 0.1 ⁇ 10 ⁇ ( ⁇ represents the center wavelength of the incident light of the target object).
  • the overall size of each micro-nano unit is 0.5 ⁇ m 2 to 40000 ⁇ m 2
  • the dielectric material in the optical filter layer 1 is polysilicon
  • the thickness is 50 nm to 2 ⁇ m.
  • the image sensor is any one or more of the following:
  • CMOS image sensor Contact Image Sensor, CIS
  • Charge Coupled Device Charge Coupled Device, CCD
  • Single Photon Avalanche Diode Single Photon Avalanche Diode, SPAD
  • focal plane photodetector array CMOS image sensor (Contact Image Sensor, CIS), Charge Coupled Device (Charge Coupled Device, CCD), Single Photon Avalanche Diode (Single Photon Avalanche Diode, SPAD) array and focal plane photodetector array.
  • CMOS image sensor CIS to achieve monolithic integration at the wafer level can minimize the distance between the image sensor and the optical filter layer, which is beneficial for Reduce the size of the unit, reduce the size of the device and the cost of packaging, SPAD can be used for weak light detection, CCD can be used for strong light detection.
  • the optical filter layer and the image sensor can be fabricated by a complementary metal oxide semiconductor (Complementary Metal Oxide Semiconductor, CMOS) integrated process, which is beneficial to reduce device failure rate, improve device yield and reduce cost.
  • CMOS complementary Metal Oxide Semiconductor
  • the optical filter can be prepared by directly growing one or more layers of dielectric material on the image sensor, and then performing etching, depositing metal material before removing the sacrificial layer for etching, and finally removing the sacrificial layer. device layer.
  • the type of the artificial neural network includes: a feedforward neural network.
  • Feedforward Neural Network also known as Deep Feedforward Network (DFN), Multi-Layer Perceptron (MLP)
  • DNN Deep Feedforward Network
  • MLP Multi-Layer Perceptron
  • the feedforward neural network has a simple structure, is easy to implement on hardware, and has a wide range of applications. It can approximate any continuous function and square-integrable function with arbitrary precision, and can accurately implement any limited training sample set.
  • a feedforward network is a static nonlinear mapping. Complex nonlinear processing capabilities can be obtained through composite mapping of simple nonlinear processing units.
  • a light-transmitting medium layer is provided before the optical filter layer and the image sensor.
  • a light-transmitting medium layer is arranged between the optical filter layer and the image sensor, which can effectively separate the optical filter layer and the image sensor layer and avoid mutual interference between the two.
  • the image sensor is a front-illuminated type, including: a metal wire layer and a light detection layer arranged from top to bottom, and the optical filter layer is integrated in the metal wire layer the side away from the light detection layer; or,
  • the image sensor is back-illuminated and includes: a light detection layer and a metal line layer arranged from top to bottom, and the light filter layer is integrated on the side of the light detection layer away from the metal line layer.
  • the silicon detection layer 21 is below the metal wire layer 22
  • the optical filter layer 1 is directly integrated on the metal wire layer 22 .
  • FIG. 14 shows a back-illuminated image sensor, the silicon detection layer 21 is above the metal wire layer 22 , and the optical filter layer 1 is directly integrated on the silicon detection layer 21 .
  • the silicon detection layer 21 is above the metal wire layer 22, which can reduce the influence of the metal wire layer on the incident light, thereby improving the quantum efficiency of the device.
  • the optical filter layer is used as the input layer and the linear layer of the artificial neural network
  • the image sensor is used as a part of the nonlinear layer of the artificial neural network (the square detection response of the image sensor is used as the artificial neural network).
  • the first nonlinear activation function), the filtering effect of the optical filter layer on the incident light entering the optical filter layer is used as the connection weight between the input layer and the linear layer.
  • the optical filter layer and the image sensor realize the related functions of the input layer, linear layer and part of the nonlinear activation function in the artificial neural network by means of hardware, so that the subsequent use of the intelligent chip for intelligent processing does not need to be performed and input.
  • this embodiment utilizes image information, spectral information, angle information of incident light, and phase information of incident light at different spatial points of the target object simultaneously, so that intelligent processing of the target object can be more accurately realized.
  • the optical filter layer is used as the input layer and the linear layer of the artificial neural network
  • the image sensor is used as a part of the nonlinear layer of the artificial neural network to project the spatial spectral information of the object to the detector.
  • the full connection and quadratic nonlinear activation of electrical signals are realized in the processor, realizing functions such as intelligent perception, recognition and/or decision-making with low power consumption, low delay and high accuracy.
  • the optical artificial neural network smart chip based on the optical filter and the image sensor in the embodiment of the present application has the following effects: the artificial neural network is partially embedded in the image sensor including various optical filter layers, so as to realize fast and accurate intelligent perception, recognition and detection. / or decision function.
  • the embodiments of the present application can also realize monolithic integration at the wafer level, so that the distance between the sensor and the optical filter layer can be minimized, which is conducive to reducing the size of the unit, reducing the volume of the device and the packaging cost.
  • a smart device including: the optical artificial neural network smart chip as described in the above embodiments.
  • the smart devices include one or more of smart phones, smart computers, smart identification devices, smart perception devices, and smart decision-making devices.
  • the smart device provided in this embodiment includes the optical artificial neural network smart chip described in the above embodiment
  • the smart device provided by this embodiment has all the beneficial effects of the optical artificial neural network smart chip described in the above embodiment. This has been described in detail in the above-mentioned embodiments, and thus will not be repeated in this embodiment.
  • FIG. 15 Another embodiment of the present application provides a method for preparing an optical artificial neural network smart chip according to the above-mentioned embodiment, as shown in FIG. 15 , which specifically includes the following steps:
  • Step 1510 preparing an optical filter layer including a light modulation structure on the surface of the image sensor
  • Step 1520 generating a processor with the function of performing full connection processing on the signal or generating a processor with the function of performing the full connection processing on the signal and the second nonlinear activation processing function;
  • Step 1530 connect the image sensor and the processor
  • the optical filter layer is used to perform different spectral modulation on the incident light entering different positions of the optical modulation structure through the optical modulation structure, so as to obtain different spectral modulations on the surface of the image sensor.
  • the image sensor converts the information carried by the incident light corresponding to the different position points after being modulated by the optical filter layer to the electrical signals corresponding to the different position points after the first nonlinear activation processing through the square detection response, and will be different from the different position points. sending the electrical signal corresponding to the location point to the processor;
  • the processor performs full connection processing on the electrical signals corresponding to different position points, or the processor performs full connection processing and second nonlinear activation processing on the electrical signals corresponding to different position points to obtain an artificial neural network. output signal.
  • the training process of the optical artificial neural intelligence chip is also included, which specifically includes:
  • the optical artificial neural intelligence chips including different light modulation structures, image sensors and processors with different fully connected parameters are trained to meet the training convergence conditions
  • the light modulation structure, image sensor and processor are obtained, and the light modulation structure, image sensor and processor satisfying the training convergence condition are used as the trained light modulation structure, image sensor and processor;
  • the processing of the processor including different light modulation structures, image sensors, and different fully connected parameters and different second nonlinear activation parameters is used.
  • the optical artificial neural intelligence chip is trained to obtain a light modulation structure, an image sensor and a processor that meet the training convergence conditions, and the light modulation structure, image sensor and processor that meet the training convergence conditions are used as the trained light modulation structure, image sensor and processor. processor.
  • an optical filter layer including a light modulation structure is prepared on the surface of the photosensitive region of the image sensor, including:
  • an optical filter layer containing a light modulation structure is obtained;
  • the input training samples and the output training samples corresponding to the intelligent processing is trained to obtain the light modulation structure, image sensor and processor that meet the training convergence conditions; parameters and the optical artificial neural network intelligent chip of the processor with different second nonlinear activation parameters are trained to obtain an optical modulation structure, an image sensor and a processor that satisfy the training convergence conditions.
  • the optical filter layer 1 can be etched by directly growing one or more layers of dielectric materials on the image sensor 2 . Before the sacrificial layer is deposited, the metal material is deposited, and finally the sacrificial layer is removed for preparation.
  • each unit can have different modulation effects on light of different wavelengths within the target range, and the modulation effects are not sensitive to the incident angle and polarization.
  • Each unit in the optical filter layer 1 corresponds to one or more pixels on the image sensor 2 . 1 was prepared directly on 2.
  • the optical filter layer 1 can be directly etched on the silicon detector layer 21 of the back-illuminated image sensor. etched and then deposited metal for preparation.
  • the light modulation structure on the optical filter layer can be dry-etched by performing light modulation structure pattern on one or more preset materials. Dry etching is to directly photosensitive the image sensor. One or more layers of preset materials on the surface of the area are removed to obtain an optical filter layer containing a light modulation structure; or one or more layers of preset materials are imprinted and transferred, and imprint transfer is performed in
  • the required structure is prepared on other substrates by etching, and then the structure is transferred to the photosensitive area of the image sensor through materials such as PDMS to obtain an optical filter layer containing a light modulation structure; or by presetting one or more layers
  • the material is subjected to external dynamic control, and the external dynamic control is to use active materials, and then the external electrode can adjust the light modulation characteristics of the corresponding area by changing the voltage, so as to obtain an optical filter layer containing a light modulation structure; It is assumed that the material is printed by partition, and the partition printing is a technology of partition printing to obtain an optical filter layer containing
  • Environmental problems are the main problems faced by my country's sustainable development, and environmental inspection work can achieve effective protection and governance of the environment.
  • Environmental testing includes real-time monitoring of air, water quality, soil, etc., providing scientific, accurate and effective monitoring for environmental management, and formulating reasonable solutions accordingly.
  • Traditional environmental pollution monitoring is based on wet chemical techniques and experimental analysis after aspirating sampling.
  • the rapid development of analytical instruments in recent years can meet the needs of many environmental pollution monitoring, but these instruments are usually limited to single-point measurement.
  • optical and spectroscopy technologies have become ideal tools for environmental pollution monitoring due to their large-scale, multi-component detection, good time resolution, and continuous real-time monitoring. Therefore, realizing a large-scale, high-resolution, small-volume, low-cost, safe and reliable real-time environmental pollution detection chip is of great significance to environmental governance and protection.
  • this embodiment provides a new type of optoelectronic chip for multi-component environment detection, the chip is composed of an optical filter layer to form the input layer and linearity of the optical artificial neural network.
  • the first nonlinear activation function of the optical artificial neural network is composed of image sensors. Identification and qualitative analysis. The content of this embodiment will be explained and described in detail below.
  • the embodiment of the present application also provides an optical artificial neural network environmental protection monitoring chip, which is used for intelligent processing tasks of environmental protection monitoring, including: an optical filter layer, an image sensor and a processor; the optical filter layer corresponds to the input of the artificial neural network layer, linear layer and the connection weight from the input layer to the linear layer, the square detection response of the image sensor corresponds to the first nonlinear activation function in the nonlinear layer of the artificial neural network; the processor corresponds to the artificial neural network The full connection and the output layer of the neural network, or the processor corresponds to the full connection of the artificial neural network, the second nonlinear activation function in the nonlinear layer, and the output layer;
  • the optical filter layer is disposed on the surface of the image sensor, and the optical filter layer includes a light modulation structure, and the light modulation structure is used for the incident light entering into different positions of the light modulation structure.
  • different spectral modulations to obtain incident light-carrying information corresponding to different positions on the surface of the image sensor;
  • the incident light-carrying information includes light intensity distribution information, spectral information, angle information of the incident light, and the Phase information of incident light;
  • the incident light includes reflected light, transmitted light and/or radiated light of environmental pollutants;
  • the image sensor converts the information carried by the incident light corresponding to the different position points after being modulated by the optical filter layer to the electrical signals corresponding to the different position points after the first nonlinear activation processing through the square detection response, and will be different from the different position points. sending the electrical signal corresponding to the location point to the processor;
  • the processor performs full connection processing on electrical signals corresponding to different locations, or the processor performs full connection processing and second nonlinear activation processing on electrical signals corresponding to different locations, so as to obtain environmental monitoring intelligence. process result;
  • the environmental protection monitoring intelligent processing task includes identification and/or qualitative analysis of environmental pollutants; the environmental protection monitoring intelligent processing results include identification results of environmental pollutants and/or qualitative analysis results of environmental pollution.
  • This embodiment implements a brand-new optical artificial neural network environmental protection monitoring chip capable of realizing the artificial neural network function, which is used for environmental protection monitoring tasks.
  • the filter layer is used as the input layer and linear layer of the artificial neural network
  • the filtering effect of the optical filter layer on the hardware chip on the incident light is used as the connection weight from the input layer to the linear layer
  • the square detection response of the image sensor on the hardware chip is used as The first nonlinear activation function in the nonlinear layer of the artificial neural network
  • the embodiment of the present application injects the spatial spectral information of environmental pollutants into a pre-trained hardware chip
  • the hardware chip is used to detect different spatial points of the environmental pollutants.
  • the image information, spectral information, angle information of incident light and phase information of incident light are analyzed by artificial neural network to obtain environmental protection monitoring results. It should be noted that the embodiment of the present application achieves low power consumption, safe, reliable, fast and accurate Environmental monitoring.
  • the hardware structure-optical filter layer on it corresponds to the input layer and linear layer of the artificial neural network
  • the hardware structure-image sensor on it corresponds to the artificial neural network.
  • a part of the nonlinear layer; the processor corresponds to another part of the nonlinear layer of the artificial neural network and the output layer.
  • the optical filter layer is disposed on the surface of the image sensor, and the optical filter layer includes a light modulation structure, and the light modulation structure is used to perform different spectral modulations on the incident light entering different positions of the light modulation structure, so as to Information carried by the incident light corresponding to different positions is obtained on the surface of the image sensor.
  • the modulation effect of the light modulation structure on the optical filter layer on the incident light is equivalent to the connection between the input layer and the linear layer. Weights.
  • the image sensor converts the incident light-carrying information corresponding to the different position points after being modulated by the optical filter layer for the first time, and then converts it into the information corresponding to the different position points after the first nonlinear activation processing through the square detection response.
  • the electrical signals corresponding to different positions are sent to the processor, and the processor performs full connection processing on the electrical signals corresponding to different positions, or the processor sends the electrical signals corresponding to different positions.
  • the full connection processing and the second nonlinear activation processing are performed to obtain the output signal of the artificial neural network.
  • the optical filter layer and the image sensor implemented in hardware replace or The related functions of the input layer, linear layer and partial nonlinear activation function in the existing artificial neural network are realized, that is, the input layer, linear layer and Part or all of the nonlinear activation functions are stripped, and the input layer, linear layer, and part or all of the nonlinear activation functions in the artificial neural network are realized by hardware, so that the optical artificial neural network can be used in the subsequent use.
  • the environmental monitoring chip When the environmental monitoring chip performs artificial neural network intelligent processing, it does not need to perform complex signal processing and algorithm processing corresponding to the input layer, linear layer and some or all of the nonlinear activation functions.
  • the processor only needs to perform full connection processing or full connection with the electrical signal and the second nonlinear activation processing, which can greatly reduce the power consumption and delay of the artificial neural network processing.
  • the optical filter layer is used as the input layer, the linear layer, and the connection weight between the input layer and the linear layer, and the square detection response of the image sensor is used as the non-linear layer of the artificial neural network.
  • the first nonlinear activation function in the linear layer; the processor is used as the fully connected and output layer of the artificial neural network, or, the processor corresponds to the second nonlinear activation in the fully connected, nonlinear layer of the artificial neural network function and output layer, it can be seen that the embodiment of the present application can not only save the complex signal processing and algorithm processing corresponding to the input layer, the linear layer and some nonlinear activation functions in the prior art, but also the embodiment of the present application actually At the same time, the image information, spectral information, angle of incident light and phase information of incident light of environmental pollutants are used, that is, the information carried by incident light at different spatial points of environmental pollutants.
  • the incident light-carrying information at the point covers the image, composition, shape, three-dimensional depth, structure and other information of environmental pollutants, so that when the identification processing is performed according to the incident light-carrying information at different spatial points of environmental pollutants, the environmental pollutants can be covered.
  • the image, composition, shape, three-dimensional depth, structure and other multi-dimensional information of pollutants can accurately measure the information of environmental pollutants.
  • the optical artificial neural network environmental protection monitoring chip includes a trained light modulation structure, an image sensor and a processor;
  • the trained light modulation structures, image sensors and processors refer to using the input training samples and output training samples corresponding to the environmental monitoring intelligent processing tasks, to perform data analysis on the light modulation structures, image sensors and processors with different light modulation structures and image sensors.
  • the optical modulation structure, image sensor and processor that meet the training convergence conditions obtained by training the optical artificial neural network environmental monitoring chip connected to the parameter processor; or, the trained optical modulation structure, image sensor and processor refer to Using the input training samples and output training samples corresponding to the environmental monitoring intelligent processing task, the processing of the processor including different light modulation structures, image sensors and processors with different fully connected parameters and different second nonlinear activation parameters An optical modulation structure, an image sensor and a processor that satisfy the training convergence conditions obtained by training the optical artificial neural network environmental protection monitoring chip;
  • the input training samples include incident light reflected, transmitted and/or radiated by different environmental pollutants; the output training samples include corresponding identification results of environmental pollutants; and/or, the input training samples include Incident light reflected, transmitted and/or radiated by environmental pollutants; the output training samples include corresponding qualitative analysis results of environmental pollution.
  • the different light modulation structures are designed and realized by adopting computer optical simulation design.
  • a large number of samples of environmental pollutants can be collected first, and the weight of the linear layer, that is, the system function of the optical filter layer, can be obtained through data training, and the required optical filter layer can be designed in reverse.
  • the output of the optical filter layer is used to further train and optimize the weight of the fully connected layer of the electrical signal, and then a high-accuracy optical artificial neural network can be realized to complete the detection of environmental pollutant samples. rapid and accurate identification and qualitative analysis.
  • the chip actually utilizes the image information, spectral information, angle information of incident light and phase information of incident light at different points in the environmental pollution space at the same time, which improves the accuracy and diversity of environmental pollution detection, and improves the accuracy and diversity of environmental pollution detection.
  • the artificial neural network is partially realized on the hardware, which improves the speed of environmental pollution detection.
  • the chip solution can use the existing CMOS process to achieve mass production, reducing the size, power consumption and cost of the device.
  • the light modulation structures in the optical filter layer include regular structures and/or irregular structures; and/or, the light modulation structures in the optical filter layer include discrete structures and/or continuous structures.
  • the optical filter layer has a single-layer structure or a multi-layer structure.
  • the light modulation structure in the optical filter layer includes a unit array composed of a plurality of micro/nano units, each micro/nano unit corresponds to one or more pixels on the image sensor; the structures of each micro/nano unit are the same or different.
  • micro-nano unit includes a regular structure and/or an irregular structure; and/or, the micro-nano unit includes a discrete structure and/or a continuous structure.
  • micro-nano unit includes a plurality of groups of micro-nano structure arrays, and the structures of each group of micro-nano structure arrays are the same or different.
  • each group of micro-nano structure arrays has the function of broadband filtering or narrow-band filtering.
  • each group of micro-nano structure arrays is a periodic structure array or a non-periodic structure array.
  • micro-nano unit has polarization independent properties.
  • micro-nano unit has quadruple rotational symmetry.
  • the filter layer is made of one or more of semiconductor materials, metal materials, liquid crystals, quantum dot materials, and perovskite materials; and/or, the filter layer is made of photonic crystal, ultra A filter layer prepared by one or more of surface, random structure, nanostructure, metal surface plasmon SPP micro-nano structure, and tunable Fabry-Perot resonant cavity.
  • the semiconductor material includes one or more of silicon, silicon oxide, silicon nitride, titanium oxide, composite materials mixed in a preset ratio, and direct bandgap compound semiconductor materials; and/or, the nanometer
  • the structure includes one or more of nanodot 2D materials, nanopillar 2D materials, and nanowire 2D materials.
  • the thickness of the optical filter layer is 0.1 ⁇ ⁇ 10 ⁇ , where ⁇ represents the center wavelength of the incident light.
  • the embodiments of the present application also provide an environmental protection monitoring device, including the optical artificial neural network environmental protection monitoring chip described in the above embodiments.
  • the environmental protection monitoring equipment may be an environmental condition detector, a pollutant content analyzer, and the like.
  • Embodiments of the present application also provide a method for preparing an optical artificial neural network environmental protection monitoring chip as described above, including:
  • the optical filter layer is used to perform different spectral modulation on the incident light entering different positions of the optical modulation structure through the optical modulation structure, so as to obtain different spectral modulations on the surface of the image sensor.
  • the image sensor converts the information carried by the incident light corresponding to the different position points after being modulated by the optical filter layer to the electrical signals corresponding to the different position points after the first nonlinear activation processing through the square detection response, and will be different from the different position points. sending the electrical signal corresponding to the location point to the processor;
  • the processor performs full connection processing on electrical signals corresponding to different locations, or the processor performs full connection processing and second nonlinear activation processing on electrical signals corresponding to different locations, so as to obtain environmental monitoring intelligence. process result.
  • the preparation method of the optical artificial neural network environmental protection monitoring chip also includes: a training process for the optical artificial neural network environmental protection monitoring chip, specifically including:
  • the optical artificial neural network environmental monitoring chips containing different light modulation structures, image sensors and processors with different fully connected parameters are trained to obtain A light modulation structure, an image sensor, and a processor that satisfy the training convergence conditions, and the light modulation structure, the image sensor, and the processor that satisfy the training convergence conditions are used as the trained light modulation structure, image sensor, and processor;
  • the processing that includes different light modulation structures, image sensors, and different fully connected parameters and different second nonlinear activation parameters
  • the optical artificial neural network environmental monitoring chip of the device is trained to obtain a light modulation structure, an image sensor and a processor that meet the training convergence conditions, and the light modulation structure, image sensor and processor that meet the training convergence conditions are used as the trained light modulation structure. , image sensor and processor.
  • the different light modulation structures are designed and realized by adopting computer optical simulation design.
  • the chip directly prepares the micro/nano modulation structure on the surface of the photosensitive area of the image sensor, and several discrete or continuous micro/nano structures constitute a unit, and the micro/nano modulation structures at different positions
  • the incident light has different spectral modulation effects, which together constitute the optical filter layer.
  • the modulation intensity of these micro-nano modulation structures to different wavelength components of incident light corresponds to the connection intensity (linear layer weight) of the artificial neural network.
  • the optical filter layer constitutes the input layer and the linear layer of the artificial neural network and the connection weight between the input layer and the linear layer.
  • the optical filter layer weights the input signal in the spectrum, and then the image sensor converts the weighted signal into The electrical signal (this part of the image sensor processing is equivalent to the first nonlinear activation function), and then the electrical signals output by the image sensors at different positions are fully connected through the processor, and the complete nonlinear activation function is realized through the second nonlinear activation function.
  • Optical artificial neural network For environmental pollution detection, a large number of environmental pollutant samples can be collected first, and the weight of the linear layer, that is, the system function of the optical filter layer, can be obtained through data training, and the required optical filter layer can be reversely designed and integrated.
  • the output of the optical filter layer is used to further train and optimize the weight of the fully connected layer of the electrical signal, and then a high-accuracy optical artificial neural network can be realized. Rapid and accurate identification and qualitative analysis of environmental pollutant samples.
  • the modulation intensity (transmittance) of the modulation structure for different wavelength components of the incident light can be obtained, which is used as the input layer of the artificial neural network to the input layer.
  • the connection weight of the linear layer, and the nonlinear activation function is implemented in the processor. After collecting a large number of environmental pollutant samples and performing data training in advance, the required micro-nano modulation structure can be designed and prepared, and the artificial The input layer, linear layer and part of the nonlinear activation function of the neural network.
  • the image sensor 2 may use a CIS wafer, and the optical filter layer 1 is directly fabricated on the CIS wafer.
  • the optical filter layer 1 contains multiple repetitive modulation units, and each modulation unit contains 4 different continuous non-periodic structure arrays.
  • the neural network data training design is usually an irregularly shaped structure.
  • Each aperiodic structure array has different broad-spectrum modulation effects on incident light, and the overall size of each modulation unit ranges from 0.5 ⁇ m 2 to 40000 ⁇ m 2 .
  • the dielectric material in the optical filter layer 1 is polysilicon, and the thickness is 50 nm to 2 ⁇ m.
  • the CIS wafer includes a silicon detector layer and a metal wire layer, and the response range is in the visible to near-infrared band; the CIS wafer is bare, and the upper Bayer filter array and microlens array are not prepared.
  • Each modulation unit corresponds to a plurality of sensor units on the CIS wafer.
  • the complete process for the environmental detection chip to detect pollutants is as follows: as shown in Figure 16, the light source under the detection instrument illuminates the detection sample, and then the reflected light is collected by the chip, and the identification result can be obtained after processing.
  • both the optical filter layer and the image sensor can be manufactured by the semiconductor CMOS integration process, and the monolithic integration is realized at the wafer level, which is beneficial to reduce the distance between the sensor and the optical filter layer, reduce the volume of the device, and reduce the packaging cost , while enabling portable detection.
  • the artificial neural network is partially embedded in the image sensor containing various optical filter layers to achieve safe, reliable, fast and accurate environmental pollution detection;
  • the detectable samples include but are not limited to air, water, soil, and the artificial neural network is introduced Training to identify pollutants, large detection range, abundant samples, high recognition accuracy, and accurate qualitative analysis;
  • C. The chip can be prepared by one-time tape-out through CMOS process, which is conducive to reducing the failure rate of the device and improving the quality of the finished product of the device. rate and reduce costs.
  • Monolithic integration at the wafer level can minimize the distance between the sensor and the optical filter layer, which is conducive to reducing the size of the unit, reducing device volume and packaging costs.
  • optical artificial neural network environmental protection monitoring chip reference may be made to the introduction of the optical artificial neural network chip in the foregoing embodiment, which is not described here to avoid repetition.
  • preparation method of the optical artificial neural network environmental protection monitoring chip reference may also be made to the introduction of the optical artificial neural network chip preparation method in the foregoing embodiments, which will not be repeated here.
  • fingerprint identification technology is a biometric identification technology, which is widely used in smartphone unlocking, access control systems, bank password verification and other fields.
  • the main process of fingerprint recognition includes fingerprint collection, fingerprint preprocessing, fingerprint feature extraction and comparison.
  • the acquisition of fingerprint images is the key to fingerprint recognition.
  • the main methods of acquiring fingerprint images include optical acquisition, capacitive sensor acquisition, thermal sensor acquisition, and ultrasonic acquisition.
  • the capacitive sensor is usually placed under the back of the mobile phone to collect fingerprints. When combined with the thermal sensor, it can also achieve liveness detection, but it cannot be used for fingerprint recognition under the screen. This is because the thickness of the screen module limits the signal acquisition of the capacitive sensor.
  • the existing under-screen fingerprint identification technologies mainly include optical acquisition and ultrasonic acquisition, but these two schemes only collect the texture image information of the fingerprint, which limits the accuracy of fingerprint identification, and the volume and power consumption of the device are still relatively large. .
  • certain groups of people have few fingerprint features and are not easy to identify; fingerprints between relatives have similarities, which can easily lead to identification errors; fingerprint information left on the surface of objects may be stolen and the security is not high. Therefore, it is necessary to combine fingerprint information with other information to improve the accuracy and security of identification. On the whole, it is of great significance to realize a fast under-screen fingerprint recognition device with high accuracy, small size, low power consumption, safety and reliability.
  • this embodiment provides a new type of optoelectronic chip for accurate fingerprint identification.
  • the chip is composed of an optical filter layer to form an input layer and a linear layer of the optical artificial neural network.
  • the first nonlinear activation function of the optical artificial neural network is formed by an image sensor.
  • the embodiment of the present application also provides an optical artificial neural network fingerprint identification chip, which is used for fingerprint identification processing tasks, including: an optical filter layer, an image sensor and a processor; the optical filter layer corresponds to the input layer of the artificial neural network , the linear layer and the connection weight from the input layer to the linear layer, the square detection response of the image sensor corresponds to the first nonlinear activation function in the nonlinear layer of the artificial neural network; the processor corresponds to the artificial neural network The full connection and the output layer of the network, or the processor corresponds to the full connection of the artificial neural network, the second nonlinear activation function in the nonlinear layer, and the output layer;
  • the optical filter layer is disposed on the surface of the image sensor, and the optical filter layer includes a light modulation structure, and the light modulation structure is used for the incident light entering into different positions of the light modulation structure.
  • different spectral modulations to obtain incident light-carrying information corresponding to different positions on the surface of the image sensor;
  • the incident light-carrying information includes light intensity distribution information, spectral information, angle information of the incident light, and the Phase information of incident light;
  • the incident light includes reflected light, transmitted light and/or radiated light of the user's fingerprint;
  • the image sensor converts the information carried by the incident light corresponding to the different position points after being modulated by the optical filter layer to the electrical signals corresponding to the different position points after the first nonlinear activation processing through the square detection response, and will be different from the different position points. sending the electrical signal corresponding to the location point to the processor;
  • the processor performs full connection processing on the electrical signals corresponding to different position points, or the processor performs full connection processing and second nonlinear activation processing on the electrical signals corresponding to different position points to obtain fingerprint identification processing. result.
  • this embodiment implements a brand-new optical artificial neural network fingerprint identification chip capable of realizing the function of artificial neural network, which is used for fingerprint identification tasks.
  • the optical filter layer above is used as the input layer and linear layer of the artificial neural network, and the filtering effect of the optical filter layer on the hardware chip on the incident light is used as the connection weight between the input layer and the linear layer, and the image sensor on the hardware chip is used.
  • the square detection response is used as the first nonlinear activation function in the nonlinear layer of the artificial neural network.
  • the image information, spectral information, angle information of incident light and phase information of incident light at different points in the fingerprint space are injected into the In the pre-trained hardware chip, the artificial neural network analysis is performed on the spatial spectral information of the fingerprint through the hardware chip to obtain the fingerprint identification result. It should be noted that the embodiment of the present application achieves low power consumption, safe, reliable, fast and accurate. Fingerprint recognition.
  • the hardware structure on it - the optical filter layer corresponds to the input layer and the linear layer of the artificial neural network
  • the hardware structure on it - the image sensor corresponds to the artificial neural network.
  • a part of the nonlinear layer; the processor corresponds to another part of the nonlinear layer of the artificial neural network and the output layer.
  • the optical filter layer is disposed on the surface of the image sensor, and the optical filter layer includes a light modulation structure, and the light modulation structure is used to perform different spectral modulations on the incident light entering different positions of the light modulation structure, so as to Information carried by the incident light corresponding to different positions is obtained on the surface of the image sensor.
  • the modulation effect of the light modulation structure on the optical filter layer on the incident light is equivalent to the connection between the input layer and the linear layer. Weights.
  • the image sensor converts the incident light-carrying information corresponding to the different position points after being modulated by the optical filter layer for the first time, and then converts it into the information corresponding to the different position points after the first nonlinear activation processing through the square detection response.
  • the electrical signals corresponding to different positions are sent to the processor, and the processor performs full connection processing on the electrical signals corresponding to different positions, or the processor sends the electrical signals corresponding to different positions.
  • the full connection processing and the second nonlinear activation processing are performed to obtain the output signal of the artificial neural network.
  • the optical filter layer and the image sensor implemented in hardware replace or The related functions of the input layer, linear layer and partial nonlinear activation function in the existing artificial neural network are realized, that is, the input layer, linear layer and Part or all of the nonlinear activation functions are stripped, and the input layer, linear layer, and part or all of the nonlinear activation functions in the artificial neural network are realized by hardware, so that the optical artificial neural network can be used in the subsequent use.
  • the fingerprint recognition chip When the fingerprint recognition chip performs artificial neural network intelligent processing, it does not need to perform complex signal processing and algorithm processing corresponding to the input layer, linear layer and some or all of the nonlinear activation functions.
  • the processor only needs to perform full connection processing or full connection with the electrical signal and the second nonlinear activation processing, which can greatly reduce the power consumption and delay of the artificial neural network processing. It can be seen that the embodiments of the present application can not only omit the complex signal processing and algorithm processing corresponding to the input layer, the linear layer and a part of nonlinear activation functions in the prior art, but also the embodiments of the present application actually use the fingerprints at the same time.
  • Image information, spectral information, angle of incident light, and phase information of incident light that is, the information carried by incident light at different points in the fingerprint space. It can be seen that since the information carried by incident light at different points in the fingerprint space covers the image of the fingerprint , composition, shape, three-dimensional depth, structure and other information, so that when the identification processing is carried out according to the information carried by the incident light at different points in the fingerprint space, it can cover the image, composition, shape, three-dimensional depth, structure and other multi-dimensional information of the fingerprint. Thus, fingerprint identification can be performed accurately.
  • the optical artificial neural network fingerprint identification chip includes a trained light modulation structure, an image sensor and a processor;
  • the trained light modulation structures, image sensors and processors refer to using the input training samples and output training samples corresponding to the fingerprint recognition processing tasks to perform data analysis on different light modulation structures, image sensors, and fully connected
  • the input training samples include incident light reflected, transmitted and/or radiated by different human fingerprints; and the output training samples include corresponding fingerprint identification results.
  • the fingerprints of a large number of people can be collected first, and the weight from the input layer to the linear layer, that is, the system function of the optical filter layer, can be obtained through data training, and then the required design can be reversed. , which is integrated over the image sensor.
  • the output of the completed optical filter layer is used to further train and optimize the weight of the fully connected layer of the electrical signal, and then a high-accuracy optical artificial neural network can be realized. Fast and accurate identification of the user's fingerprint.
  • the specific modulation pattern of the modulation structure on the optical filter layer is obtained by collecting the fingerprints of a large number of people in the early stage and training and designing the artificial neural network data. It is usually an irregular shape structure, and of course it may be a regular shape. Structure.
  • the complete process of fingerprint recognition under the screen is: the light source under the screen of the mobile phone is irradiated on the user's finger, and then the reflected light is collected by the chip, and the recognition result is obtained after internal processing.
  • the chip actually utilizes the image information and spectral information of the fingerprint at the same time, which improves the accuracy and security of fingerprint identification.
  • the chip partially realizes artificial neural network in hardware, which improves the speed of fingerprint recognition.
  • the chip solution can use the existing CMOS process to achieve mass production, which reduces the size, power consumption and cost of the device.
  • the different light modulation structures are designed and realized by adopting computer optical simulation design.
  • the light modulation structures in the optical filter layer include regular structures and/or irregular structures; and/or, the light modulation structures in the optical filter layer include discrete structures and/or continuous structures.
  • the optical filter layer has a single-layer structure or a multi-layer structure.
  • the light modulation structure in the optical filter layer includes a unit array composed of a plurality of micro/nano units, each micro/nano unit corresponds to one or more pixels on the image sensor; the structures of each micro/nano unit are the same or different.
  • micro-nano unit includes a regular structure and/or an irregular structure; and/or, the micro-nano unit includes a discrete structure and/or a continuous structure.
  • micro-nano unit includes a plurality of groups of micro-nano structure arrays, and the structures of each group of micro-nano structure arrays are the same or different.
  • each group of micro-nano structure arrays has the function of broadband filtering or narrow-band filtering.
  • each group of micro-nano structure arrays is a periodic structure array or a non-periodic structure array.
  • micro-nano unit has polarization independent properties.
  • micro-nano unit has quadruple rotational symmetry.
  • the filter layer is made of one or more of semiconductor materials, metal materials, liquid crystals, quantum dot materials, and perovskite materials; and/or, the filter layer is made of photonic crystal, ultra A filter layer prepared by one or more of surface, random structure, nanostructure, metal surface plasmon SPP micro-nano structure, and tunable Fabry-Perot resonant cavity.
  • the semiconductor material includes one or more of silicon, silicon oxide, silicon nitride, titanium oxide, composite materials mixed in a preset ratio, and direct bandgap compound semiconductor materials; and/or, the nanometer
  • the structure includes one or more of nanodot 2D materials, nanopillar 2D materials, and nanowire 2D materials.
  • the thickness of the optical filter layer is 0.1 ⁇ ⁇ 10 ⁇ , where ⁇ represents the center wavelength of the incident light.
  • Embodiments of the present application further provide a fingerprint identification device, including the optical artificial neural network fingerprint identification chip described in the above embodiments.
  • the fingerprint identification device may be a portable fingerprint identification device, or a fingerprint identification device installed in a fixed position.
  • the embodiment of the present application also provides a preparation method of an optical artificial neural network fingerprint identification chip as described in any of the above, including:
  • the optical filter layer is used to perform different spectral modulation on the incident light entering different positions of the optical modulation structure through the optical modulation structure, so as to obtain different spectral modulations on the surface of the image sensor.
  • the image sensor converts the information carried by the incident light corresponding to the different position points after being modulated by the optical filter layer to the electrical signals corresponding to the different position points after the first nonlinear activation processing through the square detection response, and will be different from the different position points. sending the electrical signal corresponding to the location point to the processor;
  • the processor performs full connection processing on the electrical signals corresponding to different position points, or the processor performs full connection processing and second nonlinear activation processing on the electrical signals corresponding to different position points to obtain fingerprint identification processing. result.
  • the preparation method of the optical artificial neural network fingerprint identification chip also includes: a training process for the optical artificial neural network fingerprint identification chip, specifically including:
  • training optical artificial neural network fingerprint recognition chips including different optical modulation structures, image sensors and processors with different fully connected parameters.
  • the light modulation structure, image sensor and processor that meet the training convergence condition are trained as the trained light modulation structure, image sensor and processor;
  • the optical artificial neural network fingerprint recognition chip is trained to obtain a light modulation structure, an image sensor and a processor that meet the training convergence conditions, and the light modulation structure, image sensor and processor that meet the training convergence conditions are used as the trained light modulation structure, image sensor and processor.
  • Image sensor and processor is
  • the fingerprint identification chip under the optical artificial neural network screen based on the micro-nano modulation structure and image sensor provided in this embodiment has the following effects: A.
  • the artificial neural network is partially embedded in the hardware chip to achieve safe, reliable, fast and accurate under-screen fingerprint recognition.
  • B. The preparation of the chip can be completed by one tap-out of the CMOS process, which is beneficial to reduce the failure rate of the device, improve the yield of the device, and reduce the cost.
  • C. Monolithic integration at the wafer level can minimize the distance between the sensor and the optical filter layer, which is conducive to reducing the size of the unit, reducing device volume and packaging costs.
  • optical artificial neural network fingerprint identification chip it should be noted that, for the detailed description of the structure of the optical artificial neural network fingerprint identification chip provided in this embodiment, reference may be made to the introduction of the optical artificial neural network chip in the foregoing embodiment. In addition, for the detailed introduction of the optical artificial neural network fingerprint identification chip preparation method, you can also refer to the introduction of the optical artificial neural network chip preparation method in the foregoing embodiment, which will not be repeated here.
  • face recognition technology is a biometric identification technology, which is widely used in access control and attendance systems, criminal investigation systems, e-commerce and other fields.
  • the main process of face recognition includes face image collection, preprocessing, feature extraction, matching and recognition.
  • face image collection preprocessing
  • feature extraction feature extraction
  • matching recognition
  • this embodiment provides a new type of optoelectronic chip for accurate face recognition, the chip is composed of an optical filter layer.
  • the first nonlinear activation function of the optical artificial neural network is formed by the image sensor.
  • the embodiment of the present application also provides an optical artificial neural network face recognition chip, which is used for face recognition processing tasks, including: an optical filter layer, an image sensor and a processor; the optical filter layer corresponds to the artificial neural network.
  • the input layer, the linear layer, and the connection weight from the input layer to the linear layer, the square detection response of the image sensor corresponds to the first nonlinear activation function in the nonlinear layer of the artificial neural network;
  • the processor corresponds to The full connection and the output layer of the artificial neural network, or, the processor corresponds to the full connection of the artificial neural network, the second nonlinear activation function in the nonlinear layer, and the output layer;
  • the optical filter layer is disposed on the surface of the image sensor, and the optical filter layer includes a light modulation structure, and the light modulation structure is used for the incident light entering into different positions of the light modulation structure.
  • different spectral modulations to obtain incident light-carrying information corresponding to different positions on the surface of the image sensor;
  • the incident light-carrying information includes light intensity distribution information, spectral information, angle information of the incident light, and the Phase information of incident light;
  • the incident light includes reflected light, transmitted light and/or radiated light of the user's face;
  • the image sensor converts the information carried by the incident light corresponding to the different position points after being modulated by the optical filter layer to the electrical signals corresponding to the different position points after the first nonlinear activation processing through the square detection response, and will be different from the different position points. sending the electrical signal corresponding to the location point to the processor;
  • the processor performs full connection processing on electrical signals corresponding to different position points, or the processor performs full connection processing and second nonlinear activation processing on electrical signals corresponding to different position points to obtain face recognition. process result.
  • this embodiment implements a brand-new optical artificial neural network face recognition chip capable of realizing the function of artificial neural network, which is used for the face recognition task.
  • the optical filter layer on the hardware chip is used as the input layer and linear layer of the artificial neural network, and the filtering effect of the optical filter layer on the hardware chip on the incident light is used as the connection weight from the input layer to the linear layer, and the image on the hardware chip is used.
  • the square detection response of the sensor is used as the first nonlinear activation function in the nonlinear layer of the artificial neural network.
  • the spatial spectral information of the face is injected into the pre-trained hardware chip, and the face is detected by the hardware chip.
  • the image information, spectral information, angle information of incident light and phase information of incident light at different points in space are analyzed by artificial neural network to obtain the face recognition result. It should be noted that the embodiment of the present application realizes low power consumption, Safe and reliable fast and accurate face recognition.
  • the hardware structure on it - the optical filter layer corresponds to the input layer and the linear layer of the artificial neural network
  • the hardware structure on it - the image sensor corresponds to the artificial neural network.
  • part of the nonlinear layer of the artificial neural network; the processor corresponds to another part of the nonlinear layer of the artificial neural network and the output layer.
  • the optical filter layer is disposed on the surface of the image sensor, and the optical filter layer includes a light modulation structure, and the light modulation structure is used to perform different spectral modulations on the incident light entering different positions of the light modulation structure, so as to Information carried by the incident light corresponding to different positions is obtained on the surface of the image sensor.
  • the modulation effect of the light modulation structure on the optical filter layer on the incident light is equivalent to the connection between the input layer and the linear layer. Weights.
  • the image sensor converts the incident light-carrying information corresponding to the different position points after being modulated by the optical filter layer for the first time, and then converts it into the information corresponding to the different position points after the first nonlinear activation processing through the square detection response.
  • the electrical signals corresponding to different positions are sent to the processor, and the processor performs full connection processing on the electrical signals corresponding to different positions, or the processor sends the electrical signals corresponding to different positions.
  • the full connection processing and the second nonlinear activation processing are performed to obtain the output signal of the artificial neural network. It can be seen that in the optical artificial neural network face recognition chip, the optical filter layer and the image sensor implemented in hardware are replaced by the optical filter layer and the image sensor. Or the related functions of the input layer, the linear layer and the partial nonlinear activation function in the existing artificial neural network are realized, that is, the input layer and the linear layer in the artificial neural network implemented by software in the prior art in the embodiment of the present application are implemented.
  • the network face recognition chip performs artificial neural network intelligent processing, it does not need to perform complex signal processing and algorithm processing corresponding to the input layer, linear layer and some or all of the nonlinear activation functions.
  • the processor in the chip can perform full connection processing or full connection with the electrical signal and the second nonlinear activation processing, which can greatly reduce the power consumption and delay of the artificial neural network processing.
  • the embodiment of the present application can not only save the complex signal processing and algorithm processing corresponding to the input layer, the linear layer and a part of the nonlinear activation function in the prior art, but also the embodiment of the present application actually uses the human face at the same time.
  • the image information, spectral information, angle of incident light and phase information of incident light that is, the information carried by incident light at different points in the face space
  • since the information carried by incident light at different points in the face space covers The image, composition, shape, three-dimensional depth, structure and other information of the face, so that the image, composition, shape, three-dimensional depth, structure of the face can be covered when the recognition processing is carried out according to the information carried by the incident light at different points in the face space. and other multi-dimensional information, so that face recognition can be performed accurately.
  • the optical artificial neural network face recognition chip includes a trained light modulation structure, an image sensor and a processor;
  • the trained light modulation structure, image sensor and processor refers to using the input training samples and output training samples corresponding to the face recognition processing task to perform the training of the light modulation structure, image sensor and the image sensor with different full range.
  • a light modulation structure, an image sensor and a processor that satisfy the training convergence conditions obtained by training an optical artificial neural network face recognition chip connected to a processor with parameters; or, the trained light modulation structure, image sensor and processor are Refers to using the input training samples and output training samples corresponding to the face recognition processing task, to the processor including different light modulation structures, image sensors, and different fully connected parameters and different second nonlinear activation parameters.
  • the optical modulation structure, image sensor and processor that meet the training convergence conditions are obtained by training the optical artificial neural network face recognition chip;
  • the input training samples include incident light reflected, transmitted and/or radiated by different faces; and the output training samples include corresponding face recognition results.
  • the different light modulation structures are designed and realized by adopting computer optical simulation design.
  • an artificial neural network photoelectric chip whose input is an object image and its frequency spectrum can be realized, and fast, accurate, safe and reliable living face recognition can be realized.
  • the faces of a large number of people can be collected first, and the weight from the input layer to the linear layer, that is, the system function of the optical filter layer, can be obtained through data training, and then the required design can be reversed. , which is integrated over the image sensor.
  • the face samples to be recognized and the output of the completed optical filter layer to further train and optimize the weight of the fully connected layer of the electrical signal, a high-accuracy optical artificial neural network can be realized. Fast and accurate recognition of user faces.
  • the specific modulation pattern of the modulation structure on the optical filter layer is obtained by collecting the faces of a large number of people in the early stage and training and designing the artificial neural network data. shape structure.
  • the complete process of face recognition is: ambient light or other light sources illuminate the user's face, and then the reflected light is collected by the chip, and the recognition result is obtained after internal processing.
  • the chip actually utilizes both the image information and spectral information of the face, which improves the accuracy and security of face recognition, especially for non-living face models, which can also be accurately excluded.
  • the chip partially realizes artificial neural network in hardware, which improves the speed of fingerprint recognition.
  • the chip solution can use the existing CMOS process to achieve mass production, which reduces the size, power consumption and cost of the device.
  • the light modulation structures in the optical filter layer include regular structures and/or irregular structures; and/or, the light modulation structures in the optical filter layer include discrete structures and/or continuous structures.
  • the optical filter layer has a single-layer structure or a multi-layer structure.
  • the light modulation structure in the optical filter layer includes a unit array composed of a plurality of micro/nano units, each micro/nano unit corresponds to one or more pixels on the image sensor; the structures of each micro/nano unit are the same or different.
  • micro-nano unit includes a regular structure and/or an irregular structure; and/or, the micro-nano unit includes a discrete structure and/or a continuous structure.
  • micro-nano unit includes multiple groups of micro-nano structure arrays, and the structures of each group of micro-nano structure arrays are the same or different.
  • each group of micro-nano structure arrays has the function of broadband filtering or narrow-band filtering.
  • each group of micro-nano structure arrays is a periodic structure array or a non-periodic structure array.
  • micro-nano unit has polarization-independent properties, and in particular, the micro-nano unit has quadruple rotational symmetry.
  • the filter layer is made of one or more of semiconductor materials, metal materials, liquid crystals, quantum dot materials, and perovskite materials; and/or, the filter layer is made of photonic crystal, ultra A filter layer prepared by one or more of surface, random structure, nanostructure, metal surface plasmon SPP micro-nano structure, and tunable Fabry-Perot resonant cavity.
  • the semiconductor material includes one or more of silicon, silicon oxide, silicon nitride, titanium oxide, composite materials mixed in a preset ratio, and direct bandgap compound semiconductor materials; and/or, the nanometer
  • the structure includes one or more of nanodot 2D materials, nanopillar 2D materials, and nanowire 2D materials.
  • the thickness of the optical filter layer is 0.1 ⁇ ⁇ 10 ⁇ , where ⁇ represents the center wavelength of the incident light.
  • the embodiments of the present application further provide a face recognition device, including the optical artificial neural network face recognition chip described in the above embodiments.
  • the face recognition device may be a portable face recognition device or a face recognition device installed in a fixed position. Since the face recognition device has similar beneficial effects to the above-mentioned optical artificial neural network face recognition chip, it will not be repeated here.
  • the embodiment of the present application also provides a preparation method of the optical artificial neural network face recognition chip as described above, including:
  • the optical filter layer is used to perform different spectral modulation on the incident light entering different positions of the optical modulation structure through the optical modulation structure, so as to obtain different spectral modulations on the surface of the image sensor.
  • the image sensor converts the information carried by the incident light corresponding to the different position points after being modulated by the optical filter layer to the electrical signals corresponding to the different position points after the first nonlinear activation processing through the square detection response, and will be different from the different position points. sending the electrical signal corresponding to the location point to the processor;
  • the processor performs full connection processing on electrical signals corresponding to different position points, or the processor performs full connection processing and second nonlinear activation processing on electrical signals corresponding to different position points to obtain face recognition. process result.
  • the preparation method of the optical artificial neural network face recognition chip also includes: a training process for the optical artificial neural network face recognition chip, specifically including:
  • the optical artificial neural network face recognition chips including different light modulation structures, image sensors and processors with different fully connected parameters are trained Obtain the light modulation structure, image sensor and processor that satisfy the training convergence condition, and use the light modulation structure, image sensor and processor that satisfy the training convergence condition as the trained light modulation structure, image sensor and processor;
  • the optical artificial neural network face recognition chip of the device is trained to obtain a light modulation structure, an image sensor and a processor that meet the training convergence conditions, and the light modulation structure, image sensor and processor that meet the training convergence conditions are used as the trained light modulation structure. Structures, Image Sensors and Processors.
  • the different light modulation structures are designed and realized by adopting computer optical simulation design.
  • the optical artificial neural network face recognition chip based on the micro-nano modulation structure and image sensor provided in this embodiment has the following effects: A. The artificial neural network is partially embedded in the hardware chip to realize safe, reliable, fast and accurate face recognition. B. The preparation of the chip can be completed in one tape-out through the CMOS process, which is beneficial to reduce the failure rate of the device, improve the yield rate of the device, and reduce the cost. C. Monolithic integration at the wafer level can minimize the distance between the sensor and the optical filter layer, which is conducive to reducing the size of the unit, reducing device volume and packaging costs.
  • optical artificial neural network face recognition chip it should be noted that, for the detailed structural description of the optical artificial neural network face recognition chip provided in this embodiment, reference may be made to the introduction of the optical artificial neural network chip in the foregoing embodiment, which is not described here to avoid repetition. In addition, for the detailed introduction of the optical artificial neural network face recognition chip preparation method, you can also refer to the introduction of the optical artificial neural network chip preparation method in the foregoing embodiment, which will not be repeated here.
  • Machine vision technology is a branch of artificial intelligence. It uses machines to replace human eyes for observation and judgment. It is widely used in industrial production, quality inspection, express sorting, driverless and other fields.
  • a typical machine vision system includes imaging system, image processing system, communication and IO system and linkage mechanism. Among them, the imaging system is responsible for collecting the image information of the target object, which is the key to the machine vision technology.
  • machine vision technology only uses the image information of objects, and the accuracy and reliability of measurement and recognition need to be improved. Therefore, it is of great significance to use information from other dimensions of objects to achieve enhanced machine vision with higher accuracy and reliability.
  • this embodiment provides a new type of optoelectronic chip for enhancing machine vision
  • the chip is composed of an optical filter layer to form the input layer and linear layer of the optical artificial neural network.
  • the first nonlinear activation function of the optical artificial neural network is formed by an image sensor.
  • the embodiment of the present application also provides an optical artificial neural network machine vision enhancement chip, which is used for machine vision intelligent processing tasks, including: an optical filter layer, an image sensor and a processor; the optical filter layer corresponds to the artificial neural network.
  • the input layer, the linear layer, and the connection weight from the input layer to the linear layer, the square detection response of the image sensor corresponds to the first nonlinear activation function in the nonlinear layer of the artificial neural network;
  • the processor corresponds to The full connection and the output layer of the artificial neural network, or, the processor corresponds to the full connection of the artificial neural network, the second nonlinear activation function in the nonlinear layer, and the output layer;
  • the optical filter layer is disposed on the surface of the image sensor, and the optical filter layer includes a light modulation structure, and the light modulation structure is used for the incident light entering into different positions of the light modulation structure.
  • different spectral modulations to obtain incident light-carrying information corresponding to different positions on the surface of the image sensor;
  • the incident light-carrying information includes light intensity distribution information, spectral information, angle information of the incident light, and the Phase information of incident light;
  • the incident light includes reflected light, transmitted light and/or radiated light of a target object in a machine vision scene;
  • the image sensor converts the information carried by the incident light corresponding to the different position points after being modulated by the optical filter layer to the electrical signals corresponding to the different position points after the first nonlinear activation processing through the square detection response, and will be different from the different position points. sending the electrical signal corresponding to the location point to the processor;
  • the processor performs full connection processing on the electrical signals corresponding to different position points, or the processor performs full connection processing and second nonlinear activation processing on the electrical signals corresponding to different position points to obtain machine vision intelligence. process result;
  • the machine vision intelligent processing task includes the recognition and/or qualitative analysis of the target object in the machine vision scene; the machine vision intelligent processing result includes the recognition result and/or the qualitative analysis result of the target object in the machine vision scene.
  • this embodiment implements a new optical artificial neural network enhanced machine vision recognition chip capable of realizing the function of artificial neural network, which is used for target object recognition tasks in various machine vision application scenarios.
  • the artificial neural network is embedded on the chip, and the optical filter layer on the hardware chip is used as the input layer and linear layer of the artificial neural network, and the filtering effect of the optical filter layer on the hardware chip on the incident light is used as the input layer to the linear layer.
  • Image information, spectral information, angle information of incident light, and phase information of incident light are incident on the pre-trained hardware chip, and the hardware chip can analyze the image information, spectral information, angle information of incident light and The phase information of the incident light is analyzed by an artificial neural network to obtain the recognition result of the target object. It should be noted that the embodiments of the present application realize fast and accurate object recognition with low power consumption, safety and reliability.
  • the hardware structure on it - the optical filter layer corresponds to the input layer and the linear layer of the artificial neural network
  • the hardware structure on it - the image sensor corresponds to the artificial neural network.
  • part of the nonlinear layer of the artificial neural network; the processor corresponds to another part of the nonlinear layer of the artificial neural network and the output layer.
  • the optical filter layer is disposed on the surface of the image sensor, and the optical filter layer includes a light modulation structure, and the light modulation structure is used to perform different spectral modulation on the information carried by the incident light entering different positions of the light modulation structure. , in order to obtain the incident light-carrying information corresponding to different positions on the surface of the image sensor.
  • the modulation effect of the light modulation structure on the optical filter layer on the incident light is equivalent to the input layer to the linear layer. connection weight.
  • the image sensor converts the incident light-carrying information corresponding to the different position points after being modulated by the optical filter layer for the first time, and then converts it into the information corresponding to the different position points after the first nonlinear activation processing through the square detection response.
  • the electrical signals corresponding to different positions are sent to the processor, and the processor performs full connection processing on the electrical signals corresponding to different positions, or the processor sends the electrical signals corresponding to different positions.
  • the full connection processing and the second nonlinear activation processing are performed to obtain the output signal of the artificial neural network.
  • the optical filter layer and the image sensor implemented in hardware are replaced by the optical filter layer and the image sensor.
  • the related functions of the input layer, the linear layer and the partial nonlinear activation function in the existing artificial neural network are realized, that is, the input layer and the linear layer in the artificial neural network implemented by software in the prior art in the embodiment of the present application are implemented.
  • some or all of the nonlinear activation functions are peeled off, and the input layer, linear layer, and some or all of the nonlinear activation functions in the artificial neural network are realized by hardware, so that the chip can be used in the future.
  • the artificial neural network does not need to perform complex signal processing and algorithm processing corresponding to the input layer, linear layer, and some or all of the nonlinear activation functions. It only needs to be enhanced by the optical artificial neural network in the machine vision chip.
  • the processor only needs to perform full connection processing or full connection with the electrical signal and the second nonlinear activation processing, which can greatly reduce the power consumption and delay of the artificial neural network processing. It can be seen that the embodiments of the present application can not only omit the complex signal processing and algorithm processing corresponding to the input layer, the linear layer and a part of nonlinear activation functions in the prior art, but also the embodiments of the present application actually use machine vision at the same time.
  • the image information, spectral information, angle of incident light, and phase information of the incident light of the target object in the application scenario that is, the information carried by the incident light at different points in space of the target object in the machine vision application scenario.
  • the information carried by the incident light at different points in the object space covers the image, composition, shape, three-dimensional depth, structure and other information of the target object, so that when the recognition processing is performed according to the information carried by the incident light at different points in the target object space, the target can be covered.
  • the image, composition, shape, three-dimensional depth, structure and other multi-dimensional information of the object can be accurately recognized.
  • the optical artificial neural network machine vision enhancement chip includes a trained light modulation structure, an image sensor and a processor;
  • the trained light modulation structures, image sensors and processors refer to using the input training samples and output training samples corresponding to the machine vision intelligent processing tasks to perform data analysis on the light modulation structures, image sensors and different full-scale light modulation
  • a light modulation structure, an image sensor and a processor that meet the training convergence conditions obtained by training an optical artificial neural network machine vision enhancement chip connected to a processor with parameters; or, the trained light modulation structure, image sensor and processor are Refers to the use of input training samples and output training samples corresponding to the machine vision intelligent processing task, to include different light modulation structures, image sensors and processors with different fully connected parameters and different second nonlinear activation parameters.
  • the optical modulation structure, image sensor and processor that meet the training convergence conditions are obtained by training the optical artificial neural network machine vision enhancement chip;
  • the input training sample includes incident light reflected, transmitted and/or radiated by a target object in a specific machine vision scene
  • the output training sample includes a target object recognition result in a specific machine vision scene
  • all The input training samples include incident light reflected, transmitted and/or radiated by a target object in a specific machine vision scene
  • the output training samples include qualitative analysis results of the target object in a specific machine vision scene.
  • an artificial neural network optoelectronic chip whose input is an object image and its frequency spectrum can be realized, and an accurate, safe and reliable machine vision target object recognition can be realized.
  • connection weight from the input layer to the linear layer can be obtained through data training, that is, the system function of the optical filter layer, and the required optical filter layer can be designed in reverse. , integrated on top of the image sensor, and then an optical artificial neural network enhanced machine vision chip that can quickly and accurately identify and judge in machine vision application scenarios can be prepared.
  • the specific modulation pattern of the modulation structure on the optical filter layer is obtained by collecting a large number of target objects in the corresponding machine vision application scene in the early stage, and is designed by artificial neural network data training. It is usually an irregular shape structure, of course, there are also May be regular shaped structures.
  • the complete process for enhanced machine vision application is: the light source illuminates the detected object, then the reflected light is collected by the chip, and then processed by the processor algorithm to obtain the recognition result, and finally the control mechanism does Take corresponding action.
  • the chip actually uses the image information and spectral information of the detected object at the same time, which improves the accuracy and security of the detected object recognition.
  • the chip solution can use the existing CMOS process to achieve mass production, which reduces the size, power consumption and cost of the device.
  • the different light modulation structures are designed and realized by adopting computer optical simulation design.
  • the light modulation structures in the optical filter layer include regular structures and/or irregular structures; and/or, the light modulation structures in the optical filter layer include discrete structures and/or continuous structures.
  • the optical filter layer has a single-layer structure or a multi-layer structure.
  • the light modulation structure in the optical filter layer includes a unit array composed of a plurality of micro/nano units, each micro/nano unit corresponds to one or more pixels on the image sensor; the structures of each micro/nano unit are the same or different.
  • micro-nano unit includes a regular structure and/or an irregular structure; and/or, the micro-nano unit includes a discrete structure and/or a continuous structure.
  • micro-nano unit includes a plurality of groups of micro-nano structure arrays, and the structures of each group of micro-nano structure arrays are the same or different.
  • each group of micro-nano structure arrays has the function of broadband filtering or narrow-band filtering.
  • each group of micro-nano structure arrays is a periodic structure array or a non-periodic structure array.
  • micro-nano unit has polarization independent properties.
  • micro-nano unit has quadruple rotational symmetry.
  • the filter layer is made of one or more of semiconductor materials, metal materials, liquid crystals, quantum dot materials, and perovskite materials; and/or, the filter layer is made of photonic crystal, ultra A filter layer prepared by one or more of surface, random structure, nanostructure, metal surface plasmon SPP micro-nano structure, and tunable Fabry-Perot resonant cavity.
  • the semiconductor material includes one or more of silicon, silicon oxide, silicon nitride, titanium oxide, composite materials mixed in a preset ratio, and direct bandgap compound semiconductor materials; and/or, the nanometer
  • the structure includes one or more of nanodot 2D materials, nanopillar 2D materials, and nanowire 2D materials.
  • the thickness of the optical filter layer is 0.1 ⁇ ⁇ 10 ⁇ , where ⁇ represents the center wavelength of the incident light.
  • the embodiments of the present application also provide an enhanced machine vision system, including a control mechanism and the optical artificial neural network enhanced machine vision chip as described in the above embodiments.
  • the control mechanism is connected with the optical artificial neural network enhanced machine vision chip, and the control mechanism performs corresponding control according to the recognition results of the artificial neural network enhanced machine vision chip, so as to complete the application target in the machine vision scene.
  • the control mechanism may be a manipulator, a manipulator, or an intelligent control button, which is not limited in this embodiment.
  • the control structure can perform corresponding control according to the predetermined control logic according to the recognition result of the optical artificial neural network enhanced machine vision chip.
  • machine vision technology is a branch of artificial intelligence. It uses machines to replace human eyes for observation and judgment. It is widely used in industrial production, quality inspection, express sorting, driverless and other fields.
  • a typical machine vision system includes imaging system, image processing system, communication and IO system and linkage mechanism. Among them, the imaging system is responsible for collecting the image information of the target object, which is the key to the machine vision technology.
  • machine vision technology only uses the image information of objects, and the accuracy and reliability of measurement and recognition need to be improved.
  • the present embodiment comprehensively utilizes the spectral information of the object, thereby realizing an enhanced machine vision system with higher accuracy and reliability.
  • the embodiment of the present application also provides a preparation method of the optical artificial neural network machine vision enhancement chip as described in any of the above, including:
  • the optical filter layer is used to perform different spectral modulation on the incident light entering different positions of the optical modulation structure through the optical modulation structure, so as to obtain different spectral modulations on the surface of the image sensor.
  • the image sensor converts the information carried by the incident light corresponding to the different position points after being modulated by the optical filter layer to the electrical signals corresponding to the different position points after the first nonlinear activation processing through the square detection response, and will be different from the different position points. sending the electrical signal corresponding to the location point to the processor;
  • the processor performs full connection processing on the electrical signals corresponding to different position points, or the processor performs full connection processing and second nonlinear activation processing on the electrical signals corresponding to different position points to obtain machine vision intelligence. process result.
  • the preparation method of the optical artificial neural network machine vision enhancement chip also includes: the training process of the optical artificial neural network machine vision enhancement chip, specifically including:
  • train the optical artificial neural network machine vision enhancement chip including different light modulation structures, image sensors and processors with different fully connected parameters Obtain the light modulation structure, image sensor and processor that satisfy the training convergence condition, and use the light modulation structure, image sensor and processor that satisfy the training convergence condition as the trained light modulation structure, image sensor and processor;
  • the optical artificial neural network machine vision enhancement chip of the device is trained to obtain a light modulation structure, an image sensor and a processor that meet the training convergence conditions, and the light modulation structure, image sensor and processor that meet the training convergence conditions are used as the trained light modulation structure. Structures, Image Sensors and Processors.
  • the different light modulation structures are designed and realized by adopting computer optical simulation design.
  • the machine vision system of unmanned driving it is necessary to automatically identify the type of obstacles in front of the vehicle, and then perform accurate and fast automatic control.
  • the machine vision system for quality inspection needs to be able to accurately and quickly identify the quality defects of the target object, so as to ensure the quality of the quality inspection and avoid missed or wrong inspections.
  • the requirement for real-time performance is very high, and the chip provided in this embodiment can well solve the problem of real-time performance.
  • the required optical filter layer can be reverse engineered to be integrated over the image sensor.
  • the optical artificial neural network enhanced machine vision chip based on the micro-nano modulation structure and image sensor has the following effects: A.
  • the artificial neural network is partially embedded in the hardware chip, which improves the performance of machine vision related applications. Real-time and reliable.
  • the preparation of the chip can be completed by one tap-out of the CMOS process, which is beneficial to reduce the failure rate of the device, improve the yield of the device, and reduce the cost.
  • C. Monolithic integration at the wafer level can minimize the distance between the sensor and the optical filter layer, which is conducive to reducing the size of the unit, reducing device volume and packaging costs.
  • optical artificial neural network enhanced machine vision chip for a detailed description of the structure of the optical artificial neural network enhanced machine vision chip provided in this embodiment, reference may be made to the introduction of the optical artificial neural network chip in the foregoing embodiment, which is not described here to avoid repetition.
  • optical artificial neural network enhanced machine vision chip preparation method please refer to the introduction of the optical artificial neural network chip preparation method in the foregoing embodiments, which will not be repeated here.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Software Systems (AREA)
  • Computing Systems (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • General Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • Biomedical Technology (AREA)
  • Artificial Intelligence (AREA)
  • Biophysics (AREA)
  • General Engineering & Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Mathematical Physics (AREA)
  • Computational Linguistics (AREA)
  • Neurology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Multimedia (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Evolutionary Biology (AREA)
  • Databases & Information Systems (AREA)
  • Medical Informatics (AREA)
  • Photometry And Measurement Of Optical Pulse Characteristics (AREA)
  • Image Analysis (AREA)

Abstract

本申请提供一种光人工神经网络智能芯片及制备方法，将光滤波器层作为人工神经网络的输入层和线性层，将光滤波器层对入射光的滤波作用作为输入层到线性层的连接权重，将图像传感器的平方检波响应作为人工神经网络的非线性层中的第一次非线性激活函数，将处理器作为人工神经网络的全连接、非线性层中的第二次非线性激活函数以及输出层，从而滤波器层和图像传感器以硬件的方式实现了人工神经网络中输入层、线性层和非线性激活函数的相关功能，使得后续在进行智能处理时不需要再进行与输入层和线性层对应的复杂的信号和算法处理，并利用了目标对象空间不同点处的图像、光谱、入射光角度和入射光相位信息，更加准确地实现对目标对象的智能处理。

Description

光人工神经网络智能芯片及制备方法
相关申请的交叉引用
本申请要求于2021年02月08日提交的申请号为202110172825.1，发明名称为“光人工神经网络智能芯片及制备方法”的中国专利申请的优先权，其通过引用方式全部并入本文。
技术领域
本申请涉及人工智能技术领域，尤其涉及一种光人工神经网络智能芯片及制备方法。
背景技术
现有的智能识别技术，通常需要先对人或物体成像，再经过图像的预处理、特征提取、特征匹配等步骤，实现对人或物体的识别。然而，只利用人或物体的二维图像信息难以保证识别的准确性，例如难以区分真实的人脸和人脸照片；并且，成像过程需要将光学信息转换为数字电子信号，再传输到计算机中进行后续的算法处理，大量数据的传输和处理造成了较大的功耗和延时。
发明内容
针对现有技术存在的问题，本申请实施例提供一种光人工神经网络智能芯片及制备方法。
具体地，本申请实施例提供了如下技术方案：
第一方面，本申请实施例提供了一种光人工神经网络智能芯片，包括：光滤波器层、图像传感器和处理器；所述光滤波器层对应人工神经网络的输入层、线性层以及所述输入层到所述线性层的连接权重，所述图像传感器的平方检波响应对应人工神经网络的非线性层中的第一次非线性激活函数；所述处理器对应人工神经网络的全连接以及输出层，或，所述处理器对应人工神经网络的全连接、非线性层中的第二次非线性激活函数以及输出层；
所述光滤波器层设置于所述图像传感器表面，所述光滤波器层包含有光调制结构，所述光调制结构用于对进入至所述光调制结构不同位置点处的入射光分别进行不同的频谱调制，以在所述图像传感器的表面得到与不同位置点对应的入射光携带信息；所述入射光携带信息包括光强度分布信息、光谱信息、所述入射光的角度信息以及所述入射光的相位信息；
所述图像传感器通过平方检波响应将与不同位置点经光滤波器层调制后对应的入射光携带信息进行第一次非线性激活处理后转换为与不同位置点对应的电信号，并将与不同位置点对应的电信号发送给所述处理器；
所述处理器将与不同位置点对应的电信号进行全连接处理，或，所述处理器将与不同位置点对应的电信号进行全连接处理以及第二次非线性激活处理，得到人工神经网络的输出信号。
进一步地，所述光人工神经网络智能芯片用于目标对象的智能处理任务；所述智能处理任务至少包括智能感知、智能识别和智能决策任务中的一种或多种；
目标对象的反射光、透射光和/或辐射光进入至训练好的光人工神经网络智能芯片中，得到所述目标对象的智能处理结果；所述智能处理结果至少包括智能感知结果、智能识别结果和/或智能决策结果中的一种或多种；
其中，训练好的光人工神经网络智能芯片是指包括训练好的光调制结构、图像传感器和处理器的光人工神经网络智能芯片；
所述训练好的光调制结构、图像传感器和处理器是指利用与所述智能处理任务对应的输入训练样本和输出训练样本，对包含不同的光调制结构、图像传感器和具有不同的全连接参数的处理器的光人工神经网络智能芯片进行训练得到的满足训练收敛条件的光调制结构、图像传感器和处理器；或，所述训练好的光调制结构、图像传感器和处理器是指利用与所述智能处理任务对应的输入训练样本和输出训练样本，对包含不同的光调制结构、图像传感器和具有不同的全连接参数以及不同的第二次非线性激活参数的处理器的光人工神经网络智能芯片进行训练得到的满足训练收敛条件的光调制结构、图像传感器和处理器。
进一步地，在对包含不同的光调制结构、图像传感器和具有不同的全 连接参数的处理器的光人工神经网络智能芯片进行训练，或，对包含不同的光调制结构、图像传感器和具有不同的全连接参数以及不同的第二次非线性激活参数的处理器的光人工神经网络智能芯片进行训练时，所述不同的光调制结构通过采用计算机光学仿真设计的方式设计实现。
进一步地，所述光滤波器层中的光调制结构包含规则结构和/或不规则结构；和/或，所述光滤波器层中的光调制结构包含离散型结构和/或连续型结构。
进一步地，所述光滤波器层为单层结构或多层结构。
进一步地，所述光滤波器层中的光调制结构包含由多个微纳单元组成的单元阵列，每个微纳单元对应图像传感器上的一个或多个像素点；各个微纳单元的结构相同或不同。
进一步地，所述微纳单元包含规则结构和/或不规则结构；和/或，所述微纳单元包含离散型结构和/或连续型结构。
进一步地，所述微纳单元包含有多组微纳结构阵列，各组微纳结构阵列的结构相同或不同。
进一步地，各组微纳结构阵列具有宽带滤波或窄带滤波的作用。
进一步地，各组微纳结构阵列为周期结构阵列或非周期结构阵列。
进一步地，所述微纳单元包含的多组微纳结构阵列中有一组或多组空结构。
进一步地，所述微纳单元具有偏振无关特性。
进一步地，所述微纳单元具有四重旋转对称性。
进一步地，所述滤波器层是由半导体材料、金属材料、液晶、量子点材料、钙钛矿材料中的一种或多种制备；和/或，所述滤波器层是由光子晶体、超表面、随机结构、纳米结构、金属表面等离激元SPP微纳结构、可调法布里-珀罗谐振腔中的一种或多种制备的滤波器层。
进一步地，所述半导体材料包括硅、氧化硅、氮化硅、氧化钛、按照预设比例混合的复合材料以及直接带隙化合物半导体材料中的一种或多种；和/或，所述纳米结构包括纳米点二维材料、纳米柱二维材料和纳米线二维材料中的一种或多种。
进一步地，所述光滤波器层的厚度为0.1λ～10λ，其中λ表示入射光 的中心波长。
进一步地，所述图像传感器为下述中的任意一项或多项：
CMOS图像传感器CIS、电荷耦合元件CCD、单光子雪崩二极管SPAD阵列和焦平面光电探测器阵列。
第二方面，本申请实施例还提供了一种智能设备，包括：如第一方面所述的光人工神经网络智能芯片。
进一步地，所述智能设备包括智能手机、智能电脑、智能识别设备、智能感知设备和智能决策设备中的一种或多种。
第三方面，本申请实施例还提供了一种如第一方面所述的光人工神经网络智能芯片的制备方法，包括：
在所述图像传感器的表面制备包含有光调制结构的光滤波器层；
生成具备对信号进行全连接处理功能的处理器或生成具备对信号进行全连接处理以及第二次非线性激活处理功能的处理器；
连接所述图像传感器和所述处理器；
其中，所述光滤波器层用于通过所述光调制结构对进入至所述光调制结构不同位置点处的入射光分别进行不同的频谱调制，以在所述图像传感器的表面得到与不同位置点对应的入射光携带信息；所述入射光携带信息包括光强度分布信息、光谱信息、所述入射光的角度信息以及所述入射光的相位信息；
所述图像传感器通过平方检波响应将与不同位置点经光滤波器层调制后对应的入射光携带信息进行第一次非线性激活处理后转换为与不同位置点对应的电信号，并将与不同位置点对应的电信号发送给所述处理器；
所述处理器将与不同位置点对应的电信号进行全连接处理，或，所述处理器将与不同位置点对应的电信号进行全连接处理以及第二次非线性激活处理，得到人工神经网络的输出信号。
进一步地，在所述图像传感器的表面制备包含有光调制结构的光滤波器层，包括：
在所述图像传感器的表面生长一层或多层预设材料；
对所述一层或多层预设材料进行光调制结构图案的刻蚀，得到包含有 光调制结构的光滤波器层；
或对所述一层或多层预设材料进行压印转移，得到包含有光调制结构的光滤波器层；
或通过对所述一层或多层预设材料进行外加动态调制，得到包含有光调制结构的光滤波器层；
或对所述一层或多层预设材料进行分区打印，得到包含有光调制结构的光滤波器层；
或对所述一层或多层预设材料进行分区生长，得到包含有光调制结构的光滤波器层；
或对所述一层或多层预设材料进行量子点转移，得到包含有光调制结构的光滤波器层。
进一步地，当所述光人工神经网络智能芯片用于目标对象的智能处理任务时，利用与所述智能处理任务对应的输入训练样本和输出训练样本，对包含不同的光调制结构、图像传感器和具有不同的全连接参数的处理器的光人工神经网络智能芯片进行训练，得到满足训练收敛条件的光调制结构、图像传感器和处理器；或，对包含不同的光调制结构、图像传感器和具有不同的全连接参数以及不同的第二次非线性激活参数的处理器的光人工神经网络智能芯片进行训练，得到满足训练收敛条件的光调制结构、图像传感器和处理器。
本申请实施例还提供了一种光人工神经网络环保监测芯片，用于环保监测智能处理任务，包括：光滤波器层、图像传感器和处理器；所述光滤波器层对应人工神经网络的输入层、线性层以及所述输入层到所述线性层的连接权重，所述图像传感器的平方检波响应对应人工神经网络的非线性层中的第一次非线性激活函数；所述处理器对应人工神经网络的全连接以及输出层，或，所述处理器对应人工神经网络的全连接、非线性层中的第二次非线性激活函数以及输出层；
所述光滤波器层设置于所述图像传感器表面，所述光滤波器层包含有光调制结构，所述光调制结构用于对进入至所述光调制结构不同位置点处的入射光分别进行不同的频谱调制，以在所述图像传感器的表面得到与不同位置点对应的入射光携带信息；所述入射光携带信息包括光强度分布信 息、光谱信息、所述入射光的角度信息以及所述入射光的相位信息；所述入射光包括环境污染物的反射光、透射光和/或辐射光；
所述图像传感器通过平方检波响应将与不同位置点经光滤波器层调制后对应的入射光携带信息进行第一次非线性激活处理后转换为与不同位置点对应的电信号，并将与不同位置点对应的电信号发送给所述处理器；
所述处理器将与不同位置点对应的电信号进行全连接处理，或，所述处理器将与不同位置点对应的电信号进行全连接处理以及第二次非线性激活处理，得到环保监测智能处理结果；
其中，所述环保监测智能处理任务包括环境污染物的识别和/或定性分析；所述环保监测智能处理结果包括环境污染物的识别结果和/或环境污染定性分析结果。
进一步地，所述光人工神经网络环保监测芯片包括训练好的光调制结构、图像传感器和处理器；
所述训练好的光调制结构、图像传感器和处理器是指利用与所述环保监测智能处理任务对应的输入训练样本以及输出训练样本，对包含不同的光调制结构、图像传感器和具有不同的全连接参数的处理器的光人工神经网络环保监测芯片进行训练得到的满足训练收敛条件的光调制结构、图像传感器和处理器；或，所述训练好的光调制结构、图像传感器和处理器是指利用与所述环保监测智能处理任务对应的输入训练样本以及输出训练样本，对包含不同的光调制结构、图像传感器和具有不同的全连接参数以及不同的第二次非线性激活参数的处理器的光人工神经网络环保监测芯片进行训练得到的满足训练收敛条件的光调制结构、图像传感器和处理器；
其中，所述输入训练样本包括由不同环境污染物反射、透射和/或辐射的入射光；所述输出训练样本包括相应的环境污染物识别结果；和/或，所述输入训练样本包括由不同环境污染物反射、透射和/或辐射的入射光；所述输出训练样本包括相应的环境污染定性分析结果。
进一步地，在对包含不同的光调制结构、图像传感器和具有不同的全连接参数的处理器的光人工神经网络环保监测芯片进行训练，或，对包含 不同的光调制结构、图像传感器和具有不同的全连接参数以及不同的第二次非线性激活参数的处理器的光人工神经网络环保监测芯片进行训练时，所述不同的光调制结构通过采用计算机光学仿真设计的方式设计实现。
进一步地，所述光滤波器层中的光调制结构包含规则结构和/或不规则结构；和/或，所述光滤波器层中的光调制结构包含离散型结构和/或连续型结构。
进一步地，所述光滤波器层为单层结构或多层结构。
进一步地，所述光滤波器层中的光调制结构包含由多个微纳单元组成的单元阵列，每个微纳单元对应图像传感器上的一个或多个像素点；各个微纳单元的结构相同或不同。
进一步地，所述微纳单元包含规则结构和/或不规则结构；和/或，所述微纳单元包含离散型结构和/或连续型结构。
进一步地，所述微纳单元包含有多组微纳结构阵列，各组微纳结构阵列的结构相同或不同。
进一步地，各组微纳结构阵列具有宽带滤波或窄带滤波的作用。
进一步地，各组微纳结构阵列为周期结构阵列或非周期结构阵列。
进一步地，所述微纳单元包含的多组微纳结构阵列中有一组或多组空结构。
进一步地，所述微纳单元具有偏振无关特性。
进一步地，所述微纳单元具有四重旋转对称性。
进一步地，所述滤波器层是由半导体材料、金属材料、液晶、量子点材料、钙钛矿材料中的一种或多种制备；和/或，所述滤波器层是由光子晶体、超表面、随机结构、纳米结构、金属表面等离激元SPP微纳结构、可调法布里-珀罗谐振腔中的一种或多种制备的滤波器层。
进一步地，所述半导体材料包括硅、氧化硅、氮化硅、氧化钛、按照预设比例混合的复合材料以及直接带隙化合物半导体材料中的一种或多种；和/或，所述纳米结构包括纳米点二维材料、纳米柱二维材料和纳米线二维材料中的一种或多种。
进一步地，所述光滤波器层的厚度为0.1λ～10λ，其中λ表示入射光的中心波长。
进一步地，本申请实施例还提供了一种环保监测设备，包括如上面所述的光人工神经网络环保监测芯片。
本申请实施例还提供了一种如上面所述的光人工神经网络环保监测芯片的制备方法，包括：
在所述图像传感器的表面制备包含有光调制结构的光滤波器层；
生成具备对信号进行全连接处理功能的处理器或生成具备对信号进行全连接处理以及第二次非线性激活处理功能的处理器；
连接所述图像传感器和所述处理器；
其中，所述光滤波器层用于通过所述光调制结构对进入至所述光调制结构不同位置点处的入射光分别进行不同的频谱调制，以在所述图像传感器的表面得到与不同位置点对应的入射光携带信息；所述入射光携带信息包括光强度分布信息、光谱信息、所述入射光的角度信息以及所述入射光的相位信息；
所述图像传感器通过平方检波响应将与不同位置点经光滤波器层调制后对应的入射光携带信息进行第一次非线性激活处理后转换为与不同位置点对应的电信号，并将与不同位置点对应的电信号发送给所述处理器；
所述处理器将与不同位置点对应的电信号进行全连接处理，或，所述处理器将与不同位置点对应的电信号进行全连接处理以及第二次非线性激活处理，得到环保监测智能处理结果。
进一步地，所述光人工神经网络环保监测芯片的制备方法，还包括：对所述光人工神经网络环保监测芯片的训练过程，具体包括：
利用与所述环保监测智能处理任务对应的输入训练样本以及输出训练样本，对包含不同的光调制结构、图像传感器和具有不同的全连接参数的处理器的光人工神经网络环保监测芯片进行训练得到满足训练收敛条件的光调制结构、图像传感器和处理器，并将满足训练收敛条件的光调制结构、图像传感器和处理器作为训练好的光调制结构、图像传感器和处理器；
或，利用与所述环保监测智能处理任务对应的输入训练样本以及输出训练样本，对包含不同的光调制结构、图像传感器和具有不同的全连接参 数以及不同的第二次非线性激活参数的处理器的光人工神经网络环保监测芯片进行训练得到满足训练收敛条件的光调制结构、图像传感器和处理器，并将满足训练收敛条件的光调制结构、图像传感器和处理器作为训练好的光调制结构、图像传感器和处理器。
进一步地，在对包含不同的光调制结构、图像传感器和具有不同的全连接参数的处理器的光人工神经网络环保监测芯片进行训练，或，对包含不同的光调制结构、图像传感器和具有不同的全连接参数以及不同的第二次非线性激活参数的处理器的光人工神经网络环保监测芯片进行训练时，所述不同的光调制结构通过采用计算机光学仿真设计的方式设计实现。
本申请实施例还提供了一种光人工神经网络指纹识别芯片，用于指纹识别处理任务，包括：光滤波器层、图像传感器和处理器；所述光滤波器层对应人工神经网络的输入层、线性层以及所述输入层到所述线性层的连接权重，所述图像传感器的平方检波响应对应人工神经网络的非线性层中的第一次非线性激活函数；所述处理器对应人工神经网络的全连接以及输出层，或，所述处理器对应人工神经网络的全连接、非线性层中的第二次非线性激活函数以及输出层；
所述光滤波器层设置于所述图像传感器表面，所述光滤波器层包含有光调制结构，所述光调制结构用于对进入至所述光调制结构不同位置点处的入射光分别进行不同的频谱调制，以在所述图像传感器的表面得到与不同位置点对应的入射光携带信息；所述入射光携带信息包括光强度分布信息、光谱信息、所述入射光的角度信息以及所述入射光的相位信息；所述入射光包括用户指纹的反射光、透射光和/或辐射光；
所述图像传感器通过平方检波响应将与不同位置点经光滤波器层调制后对应的入射光携带信息进行第一次非线性激活处理后转换为与不同位置点对应的电信号，并将与不同位置点对应的电信号发送给所述处理器；
所述处理器将与不同位置点对应的电信号进行全连接处理，或，所述处理器将与不同位置点对应的电信号进行全连接处理以及第二次非线性激活处理，得到指纹识别处理结果。
进一步地，所述光人工神经网络指纹识别芯片包括训练好的光调制结 构、图像传感器和处理器；
所述训练好的光调制结构、图像传感器和处理器是指利用与所述指纹识别处理任务对应的输入训练样本和输出训练样本，对包含不同的光调制结构、图像传感器和具有不同的全连接参数的处理器的光人工神经网络指纹识别芯片进行训练得到的满足训练收敛条件的光调制结构、图像传感器和处理器；或，所述训练好的光调制结构、图像传感器和处理器是指利用与所述指纹识别处理任务对应的输入训练样本和输出训练样本，对包含不同的光调制结构、图像传感器和具有不同的全连接参数以及不同的第二次非线性激活参数的处理器的光人工神经网络指纹识别芯片进行训练得到的满足训练收敛条件的光调制结构、图像传感器和处理器；
其中，所述输入训练样本包括由不同人体指纹反射、透射和/或辐射的入射光；所述输出训练样本包括相应的指纹识别结果。
进一步地，在对包含不同的光调制结构、图像传感器和具有不同的全连接参数的处理器的光人工神经网络指纹识别芯片进行训练，或，对包含不同的光调制结构、图像传感器和具有不同的全连接参数以及不同的第二次非线性激活参数的处理器的光人工神经网络指纹识别芯片进行训练时，所述不同的光调制结构通过采用计算机光学仿真设计的方式设计实现。
进一步地，所述光滤波器层中的光调制结构包含规则结构和/或不规则结构；和/或，所述光滤波器层中的光调制结构包含离散型结构和/或连续型结构。
进一步地，所述光滤波器层为单层结构或多层结构。
进一步地，所述光滤波器层中的光调制结构包含由多个微纳单元组成的单元阵列，每个微纳单元对应图像传感器上的一个或多个像素点；各个微纳单元的结构相同或不同。
进一步地，所述微纳单元包含规则结构和/或不规则结构；和/或，所述微纳单元包含离散型结构和/或连续型结构。
进一步地，所述微纳单元包含有多组微纳结构阵列，各组微纳结构阵列的结构相同或不同。
进一步地，各组微纳结构阵列具有宽带滤波或窄带滤波的作用。
进一步地，各组微纳结构阵列为周期结构阵列或非周期结构阵列。
进一步地，所述微纳单元包含的多组微纳结构阵列中有一组或多组空结构。
进一步地，所述微纳单元具有偏振无关特性。
进一步地，所述微纳单元具有四重旋转对称性。
进一步地，所述滤波器层是由半导体材料、金属材料、液晶、量子点材料、钙钛矿材料中的一种或多种制备；和/或，所述滤波器层是由光子晶体、超表面、随机结构、纳米结构、金属表面等离激元SPP微纳结构、可调法布里-珀罗谐振腔中的一种或多种制备的滤波器层。
进一步地，所述半导体材料包括硅、氧化硅、氮化硅、氧化钛、按照预设比例混合的复合材料以及直接带隙化合物半导体材料中的一种或多种；和/或，所述纳米结构包括纳米点二维材料、纳米柱二维材料和纳米线二维材料中的一种或多种。
进一步地，所述光滤波器层的厚度为0.1λ～10λ，其中λ表示入射光的中心波长。
本申请实施例还提供了一种指纹识别设备，包括如上面所述的光人工神经网络指纹识别芯片。
本申请实施例还提供了一种如上面任一项所述的光人工神经网络指纹识别芯片的制备方法，包括：
在所述图像传感器的表面制备包含有光调制结构的光滤波器层；
生成具备对信号进行全连接处理功能的处理器或生成具备对信号进行全连接处理以及第二次非线性激活处理功能的处理器；
连接所述图像传感器和所述处理器；
其中，所述光滤波器层用于通过所述光调制结构对进入至所述光调制结构不同位置点处的入射光分别进行不同的频谱调制，以在所述图像传感器的表面得到与不同位置点对应的入射光携带信息；所述入射光携带信息包括光强度分布信息、光谱信息、所述入射光的角度信息以及所述入射光的相位信息；
所述图像传感器通过平方检波响应将与不同位置点经光滤波器层调制后对应的入射光携带信息进行第一次非线性激活处理后转换为与不同位置点对应的电信号，并将与不同位置点对应的电信号发送给所述处理 器；
所述处理器将与不同位置点对应的电信号进行全连接处理，或，所述处理器将与不同位置点对应的电信号进行全连接处理以及第二次非线性激活处理，得到指纹识别处理结果。
进一步地，所述光人工神经网络指纹识别芯片的制备方法，还包括：对所述光人工神经网络指纹识别芯片的训练过程，具体包括：
利用与所述指纹识别处理任务对应的输入训练样本和输出训练样本，对包含不同的光调制结构、图像传感器和具有不同的全连接参数的处理器的光人工神经网络指纹识别芯片进行训练得到满足训练收敛条件的光调制结构、图像传感器和处理器，并将满足训练收敛条件的光调制结构、图像传感器和处理器作为训练好的光调制结构、图像传感器和处理器；
或，利用与所述指纹识别处理任务对应的输入训练样本和输出训练样本，对包含不同的光调制结构、图像传感器和具有不同的全连接参数以及不同的第二次非线性激活参数的处理器的光人工神经网络指纹识别芯片进行训练得到满足训练收敛条件的光调制结构、图像传感器和处理器，并将满足训练收敛条件的光调制结构、图像传感器和处理器作为训练好的光调制结构、图像传感器和处理器。
本申请实施例还提供了一种光人工神经网络人脸识别芯片，用于人脸识别处理任务，包括：光滤波器层、图像传感器和处理器；所述光滤波器层对应人工神经网络的输入层、线性层以及所述输入层到所述线性层的连接权重，所述图像传感器的平方检波响应对应人工神经网络的非线性层中的第一次非线性激活函数；所述处理器对应人工神经网络的全连接以及输出层，或，所述处理器对应人工神经网络的全连接、非线性层中的第二次非线性激活函数以及输出层；
所述光滤波器层设置于所述图像传感器表面，所述光滤波器层包含有光调制结构，所述光调制结构用于对进入至所述光调制结构不同位置点处的入射光分别进行不同的频谱调制，以在所述图像传感器的表面得到与不同位置点对应的入射光携带信息；所述入射光携带信息包括光强度分布信息、光谱信息、所述入射光的角度信息以及所述入射光的相位信息；所述入射光包括用户人脸的反射光、透射光和/或辐射光；
所述图像传感器通过平方检波响应将与不同位置点经光滤波器层调制后对应的入射光携带信息进行第一次非线性激活处理后转换为与不同位置点对应的电信号，并将与不同位置点对应的电信号发送给所述处理器；
所述处理器将与不同位置点对应的电信号进行全连接处理，或，所述处理器将与不同位置点对应的电信号进行全连接处理以及第二次非线性激活处理，得到人脸识别处理结果。
进一步地，所述光人工神经网络人脸识别芯片包括训练好的光调制结构、图像传感器和处理器；
所述训练好的光调制结构、图像传感器和处理器是指利用与所述人脸识别处理任务对应的输入训练样本和输出训练样本，对包含不同的光调制结构、图像传感器和具有不同的全连接参数的处理器的光人工神经网络人脸识别芯片进行训练得到的满足训练收敛条件的光调制结构、图像传感器和处理器；或，所述训练好的光调制结构、图像传感器和处理器是指利用与所述人脸识别处理任务对应的输入训练样本和输出训练样本，对包含不同的光调制结构、图像传感器和具有不同的全连接参数以及不同的第二次非线性激活参数的处理器的光人工神经网络人脸识别芯片进行训练得到的满足训练收敛条件的光调制结构、图像传感器和处理器；
其中，所述输入训练样本包括由不同人脸反射、透射和/或辐射的入射光；所述输出训练样本包括相应的人脸识别结果。
进一步地，在对包含不同的光调制结构、图像传感器和具有不同的全连接参数的处理器的光人工神经网络人脸识别芯片进行训练，或，对包含不同的光调制结构、图像传感器和具有不同的全连接参数以及不同的第二次非线性激活参数的处理器的光人工神经网络人脸识别芯片进行训练时，所述不同的光调制结构通过采用计算机光学仿真设计的方式设计实现。
进一步地，所述光滤波器层中的光调制结构包含规则结构和/或不规则结构；和/或，所述光滤波器层中的光调制结构包含离散型结构和/或连续型结构。
进一步地，所述光滤波器层为单层结构或多层结构。
进一步地，所述光滤波器层中的光调制结构包含由多个微纳单元组成 的单元阵列，每个微纳单元对应图像传感器上的一个或多个像素点；各个微纳单元的结构相同或不同。
进一步地，所述微纳单元包含规则结构和/或不规则结构；和/或，所述微纳单元包含离散型结构和/或连续型结构。
进一步地，所述微纳单元包含有多组微纳结构阵列，各组微纳结构阵列的结构相同或不同。
进一步地，各组微纳结构阵列具有宽带滤波或窄带滤波的作用。
进一步地，各组微纳结构阵列为周期结构阵列或非周期结构阵列。
进一步地，所述微纳单元包含的多组微纳结构阵列中有一组或多组空结构。
进一步地，所述微纳单元具有偏振无关特性，特别地，所述微纳单元具有四重旋转对称性。
进一步地，所述光滤波器层由一层或多层结构构成；
所述滤波器层是由半导体材料、金属材料、液晶、量子点材料、钙钛矿材料中的一种或多种制备；和/或，所述滤波器层是由光子晶体、超表面、随机结构、纳米结构、金属表面等离激元SPP微纳结构、可调法布里-珀罗谐振腔中的一种或多种制备的滤波器层。
进一步地，所述半导体材料包括硅、氧化硅、氮化硅、氧化钛、按照预设比例混合的复合材料以及直接带隙化合物半导体材料中的一种或多种；和/或，所述纳米结构包括纳米点二维材料、纳米柱二维材料和纳米线二维材料中的一种或多种。
进一步地，所述光滤波器层的厚度为0.1λ～10λ，其中λ表示入射光的中心波长。
本申请实施例还提供了一种人脸识别设备，包括如上面所述的光人工神经网络人脸识别芯片。
本申请实施例还提供了一种如上面所述的光人工神经网络人脸识别芯片的制备方法，包括：
在所述图像传感器的表面制备包含有光调制结构的光滤波器层；
生成具备对信号进行全连接处理功能的处理器或生成具备对信号进行全连接处理以及第二次非线性激活处理功能的处理器；
连接所述图像传感器和所述处理器；
其中，所述光滤波器层用于通过所述光调制结构对进入至所述光调制结构不同位置点处的入射光分别进行不同的频谱调制，以在所述图像传感器的表面得到与不同位置点对应的入射光携带信息；所述入射光携带信息包括光强度分布信息、光谱信息、所述入射光的角度信息以及所述入射光的相位信息；
所述图像传感器通过平方检波响应将与不同位置点经光滤波器层调制后对应的入射光携带信息进行第一次非线性激活处理后转换为与不同位置点对应的电信号，并将与不同位置点对应的电信号发送给所述处理器；
所述处理器将与不同位置点对应的电信号进行全连接处理，或，所述处理器将与不同位置点对应的电信号进行全连接处理以及第二次非线性激活处理，得到人脸识别处理结果。
进一步地，所述光人工神经网络人脸识别芯片的制备方法，还包括：对所述光人工神经网络人脸识别芯片的训练过程，具体包括：
利用与所述人脸识别处理任务对应的输入训练样本和输出训练样本，对包含不同的光调制结构、图像传感器和具有不同的全连接参数的处理器的光人工神经网络人脸识别芯片进行训练得到满足训练收敛条件的光调制结构、图像传感器和处理器，并将满足训练收敛条件的光调制结构、图像传感器和处理器作为训练好的光调制结构、图像传感器和处理器；
或，利用与所述人脸识别处理任务对应的输入训练样本和输出训练样本，对包含不同的光调制结构、图像传感器和具有不同的全连接参数以及不同的第二次非线性激活参数的处理器的光人工神经网络人脸识别芯片进行训练得到满足训练收敛条件的光调制结构、图像传感器和处理器，并将满足训练收敛条件的光调制结构、图像传感器和处理器作为训练好的光调制结构、图像传感器和处理器。
进一步地，在对包含不同的光调制结构、图像传感器和具有不同的全连接参数的处理器的光人工神经网络人脸识别芯片进行训练，或，对包含不同的光调制结构、图像传感器和具有不同的全连接参数以及不同的第二次非线性激活参数的处理器的光人工神经网络人脸识别芯片进行训练时， 所述不同的光调制结构通过采用计算机光学仿真设计的方式设计实现。
本申请实施例还提供了一种光人工神经网络机器视觉增强芯片，用于机器视觉智能处理任务，包括：光滤波器层、图像传感器和处理器；所述光滤波器层对应人工神经网络的输入层、线性层以及所述输入层到所述线性层的连接权重，所述图像传感器的平方检波响应对应人工神经网络的非线性层中的第一次非线性激活函数；所述处理器对应人工神经网络的全连接以及输出层，或，所述处理器对应人工神经网络的全连接、非线性层中的第二次非线性激活函数以及输出层；
所述光滤波器层设置于所述图像传感器表面，所述光滤波器层包含有光调制结构，所述光调制结构用于对进入至所述光调制结构不同位置点处的入射光分别进行不同的频谱调制，以在所述图像传感器的表面得到与不同位置点对应的入射光携带信息；所述入射光携带信息包括光强度分布信息、光谱信息、所述入射光的角度信息以及所述入射光的相位信息；所述入射光包括机器视觉场景下目标对象的反射光、透射光和/或辐射光；
所述图像传感器通过平方检波响应将与不同位置点经光滤波器层调制后对应的入射光携带信息进行第一次非线性激活处理后转换为与不同位置点对应的电信号，并将与不同位置点对应的电信号发送给所述处理器；
所述处理器将与不同位置点对应的电信号进行全连接处理，或，所述处理器将与不同位置点对应的电信号进行全连接处理以及第二次非线性激活处理，得到机器视觉智能处理结果；
其中，所述机器视觉智能处理任务包括机器视觉场景下目标对象的识别和/或定性分析；所述机器视觉智能处理结果包括机器视觉场景下目标对象的识别结果和/或定性分析结果。
进一步地，所述光人工神经网络机器视觉增强芯片包括训练好的光调制结构、图像传感器和处理器；
所述训练好的光调制结构、图像传感器和处理器是指利用与所述机器视觉智能处理任务对应的输入训练样本以及输出训练样本，对包含不同的光调制结构、图像传感器和具有不同的全连接参数的处理器的光人工神经网络机器视觉增强芯片进行训练得到的满足训练收敛条件的光调制结构、 图像传感器和处理器；或，所述训练好的光调制结构、图像传感器和处理器是指利用与所述机器视觉智能处理任务对应的输入训练样本以及输出训练样本，对包含不同的光调制结构、图像传感器和具有不同的全连接参数以及不同的第二次非线性激活参数的处理器的光人工神经网络机器视觉增强芯片进行训练得到的满足训练收敛条件的光调制结构、图像传感器和处理器；
其中，所述输入训练样本包括由特定机器视觉场景下的目标对象反射、透射和/或辐射的入射光，所述输出训练样本包括特定机器视觉场景下的目标对象识别结果；和/或，所述输入训练样本包括由特定机器视觉场景下的目标对象反射、透射和/或辐射的入射光，所述输出训练样本包括特定机器视觉场景下的目标对象定性分析结果。
进一步地，在对包含不同的光调制结构、图像传感器和具有不同的全连接参数的处理器的光人工神经网络机器视觉增强芯片进行训练，或，对包含不同的光调制结构、图像传感器和具有不同的全连接参数以及不同的第二次非线性激活参数的处理器的光人工神经网络机器视觉增强芯片进行训练时，所述不同的光调制结构通过采用计算机光学仿真设计的方式设计实现。
进一步地，所述光滤波器层中的光调制结构包含规则结构和/或不规则结构；和/或，所述光滤波器层中的光调制结构包含离散型结构和/或连续型结构。
进一步地，所述光滤波器层为单层结构或多层结构。
进一步地，所述光滤波器层中的光调制结构包含由多个微纳单元组成的单元阵列，每个微纳单元对应图像传感器上的一个或多个像素点；各个微纳单元的结构相同或不同。
进一步地，所述微纳单元包含规则结构和/或不规则结构；和/或，所述微纳单元包含离散型结构和/或连续型结构。
进一步地，所述微纳单元包含有多组微纳结构阵列，各组微纳结构阵列的结构相同或不同。
进一步地，各组微纳结构阵列具有宽带滤波或窄带滤波的作用。
进一步地，各组微纳结构阵列为周期结构阵列或非周期结构阵列。
进一步地，所述微纳单元包含的多组微纳结构阵列中有一组或多组空结构。
进一步地，所述微纳单元具有偏振无关特性。
进一步地，所述微纳单元具有四重旋转对称性。
进一步地，所述滤波器层是由半导体材料、金属材料、液晶、量子点材料、钙钛矿材料中的一种或多种制备；和/或，所述滤波器层是由光子晶体、超表面、随机结构、纳米结构、金属表面等离激元SPP微纳结构、可调法布里-珀罗谐振腔中的一种或多种制备的滤波器层。
进一步地，所述半导体材料包括硅、氧化硅、氮化硅、氧化钛、按照预设比例混合的复合材料以及直接带隙化合物半导体材料中的一种或多种；和/或，所述纳米结构包括纳米点二维材料、纳米柱二维材料和纳米线二维材料中的一种或多种。
进一步地，所述光滤波器层的厚度为0.1λ～10λ，其中λ表示入射光的中心波长。
本申请实施例还提供了一种增强机器视觉系统，包括控制机构以及如上面所述的光人工神经网络机器视觉增强芯片。
本申请实施例还提供了一种如上面任一项所述的光人工神经网络机器视觉增强芯片的制备方法，包括：
在所述图像传感器的表面制备包含有光调制结构的光滤波器层；
生成具备对信号进行全连接处理功能的处理器或生成具备对信号进行全连接处理以及第二次非线性激活处理功能的处理器；
连接所述图像传感器和所述处理器；
其中，所述光滤波器层用于通过所述光调制结构对进入至所述光调制结构不同位置点处的入射光分别进行不同的频谱调制，以在所述图像传感器的表面得到与不同位置点对应的入射光携带信息；所述入射光携带信息包括光强度分布信息、光谱信息、所述入射光的角度信息以及所述入射光的相位信息；
所述图像传感器通过平方检波响应将与不同位置点经光滤波器层调制后对应的入射光携带信息进行第一次非线性激活处理后转换为与不同位置点对应的电信号，并将与不同位置点对应的电信号发送给所述处理 器；
所述处理器将与不同位置点对应的电信号进行全连接处理，或，所述处理器将与不同位置点对应的电信号进行全连接处理以及第二次非线性激活处理，得到机器视觉智能处理结果。
进一步地，所述光人工神经网络机器视觉增强芯片的制备方法，还包括：对所述光人工神经网络机器视觉增强芯片的训练过程，具体包括：
利用与所述机器视觉智能处理任务对应的输入训练样本以及输出训练样本，对包含不同的光调制结构、图像传感器和具有不同的全连接参数的处理器的光人工神经网络机器视觉增强芯片进行训练得到满足训练收敛条件的光调制结构、图像传感器和处理器，并将满足训练收敛条件的光调制结构、图像传感器和处理器作为训练好的光调制结构、图像传感器和处理器；
或，利用与所述机器视觉智能处理任务对应的输入训练样本以及输出训练样本，对包含不同的光调制结构、图像传感器和具有不同的全连接参数以及不同的第二次非线性激活参数的处理器的光人工神经网络机器视觉增强芯片进行训练得到满足训练收敛条件的光调制结构、图像传感器和处理器，并将满足训练收敛条件的光调制结构、图像传感器和处理器作为训练好的光调制结构、图像传感器和处理器。
进一步地，在对包含不同的光调制结构、图像传感器和具有不同的全连接参数的处理器的光人工神经网络机器视觉增强芯片进行训练，或，对包含不同的光调制结构、图像传感器和具有不同的全连接参数以及不同的第二次非线性激活参数的处理器的光人工神经网络机器视觉增强芯片进行训练时，所述不同的光调制结构通过采用计算机光学仿真设计的方式设计实现。
本申请实施例提供的光人工神经网络智能芯片及制备方法，实现了一种能够实现人工神经网络功能的全新智能芯片，在该智能芯片中，光滤波器层对应人工神经网络的输入层和线性层，图像传感器对应人工神经网络的非线性层的一部分；所述处理器对应人工神经网络的非线性层的另一部分以及输出层。具体地，光滤波器层设置于图像传感器的表面，光滤波器层包含有光调制结构，光调制结构用于对进入至光调制结构不同位置点处 的入射光分别进行不同的频谱调制，以在图像传感器的表面得到与不同位置点对应的入射光携带信息，在本申请实施例中，光滤波器层上的光调制结构对入射光的调制作用相当于输入层到所述线性层的连接权重。同时，在本申请实施例中，图像传感器通过平方检波响应将与不同位置点经光滤波器层调制后对应的入射光携带信息进行第一次非线性激活处理后转换为与不同位置点对应的电信号，并将与不同位置点对应的电信号发送给所述处理器，处理器将与不同位置点对应的电信号进行全连接处理，或所述处理器将与不同位置点对应的电信号进行全连接处理以及第二次非线性激活处理，得到人工神经网络的输出信号，由此可见，在该智能芯片中，所述光滤波器层对应人工神经网络的输入层、线性层以及所述输入层到所述线性层的连接权重，所述图像传感器的平方检波响应对应人工神经网络的非线性层中的第一次非线性激活函数；所述处理器对应人工神经网络的全连接以及输出层，或，所述处理器对应人工神经网络的全连接、非线性层中的第二次非线性激活函数以及输出层，也即该智能芯片中的光滤波器层和图像传感器实现了人工神经网络中输入层、线性层和部分非线性激活函数的相关功能，也即本申请实施例将现有技术中采用软件实现的人工神经网络中的输入层、线性层以及一部分或全部的非线性激活函数进行了剥离，利用硬件的方式实现了人工神经网络中的输入层、线性层以及一部分或全部的非线性激活函数这些结构，从而使得后续在使用该智能芯片进行人工神经网络智能处理时不需要再进行与输入层、线性层以及一部分或全部非线性激活函数对应的复杂的信号处理和算法处理，只需由智能芯片中的处理器进行与电信号全连接处理或全连接以及第二次非线性激活处理即可，这样可以大幅降低人工神经网络处理时的功耗和延时。由此可见，本申请实施例将光滤波器层作为人工神经网络的输入层、线性层以及所述输入层到所述线性层的连接权重，将图像传感器的平方检波响应作为人工神经网络的非线性层中的第一次非线性激活函数；将处理器作为人工神经网络的全连接以及输出层，或，将处理器对应人工神经网络的全连接、非线性层中的第二次非线性激活函数以及输出层，由此可见，本申请实施例不但能够省去现有技术中与输入层、线性层和一部分非线性激活函数对应的复杂的信号处理和算法处理，而且本申请实施例实际上同时利用了目标 对象的图像信息、光谱信息、入射光的角度和入射光的相位信息，即目标对象空间不同点处的入射光携带信息，由此可见，由于目标对象空间不同点处的入射光携带信息涵盖了目标对象的图像、成分、形状、三维深度、结构等信息，从而在依据目标对象空间不同点处的入射光携带信息进行识别处理时，可以涵盖目标对象的图像、成分、形状、三维深度、结构等多维度的信息，从而可以解决背景技术部分所提到的采用目标对象的二维图像信息难以保证识别的准确性例如难以区分是真实人物还是图片的问题，由此可见，本申请实施例提供的光人工神经网络芯片，不但能够实现低功耗和低延时的效果，还能够实现高准确率的效果，从而可以为应用于智能感知、识别和/或决策等智能处理任务做好准备。
附图说明
为了更清楚地说明本申请实施例或现有技术中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作一简单地介绍，显而易见地，下面描述中的附图是本申请的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1是本申请第一个实施例提供的光人工神经网络智能芯片的结构示意图；
图2是本申请一实施例提供的光人工神经网络智能芯片识别原理示意图；
图3是本申请一实施例提供的光人工神经网络智能芯片拆解示意图；
图4是本申请一实施例提供的目标对象识别过程示意图；
图5是本申请一实施例提供的一种光滤波器层的俯视图；
图6是本申请一实施例提供的另一种光滤波器层的俯视图；
图7是本申请一实施例提供的又一种光滤波器层的俯视图；
图8是本申请一实施例提供的另又一种光滤波器层的俯视图；
图9是本申请一实施例提供的再又一种光滤波器层的俯视图；
图10是本申请一实施例提供的还又一种光滤波器层的俯视图；
图11是本申请一实施例提供的微纳结构宽带滤波效果示意图；
图12是本申请一实施例提供的微纳结构窄带滤波效果示意图；
图13是本申请一实施例提供的前照式的图像传感器结构示意图；
图14是本申请一实施例提供的后照式的图像传感器结构示意图；
图15是本申请第三个实施例提供的光人工神经网络智能芯片的制备方法的流程示意图；
图16是本申请一实施例提供的污染物样本识别过程示意图；
图17是本申请一实施例提供的指纹识别过程示意图；
图18是本申请一实施例提供的人脸识别过程示意图；
图19是本申请一实施例提供的机器视觉增强识别过程示意图。
具体实施方式
为使本申请实施例的目的、技术方案和优点更加清楚，下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例是本申请一部分实施例，而不是全部的实施例。基于本申请中的实施例，本领域普通技术人员在没有作出创造性劳动前提下所获得的所有其他实施例，都属于本申请保护的范围。
现有的智能识别技术，通常需要先对人或物体成像，再经过图像的预处理、特征提取、特征匹配等步骤，实现对人或物体的识别。然而，只利用人或物体的二维图像信息难以保证识别的准确性，例如难以区分真实的人脸和人脸照片，并且成像过程需要将光学信息转换为数字电子信号，再传输到计算机中进行后续的算法处理，大量数据的传输和处理造成了较大的功耗和延时。基于此，本申请实施例提供一种光人工神经网络智能芯片，该智能芯片中的光滤波器层对应人工神经网络的输入层、线性层以及所述输入层到所述线性层的连接权重，图像传感器的平方检波响应对应人工神经网络的非线性层中的第一次非线性激活函数；处理器对应人工神经网络的全连接以及输出层，或，所述处理器对应人工神经网络的全连接、非线性层中的第二次非线性激活函数以及输出层，本申请实施例利用光滤波器层和图像传感器将目标对象的空间光谱信息投影成电信号，然后在处理器中实现电信号的全连接处理或全连接处理以及第二次非线性激活处理，由 此可见，本申请实施例不但能够省去现有技术中与输入层、线性层以及部分或全部非线性激活函数对应的复杂的信号处理和算法处理，而且本申请实施例实际上同时利用了目标对象的图像信息、光谱信息、入射光的角度和入射光的相位信息，即目标对象空间不同点处的入射光携带信息，由此可见，由于目标对象空间不同点处的入射光携带信息涵盖了目标对象的图像、成分、形状、三维深度、结构等信息，从而在依据目标对象空间不同点处的入射光携带信息进行识别处理时，可以涵盖目标对象的图像、成分、形状、三维深度、结构等多维度的信息，从而可以提高智能处理(如智能识别)的准确性，由此可见，本申请实施例提供的光人工神经网络芯片，不但能够实现低功耗和低延时的效果，还能够提高智能处理的准确率，从而可以较好地应用在智能感知、识别和/或决策等智能处理领域。下面将通过具体实施例对本申请提供的内容进行详细解释和说明。
如图1所示，本申请第一个实施例提供的光人工神经网络智能芯片，包括：光滤波器层1、图像传感器2和处理器3；所述光滤波器层1对应人工神经网络的输入层、线性层以及所述输入层到所述线性层的连接权重，所述图像传感器2的平方检波响应对应人工神经网络的非线性层中的第一次非线性激活函数；所述处理器3对应人工神经网络的全连接以及输出层，或，所述处理器对应人工神经网络的全连接、非线性层中的第二次非线性激活函数以及输出层；
所述光滤波器层1设置于所述图像传感器的表面或图像传感器的感光区域的表面，所述光滤波器层1包含有光调制结构，所述光滤波器层1用于通过所述光调制结构对进入至所述光调制结构不同位置点处的入射光分别进行随波长变化强度调制的频谱调制，即对不同波长的入射光进行不同的强度调制，以在所述图像传感器的表面得到与不同位置点对应的入射光携带信息；所述入射光携带信息包括光强度分布信息、光谱信息、所述入射光的角度信息以及所述入射光的相位信息；
在本实施例中，所述图像传感器2的平方检波响应是指图像传感器探测到的是入射光场的强度信息，而入射光场的强度信息为光场信号取模的平方，也即所述图像传感器2通过平方检波响应将与不同位置点经光滤波器层1调制后对应的入射光携带信息进行第一次非线性激活处理后转换为 与不同位置点对应的电信号，并将与不同位置点对应的电信号发送给所述处理器3；所述电信号为经光滤波器层调制后的图像信号；
所述处理器3用于将与不同位置点对应的电信号进行全连接处理，或所述处理器将与不同位置点对应的电信号进行全连接处理以及第二次非线性激活处理，得到人工神经网络的输出信号。
在本实施例中，光滤波器层1设置于图像传感器的表面，光滤波器层1包含有光调制结构，光滤波器层1用于通过光调制结构对进入至光调制结构不同位置点处的入射光分别进行不同的频谱调制，以在图像传感器的表面得到与不同位置点对应的调制后的入射光携带信息。由此可见，在实施例中，光滤波器层上的光调制结构对入射光的调制作用可以看作是输入层到所述线性层的连接权重；
在本实施例中，图像传感器2对调制后的入射光携带信息进行光电转换时，由于图像传感器2能探测光的强度信息，故其对光场分布信号进行处理得到的电信号正比于光场分布信号的模的平方，因此，图像传感器2存在平方检波响应，故可以将图像传感器2看作为人工神经网络的非线性层的一部分，也即可以将图像传感器2的平方检波响应看作为人工神经网络的第一次非线性激活函数。
在本实施例中，图像传感器2通过平方检波响应将与不同位置点经光滤波器层1调制后对应的入射光携带信息进行第一次非线性激活处理后转换为与不同位置点对应的电信号，即经光滤波器层调制后的图像信号，与此同时，与图像传感器2连接的处理器3用于将与不同位置点对应的电信号进行全连接处理或进行全连接以及第二次非线性激活处理，得到人工神经网络的输出信号。
在本实施例中，光滤波器层1包含有光调制结构，通过光调制结构对进入至光调制结构不同位置点处的入射光(例如待识别目标的反射光、透射光、辐射光等相关作用光)进行不同强度的频谱调制，以在图像传感器2的表面得到与不同位置点对应的入射光携带信息。
在本实施例中，可以理解的是，调制强度与光调制结构的具体结构形式有关，例如，可以通过设计不同的光调制结构(如改变光调制结构的形状和/或尺寸参数)来实现不同的调制强度。
在本实施例中，可以理解的是，光滤波器层1上不同位置处的光调制结构对入射光具有不同的频谱调制作用，光调制结构对入射光不同波长成分的调制强度对应于人工神经网络线性层的连接强度，也即对应输入层以及输入层到线性层的连接权重。需要说明的是，光滤波器层1是由多个光滤波器单元组成的，每个光滤波器单元内不同位置处的光调制结构是不同的，因此对入射光具有不同的频谱调制作用；光滤波器单元之间不同位置处的光调制结构可以相同或不同，因此对入射光具有相同或不同的频谱调制作用。
在本实施例中，图像传感器2通过平方检波响应将与不同位置点经光滤波器层1调制后对应的入射光携带信息进行第一次非线性激活处理后转换为与不同位置点对应的电信号，并将与不同位置点对应的电信号发送给处理器3，图像传感器2对应神经网络的非线性层的一部分。
在本实施例中，处理器3将不同位置点的电信号进行全连接处理或，处理器3将不同位置点的电信号进行全连接处理以及第二次非线性激活处理，进而得到人工神经网络的输出信号。
可以理解的是，在本实施例中，图像传感器2对应神经网络非线性层的一部分，处理器3对应神经网络的非线性层的另一部分以及输出层，也可以理解成对应神经网络中除输入层、线性层和非线性层中的第一次非线性激活函数以外的剩余层(其他所有层)。
在本实施例中，需要说明的是，图像传感器2的平方检波响应对应神经网络非线性层中的第一次非线性激活函数，在这种情况下，可以是处理器中只进行全连接处理，不再进行第二次非线性激活处理，也可以是处理器中既进行全连接处理，又进行第二次非线性激活处理。具体可根据芯片的实际应用场景确定，本实施例对此不作限定。
此外，需要补充说明的是，处理器3可以设置在所述智能芯片内，也即所述处理器3可以和所述滤波器层1以及图像传感器2一起设置在智能芯片内，也可以单独地设置在智能芯片外，并通过数据线或连接器件与智能芯片内中的图像传感器2连接，本实施例对此不作限定。
此外，需要说明的是，所述处理器3可以采用计算机实现，也可以采用具有一定运算能力的ARM或FPGA电路板实现，还可以采用微处理器 实现，本实施例对此不做限定。此外，正如前面所述，所述处理器3可以集成在所述智能芯片内，也可以独立于所述智能芯片外设置。当所述处理器3独立于所述智能芯片外设置时，可以通过信号读出电路将图像传感器2中的电信号读出至处理器3中，进而由处理器3对读出的电信号进行全连接处理与非线性激活处理。
在本实施例中，可以理解的是，处理器3在进行第二次非线性激活处理时，可以采用非线性激活函数实现，例如可以采用Sigmoid函数、Tanh函数、ReLU函数等，本实施例对此不作限定。
在本实施例中，光滤波器层1对应人工神经网络的输入层、线性层以及输入层到线性层的连接权重，图像传感器2对应于人工神经网络的非线性层的一部分，也即图像传感器2的平方检波响应对应于人工神经网络的第一非线性激活函数，图像传感器2用于通过平方检波响应将空间不同位置点的入射光携带信息进行非线性激活处理进而转化为电信号，处理器3对应人工神经网络的剩余层，将不同位置的电信号进行全连接，也可以进一步经由第二次非线性激活函数得到人工神经网络的输出信号，实现对特定目标的智能感知、识别和/或决策。
如图2左侧所示，光人工神经网络智能芯片包括光滤波器层1、图像传感器2和处理器3，在图2中，处理器3采用信号读出电路和计算机来实现。如图2右侧所示，光人工神经网络智能芯片中的光滤波器层1对应人工神经网络的输入层和线性层，图像传感器2对应人工神经网络的非线性层的一部分，处理器3对应人工神经网络的非线性层的另一部分和输出层，光滤波器层1对进入光滤波器层1的入射光的滤波作用对应输入层到线性层的连接权重，图像传感器2的平方检波响应对应人工神经网络的第一非线性激活函数，由此可见，本实施例提供的智能芯片中的光滤波器层和图像传感器通过硬件的方式实现了人工神经网络中输入层、线性层和部分或全部非线性激活函数的相关功能，从而使得后续在使用该智能芯片进行智能处理时不需要再进行与输入层和线性层对应的复杂的信号处理和算法处理(例如省略了输入层到线性层的连接权重等这些计算)，这样可以大幅降低人工神经网络处理时的功耗和延时。此外，由于本实施例同时利用了目标对象的图像信息、光谱信息、入射光的角度和入射光的相位信 息，从而可以更加准确地实现对目标对象的智能处理。
如图2右侧所示，光滤波器层1对入射光具有不同的宽带频谱调制作用，将对应单元位置处的入射光频谱P_λ投影/连接到出射的光场E_N上；图像传感器2的平方检波响应对应于光人工神经网的非线性激活函数，将光滤波器层1的出射光场E_N转换为图像传感器的光电流响应I_N上。处理器3包括信号读出电路和计算机，处理器3中的信号读出电路将光电流响应读出I_N至传输到计算机中，由计算机进行电信号的全连接处理或再次进行非线性激活处理，最后输出结果。
如图3所示，光滤波器层1上的光调制结构集成在图像传感器2表面，对入射光进行调制，将入射光的频谱信息投影/连接到图像传感器2的不同像素点上，得到包含入射光携带信息的电信号，也即入射光经过光滤波器层1后，由图像传感器2的平方检波响应进行非线性激活后转换成电信号，形成包含入射光的频谱信息的图像，最后由与图像传感器2连接的处理器3对包含入射光的频谱信息和图像信息的电信号进行处理，进而获得输出结果。由此可见，本实施例提供的光人工神经网络芯片实际上同时利用了目标对象的图像信息、光谱信息、入射光的角度和入射光的相位信息，即空间不同点处的入射光携带信息，并在硬件上嵌入了人工神经网络，从空间图像、光谱、角度、相位信息中可以进一步提取物质成分、图像形状、三维深度等信息，从而可以解决背景技术部分所提到的采用目标对象的二维图像信息难以保证识别的准确性例如难以区分是真实人物还是图片的问题，实现面向不同应用领域的智能感知、识别和/或决策功能，并且实现了低功耗、低延时和高准确率的频谱光人工神经网智能芯片。
本申请实施例提供的光人工神经网络智能芯片，在该智能芯片中，光滤波器层对应人工神经网络的输入层和线性层，图像传感器对应人工神经网络的非线性层的一部分；所述处理器对应人工神经网络的非线性层的另一部分以及输出层。具体地，光滤波器层设置于图像传感器的表面，光滤波器层包含有光调制结构，光调制结构用于对进入至光调制结构不同位置点处的入射光分别进行不同的频谱调制，以在图像传感器的表面得到与不同位置点对应的入射光携带信息，在本申请实施例中，光滤波器层上的光调制结构对入射光的调制作用相当于输入层到所述线性层的连接权重。同 时，在本申请实施例中，图像传感器通过平方检波响应将与不同位置点经光滤波器层调制后对应的入射光携带信息进行第一次非线性激活处理后转换为与不同位置点对应的电信号，并将与不同位置点对应的电信号发送给所述处理器，处理器将与不同位置点对应的电信号进行全连接处理，或所述处理器将与不同位置点对应的电信号进行全连接处理以及第二次非线性激活处理，得到人工神经网络的输出信号，由此可见，在该智能芯片中，所述光滤波器层对应人工神经网络的输入层、线性层以及所述输入层到所述线性层的连接权重，所述图像传感器的平方检波响应对应人工神经网络的非线性层中的第一次非线性激活函数；所述处理器对应人工神经网络的全连接以及输出层，或，所述处理器对应人工神经网络的全连接、非线性层中的第二次非线性激活函数以及输出层，也即该智能芯片中的光滤波器层和图像传感器实现了人工神经网络中输入层、线性层和部分非线性激活函数的相关功能，也即本申请实施例将现有技术中采用软件实现的人工神经网络中的输入层、线性层以及一部分或全部的非线性激活函数进行了剥离，利用硬件的方式实现了人工神经网络中的输入层、线性层以及一部分或全部的非线性激活函数这些结构，从而使得后续在使用该智能芯片进行人工神经网络智能处理时不需要再进行与输入层、线性层以及一部分或全部非线性激活函数对应的复杂的信号处理和算法处理，只需由智能芯片中的处理器进行与电信号全连接处理或全连接以及第二次非线性激活处理即可，这样可以大幅降低人工神经网络处理时的功耗和延时。由此可见，本申请实施例将光滤波器层作为人工神经网络的输入层、线性层以及所述输入层到所述线性层的连接权重，将图像传感器的平方检波响应作为人工神经网络的非线性层中的第一次非线性激活函数；将处理器作为人工神经网络的全连接以及输出层，或，将处理器对应人工神经网络的全连接、非线性层中的第二次非线性激活函数以及输出层，由此可见，本申请实施例不但能够省去现有技术中与输入层、线性层和一部分非线性激活函数对应的复杂的信号处理和算法处理，而且本申请实施例实际上同时利用了目标对象的图像信息、光谱信息、入射光的角度和入射光的相位信息，即目标对象空间不同点处的入射光携带信息，由此可见，由于目标对象空间不同点处的入射光携带信息涵盖了目标对象的图像、成分、形状、三维深度、 结构等信息，从而在依据目标对象空间不同点处的入射光携带信息进行识别处理时，可以涵盖目标对象的图像、成分、形状、三维深度、结构等多维度的信息，从而可以解决背景技术部分所提到的采用目标对象的二维图像信息难以保证识别的准确性例如难以区分是真实人物还是图片的问题，由此可见，本申请实施例提供的光人工神经网络芯片，不但能够实现低功耗和低延时的效果，还能够实现高准确率的效果，从而可以为应用于智能感知、识别和/或决策等智能处理任务做好准备。
基于上述实施例的内容，在本实施例中，所述光人工神经网络智能芯片用于目标对象的智能处理任务；所述智能处理任务至少包括智能感知、智能识别和智能决策任务中的一种或多种；
目标对象的反射光、透射光和/或辐射光进入至训练好的光人工神经网络智能芯片中，得到所述目标对象的智能处理结果；所述智能处理结果至少包括智能感知结果、智能识别结果和/或智能决策结果中的一种或多种；
训练好的光人工神经网络智能芯片是指包括训练好的光调制结构、图像传感器和处理器的光人工神经网络智能芯片；所述训练好的光调制结构、图像传感器和处理器是指利用与所述智能处理任务对应的输入训练样本和输出训练样本，对包含不同的光调制结构、图像传感器和具有不同的全连接参数的处理器的光人工神经网络智能芯片进行训练得到的满足训练收敛条件的光调制结构、图像传感器和处理器；或，所述训练好的光调制结构、图像传感器和处理器是指利用与所述智能处理任务对应的输入训练样本和输出训练样本，对包含不同的光调制结构、图像传感器和具有不同的全连接参数以及不同的第二次非线性激活参数的处理器的光人工神经网络智能芯片进行训练得到的满足训练收敛条件的光调制结构、图像传感器和处理器。
在本实施例中，所述输入训练样本包括由相应智能处理任务中的目标对象反射、透射和/或辐射的入射光；所述输出训练样本包括目标对象的智能处理结果(如识别结果、感知结果、决策结果或定性分析结果等)。
在本实施例中，光人工神经网络智能芯片可以用于目标对象的智能处理任务，例如，包括智能感知、智能识别和智能决策任务中的一种或多种任务。
在本实施例中，可以理解的是，智能感知是指将物理世界的信号通过摄像头、麦克风或者其他传感器的硬件设备，借助语音识别、图像识别等前沿技术，映射到数字世界，再将这些数字信息进一步提升至可认知的层次，比如记忆、理解、规划、决策等等。智能识别是指利用计算机对图像进行处理、分析和理解，以识别各种不同模式的目标和对象的技术，现阶段智能识别技术一般分为人脸识别与商品识别，人脸识别主要运用在安全检查、身份核验与移动支付中，商品识别主要运用在商品流通过程中，特别是无人货架、智能零售柜等无人零售领域。智能决策是指解决由计算机自动组织和协调多模型运行，对大量数据库中数据的存取和处理，进行相应数据处理和数值计算。
在本实施例中，目标对象的反射光、透射光和/或辐射光进入至训练好的光人工神经网络智能芯片中，得到目标对象的智能处理结果。
在本实施例中，以目标对象的识别任务为例进行说明，可以理解的是，在利用该智能芯片进行识别任务时，首先需要对光人工神经网络智能芯片进行训练，这里对光人工神经网络智能芯片进行训练是指通过训练确定适用于当前识别任务的光调制结构，以及，适用于当前识别任务的全连接参数与非线性激活参数。
可以理解的是，由于光滤波器层对进入光滤波器层的入射光的滤波作用对应人工神经网络输入层到线性层的连接权重，因此，在训练时，改变光滤波器层中的光调制结构相当于改变人工神经网络输入层到线性层的连接权重，通过训练收敛条件，确定出适用于当前识别任务的光调制结构，以及，适用于当前识别任务的全连接参数与非线性激活参数，从而完成对智能芯片的训练。
可以理解的是，在对智能芯片训练后，就可以使用该智能芯片执行识别任务。具体地，携带有目标对象图像信息以及空间光谱信息的入射光进入训练好的智能芯片的光滤波器层1后，光滤波器层1中的光调制结构会对入射光进行调制，调制后的光信号强度由图像传感器2探测并转换成电信号，再由处理器3进行全连接处理或同时进行全连接以及第二次非线性激活处理，就能得到目标对象的识别结果。
如图4所示，对于目标对象识别的完整流程为：宽谱光源100照射到 目标对象200上，然后目标对象的反射光或透射光由光人工神经网络智能芯片300采集，或者目标对象直接向外辐射的光由光人工神经网络智能芯片300采集，由智能芯片中的光滤波器层、图像传感器和处理器进行处理后，即可得到识别结果。
其中，训练好的光人工神经网络智能芯片是指包括训练好的光调制结构、图像传感器和处理器的光人工神经网络智能芯片；所述训练好的光调制结构、图像传感器和处理器是指利用与所述智能处理任务对应的输入训练样本和输出训练样本，对包含不同的光调制结构、图像传感器和具有不同的全连接参数与不同的第二次非线性激活参数的处理器的光人工神经网络智能芯片进行训练得到的满足训练收敛条件的光调制结构、图像传感器和处理器。
举例来说，对于智能识别任务来说，与智能识别任务对应的输入训练样本为识别对象样本，与智能识别任务对应的输出训练样本为所述识别对象样本的识别结果。可以理解的是，对于识别任务来说，由于本实施例提供的智能芯片的优势还在于能够获取到识别对象空间不同点处的图像信息、光谱信息、入射光的角度信息以及入射光的相位信息，因此，为充分利用该优势，对于作为输入训练样本的识别对象样本优先采用真实的识别对象，而不是识别对象的二维图像。当然，这并不代表不可以将二维图像作为识别对象样本。
此外，在本实施例提供的光人工神经网络智能芯片还可以用于目标对象的其他智能处理任务，如智能感知、智能决策等任务。
在本实施例中，光滤波器层1作为神经网络的输入层和线性层，将图像传感器2作为神经网络的非线性层的一部分(也即将图像传感器2的平方检波响应作为神经网络的第一非线性激活函数)，为了使神经网络的损失函数最小，将光滤波器层中的光调制结构对目标对象的入射光中不同波长成分的调制强度作为神经网络的输入层到线性层的连接权重，通过调整滤波器的结构可以调整目标对象的入射光中不同波长成分的调制强度，从而实现对输入层到线性层连接权重的调整，进而优化神经网络的训练。
因此，本实施例中光调制结构是基于神经网络训练得到的，通过计算机对训练样本进行光学仿真，获取训练样本中光调制结构对智能处理任务 中目标对象的入射光不同波长成分的样本调制强度，将样本调制强度作为神经网络的输入层到线性层的连接权重，进行非线性激活，并利用与智能处理任务对应的训练样本进行神经网络训练，直至神经网络收敛时将对应的训练样本光调制结构作为对应智能处理任务的光滤波器层。
由此可见，本实施例通过在物理层实现神经网络的输入层和线性层(光滤波器层)以及非线性层的一部分(图像传感器2的平方检波响应作为神经网络的第一次非线性激活函数)，不但能够省去现有技术中与输入层、线性层以及部分或全部非线性激活函数对应的复杂的信号处理和算法处理。同时本申请实施例实际上同时利用了目标对象空间不同点处的图像信息、光谱信息、入射光的角度信息以及入射光的相位信息，即目标对象空间不同点处的入射光携带信息，由此可见，由于目标对象空间不同点处的入射光携带信息涵盖了目标对象的图像、成分、形状、三维深度、结构等信息，从而在依据目标对象空间不同点处的入射光携带信息进行识别处理时，可以涵盖目标对象的图像、成分、形状、三维深度、结构等多维度的信息，从而可以解决背景技术部分所提到的采用目标对象的二维图像信息难以保证识别的准确性例如难以区分是真实人物还是图片的问题，由此可见，本申请实施例提供的光人工神经网络芯片，不但能够实现低功耗和低延时的效果，还能够实现高准确率的效果，从而可以为应用于智能感知、识别和/或决策等智能处理任务做好准备。
基于上述实施例的内容，在本实施例中，在对包含不同的光调制结构、图像传感器和具有不同的全连接参数的处理器的光人工神经网络智能芯片进行训练，或，对包含不同的光调制结构、图像传感器和具有不同的全连接参数以及不同的第二次非线性激活参数的处理器的光人工神经网络智能芯片进行训练时，所述不同的光调制结构通过采用计算机光学仿真设计的方式设计实现。
在本实施例中，光学仿真能让用户在制作物理原型之前就通过数字环境体验产品，例如对于汽车来说，因为光照及反射光会干扰驾驶员注意力，尤其是在夜间行驶的情况下，合适的光学仿真解决方案不仅能有效帮助用户提高设计效率，还能对光线与材料的交互进行仿真，以便了解产品在真实条件下的展示效果。因此，本实施例通过计算机光学仿真对光调制结构 进行设计，通过光学仿真调整光调制结构，直至神经网络收敛时确定对应的光调制结构为最终需要制作的光调制结构尺寸，节省原型制作时间和成本，提高产品效率，轻松解决复杂的光学问题。
例如，可以通过FDTD软件来对光调制结构进行仿真设计，在光学仿真中改变光调制结构，从而可以准确地预测光调制结构对不同入射光的调制强度，并将其作为神经网络输入层与线性层的连接权重，对光人工神经网络智能芯片进行训练，准确获取光调制结构。
由此可见，本实施例通过采用计算机光学仿真设计的方式设计光调制结构，节省了光调制结构原型制作时间和成本，提高产品效率。
基于上述实施例的内容，在本实施例中，所述光滤波器层中的光调制结构包含规则结构和/或不规则结构；和/或，所述光滤波器层中的光调制结构包含离散型结构和/或连续型结构。
在本实施例中，所述光滤波器层中的光调制结构可以只包括规则结构，也可以只包括不规则结构，还可以既包括规则结构，又包括不规则结构。
在本实施例中，这里光调制结构包括规则结构可以指：光调制结构包含的最小调制单元是规则的结构，如最小调制单元可以长方形、正方形和圆形等规则的图形。此外，这里光调制结构包括规则结构还可以指：光调制结构包含的最小调制单元的排布方式是规则的，如排布方式可以是规则的阵列形式、圆形形式、梯形形式、多边形形式等。此外，这里光调制结构包括规则结构还可以指：光调制结构包含的最小调制单元是规则的结构，同时最小调制单元的排布方式也是规则的等。
在本实施例中，这里的光调制结构包括不规则结构可以指：光调制结构包含的最小调制单元是不规则的结构，如最小调制单元可以不规则多边形、随机形状等不规则的图形。此外，这里光调制结构包括不规则结构还可以指：光调制结构包含的最小调制单元的排布方式是不规则的，如排布方式可以是不规则的多边形形式、随机排列形式等。此外，这里光调制结构包括不规则结构还可以指：光调制结构包含的最小调制单元是不规则的结构，同时最小调制单元的排布方式也是不规则的等。
在本实施例中，所述光滤波器层中的光调制结构可以包括离散型结 构，也可以包括连续型结构，还可以既包括离散型结构，又包括连续型结构。
在本实施例中，这里光调制结构包括连续型结构可以指：光调制结构是由连续的调制图案构成的；这里光调制结构包括离散型结构可以指：光调制结构是由离散的调制图案构成的。
可以理解的是，这里连续的调制图案可以指直线型图案、波浪线型图案、折线型图案等等。
可以理解的是，这里离散的调制图案可以指由离散的图形(如离散的点、离散的三角形、离散的星形等)形成的调制图案。
在本实施例中，需要说明的是，光调制结构对不同波长的光具有不同的调制作用，具体的调制方式包括但不限于散射、吸收、干涉、表面等离激元、谐振增强等。通过设计不同的滤波器结构，使得光通过不同组的滤波器结构后，对应的透射谱不同。
基于上述实施例的内容，在本实施例中，所述光滤波器层为单层结构或多层结构。
在本实施例中，需要说明的是，所述光滤波器层可以为单层滤波器结构，也可以是多层滤波器结构，例如可以是两层、三层、四层等多层结构。
在本实施例中，如图1所示，所述光滤波器层1为单层结构，光滤波器层1的厚度与目标波长范围相关，对于波长400nm～10μm，光栅结构的厚度可以为50nm～5μm。
可以理解的是，由于光滤波器层1的作用是对入射光进行频谱调制，因此，优选折射率高、损耗小的材料制备，例如可以选择硅、锗、锗硅材料、硅的化合物、锗的化合物、III-V族材料等进行制备，其中硅的化合物包括但不限于氮化硅、二氧化硅、碳化硅等。
此外，需要说明的是，为在输入层和线性层之间形成更多或更复杂的连接权重，优选地，可以将所述光滤波器层1设置为多层结构，各层对应的光调制结构可以设置为不同的结构，从而增加光滤波器层对入射光的频谱调制能力，从而可以在输入层和线性层之间形成更多或更为复杂的连接权重，进而提高智能芯片在处理智能任务时的准确度。
此外，需要说明的是，对于包含多层结构的滤波器层来说，每层结构 的材料可以相同，也可以不同，举例来说，对于有二层的光滤波器层1，第一层可以为硅层，第二层可以为氮化硅层。
需要说明的是，光滤波器层1厚度与目标波长范围相关，对于波长400nm～10μm，多层结构总的厚度可以为50nm～5μm。
基于上述实施例的内容，在本实施例中，所述光滤波器层中的光调制结构包含由多个微纳单元组成的单元阵列，每个微纳单元对应图像传感器上的一个或多个像素点；各个微纳单元的结构相同或不同。
在本实施例中，为能够获得呈阵列分布的连接权重(用于连接输入层与线性层之间的连接权重)以便于处理器进行后续的全连接与非线性激活处理，优选地，在本实施例中，光调制结构为阵列结构形式，具体地，光调制结构包括由多个微纳单元组成的单元阵列，每个微纳单元对应图像传感器上的一个或多个像素点。需要说明的是，各个微纳单元的结构可以相同，也可以不同。此外，需要说明的是，各个微纳单元的结构可以是周期的，也可以是非周期的。此外，需要说明的是，各个微纳单元还可以进一步包含有多组微纳结构阵列，各组微纳结构阵列的结构相同或不同等。
下面结合图5～图9进行举例说明，在本实施例中，如图5所示，光滤波器层1包含多个重复连续或离散的微纳单元，如11、22、33、44、55、66，每个微纳单元结构相同(且每个微纳单元为非周期结构)，每个微纳单元对应图像传感器2上的一个或多个像素点；如图6所示，光滤波器层1包含多个重复的微纳单元，如11、22、33、44、55、66，每个微纳单元结构相同(与图5的区别在于图6中每个微纳单元为周期结构)，每个微纳单元对应图像传感器2上的一个或多个像素点；如图7所示，光滤波器层1包含多个重复的微纳单元，如11、22、33、44、55、66，每个微纳单元结构相同(且每个微纳单元为周期结构)，每个微纳单元对应图像传感器2上的一个或多个像素点，与图6的区别在于图7中的各微纳单元内周期阵列的单元形状具有四重旋转对称性；如图8所示，光滤波器层1包含多个微纳单元，如11、22、33、44、55、66，与图6的区别在于每个微纳单元结构互不相同，每个微纳单元对应图像传感器2上的一个或多个像素点，本实施例中光滤波器层1包含多个互不相同的微纳单元，也即智能芯片上不同区域对入射光的调制作用不同，从而提高了设计的自由度，进而 也可以提升识别的准确率。如图9所示，光滤波器层1包含多个重复的微纳单元，如11、22、33、44、55、66，每个微纳单元结构相同，与图5的区别在于各微纳单元是由分立的非周期阵列结构组成的，每个微纳单元对应图像传感器2上的一个或多个像素点。
在本实施例中，微纳单元对不同波长的光具有不同的调制作用，具体的调制方式包括但不限于散射、吸收、干涉、表面等离激元、谐振增强等。通过设计不同的滤波器结构，使得光通过不同组的滤波器结构后，对应的透射谱不同。
基于上述实施例的内容，在本实施例中，所述微纳单元包含规则结构和/或不规则结构；和/或，所述微纳单元包含离散型结构和/或连续型结构。
在本实施例中，所述微纳单元可以只包括规则结构，也可以只包括不规则结构，还可以既包括规则结构，又包括不规则结构。
在本实施例中，这里微纳单元包括规则结构可以指：微纳单元包含的最小调制单元是规则的结构，如最小调制单元可以长方形、正方形和圆形等规则的图形。此外，这里微纳单元包括规则结构还可以指：微纳单元包含的最小调制单元的排布方式是规则的，如排布方式可以是规则的阵列形式、圆形形式、梯形形式、多边形形式等。此外，这里微纳单元包括规则结构还可以指：微纳单元包含的最小调制单元是规则的结构，同时最小调制单元的排布方式也是规则的等。
在本实施例中，这里的微纳单元包括不规则结构可以指：微纳单元包含的最小调制单元是不规则的结构，如最小调制单元可以不规则多边形、随机形状等不规则的图形。此外，这里微纳单元包括不规则结构还可以指：微纳单元包含的最小调制单元的排布方式是不规则的，如排布方式可以是不规则的多边形形式、随机排列形式等。此外，这里微纳单元包括不规则结构还可以指：微纳单元包含的最小调制单元是不规则的结构，同时最小调制单元的排布方式也是不规则的等。
在本实施例中，所述光滤波器层中的微纳单元可以包括离散型结构，也可以包括连续型结构，还可以既包括离散型结构，又包括连续型结构。
在本实施例中，这里微纳单元包括连续型结构可以指：微纳单元是由连续的调制图案构成的；这里微纳单元包括离散型结构可以指：微纳单元 是由离散的调制图案构成的。
可以理解的是，这里连续的调制图案可以指直线型图案、波浪线型图案、折线型图案等等。
可以理解的是，这里离散的调制图案可以指由离散的图形(如离散的点、离散的三角形、离散的星形等)形成的调制图案。
在本实施例中，需要说明的是，不同微纳单元对不同波长的光具有不同的调制作用，具体的调制方式包括但不限于散射、吸收、干涉、表面等离激元、谐振增强等。通过设计不同的微纳单元，使得光通过不同组的微纳单元后，对应的透射谱不同。
基于上述实施例的内容，在本实施例中，所述微纳单元包含有多组微纳结构阵列，各组微纳结构阵列的结构相同或不同。
在本实施例中，如图5所示，光滤波器层1包含多个重复的微纳单元，如11、22、33、44、55、66，每个微纳单元包含有多组微纳结构阵列，如微纳单元11包含4个不同的微纳结构阵列110、111、112和113，滤波单元44包含4个不同的微纳结构阵列440、441、442和443。如图10所示，光滤波器层1包含多个微纳单元，如11、22、33、44、55、66，每个微纳单元包含有多组微纳结构阵列，如微纳单元11包含4个相同的微纳结构阵列110、111、112和113。
需要说明的是，这里只是以包括四组微纳结构阵列的微纳单元进行举例说明，并不起到限制作用，在实际应用中，还可以根据需要设置包括六组、八组或其他数量组微纳结构阵列的微纳单元。
在本实施例中，微纳单元内的每组微纳结构阵列对不同波长的光具有不同的调制作用，并且各组滤波结构之间对输入光的调制作用也不同，具体的调制方式包括但不限于散射、吸收、干涉、表面等离激元、谐振增强等。通过设计不同的微纳结构阵列，使得光通过不同组的微纳结构阵列后，对应的透射谱不同。
基于上述实施例的内容，在本实施例中，各组微纳结构阵列具有宽带滤波或窄带滤波的作用。
在本实施例中，为了能够获取目标对象的入射光不同波长成分的调制强度作为神经网络输入层和线性层的连接权重，通过采用不同的微纳结构 阵列实现宽带滤波和窄带滤波，因此本实施例中微纳结构阵列通过对目标对象的入射光进行宽带滤波或窄带滤波，获取目标对象的入射光不同波长成分的调制强度。如图11和图12所示，光滤波器层中各组微纳结构阵列具有宽带滤波或窄带滤波的作用。
可以理解的是，对于各组微纳结构阵列来说，可以都具备宽带滤波作用，也可以都具备窄带滤波作用，也可以部分具备宽带滤波作用，部分具备窄带滤波作用。此外，各组微纳结构阵列的宽带滤波范围和窄带滤波范围也可以相同或不同。举例来说，通过设计微纳单元内各组微纳结构的周期、占空比、半径、边长等尺寸参数，使其具有窄带滤波作用，即只有一个(或较少个)波长的光可以通过。又如，通过设计微纳单元内各组微纳结构的周期、占空比、半径、边长等尺寸参数，使其具有宽带滤波作用，即允许较多波长或所有波长的光可以通过。
可以理解的是，在具体使用时，可以根据应用场景进行宽带滤波、窄带滤波或其组合的方式确定各组微纳结构阵列的滤波状态。
基于上述实施例的内容，在本实施例中，各组微纳结构阵列为周期结构阵列或非周期结构阵列。
在本实施例中，各组微纳结构阵列可以均为周期结构阵列，也可以均为非周期结构阵列，也可以部分为周期结构阵列，部分为非周期结构阵列。其中，周期结构阵列易于进行光学仿真设计，非周期结构阵列可以实现更复杂的调制作用。
在本实施例中，如图5所示，光滤波器层1包含多个重复的微纳单元，如11、22、33、44、55、66，每个微纳单元由多组微纳结构阵列组成，各微纳结构阵列结构互不相同，且微纳结构阵列为非周期性结构。其中，非周期性结构指微纳结构阵列上的调制孔形状按照非周期排列方式进行排布。如图5所示，微纳单元11包含4个不同的非周期结构阵列110、111、112和113，微纳单元44包含4个不同的非周期结构阵列440、441、442和443，非周期性结构的微纳结构阵列是通过前期针对智能处理任务，由神经网络数据训练设计得到的，通常是不规则形状的结构。如图6所示，光滤波器层1包含多个重复的微纳单元，如11、22、33、44、55、66，每个微纳单元由多组微纳结构阵列组成，各微纳结构阵列结构互不相同，与 图5所不同的是，微纳结构阵列为周期性结构。其中，周期性结构指微纳结构阵列上的调制孔形状按照周期排列方式进行排布，周期的大小通常为20nm～50μm。如图6所示，微纳单元11包含4个不同的周期结构阵列110、111、112和113，微纳单元44包含4个不同的周期结构阵列440、441、442和443，周期性结构的滤波器结构是通过前期针对智能处理任务，由神经网络数据训练设计得到的，通常是不规则形状的结构。如图7所示，光滤波器层1包含多个互不相同的微纳单元，如11、22、33、44、55、66，每个微纳单元由多组微纳结构阵列组成，各微纳结构阵列结构互不相同，且微纳结构阵列为周期性结构。其中，周期性结构指滤波器结构上的形状按照周期排列方式进行排布，周期的大小通常为20nm～50μm。如图7所示，微纳单元11与微纳单元12的微纳结构阵列互不相同，微纳单元11包含4个不同的周期结构阵列110、111、112和113，微纳单元44包含4个不同的周期结构阵列440、441、442和443，周期性结构的微纳结构阵列是通过前期针对智能处理任务，由神经网络数据训练设计得到的，通常是不规则形状的结构。
需要说明的是，图5～图9每个微纳单元均包括四组微纳结构阵列，四组微纳结构阵列分别采用四种不同形状的调制孔形成，四组微纳结构阵列用于对入射光具有不同的调制作用。需要说明的是，这里只是以包括四组微纳结构阵列的微纳单元进行举例说明，并不起到限制作用，在实际应用中，还可以根据需要设置包括六组、八组或其他数量组微纳结构阵列的微纳单元。在本实施例中，四种不同形状可以为圆形、十字形、正多边形和矩形(不限于此)。
在本实施例中，微纳单元内的每组微纳结构阵列对不同波长的光具有不同的调制作用，并且各组微纳结构阵列之间对输入光的调制作用也不同，具体的调制方式包括但不限于散射、吸收、干涉、表面等离激元、谐振增强等。通过设计不同的微纳结构阵列，使得光通过不同组的微纳结构阵列后，对应的透射谱不同。
基于上述实施例的内容，在本实施例中，所述微纳单元包含的多组微纳结构阵列中有一组或多组空结构。
下面结合图9所示例子进行举例说明，在本实施例中，如图9所示， 光滤波器层1包含多个重复的微纳单元，如11、22、33、44、55、66，每个微纳单元由多组微纳结构阵列组成，多组微纳结构阵列对应的结构互不相同，微纳结构阵列为周期性结构，与上述实施例所不同的是，对于任一微纳单元包含一组或多组空结构，所述空结构用于直通入射光。可以理解的是，当多组微纳结构阵列中包含有一组或多组空结构时，可以形成更为丰富的频谱调制效果，从而满足特定场景下的频谱调制需求(或满足特定场景下输入层与线性层之间的特定连接权重需求)。
如图9所示，每个微纳单元均包括一组微纳结构阵列和三组空结构，微纳单元11包含1个非周期结构阵列111，微纳单元22包含1个非周期结构阵列221，微纳单元33包含1个非周期结构阵列331，微纳单元44包含1个非周期结构阵列441，微纳单元55包含1个非周期结构阵列551，微纳单元66包含1个非周期结构阵列661，其中微纳结构阵列用于对入射光进行不同的调制。需要说明的是，这里只是以包括一组微纳结构阵列和三组空结构进行举例说明，并不起到限制作用，在实际应用中，还可以根据需要设置包括一组微纳结构阵列和五组空结构或其他数量组微纳结构阵列的微纳单元。在本实施例中，微纳结构阵列可以采用圆形、十字形、正多边形和矩形的调制孔制成(不限于此)。
需要说明的是，微纳单元包含的多组微纳结构阵列中也可以均不包含空结构，即多组微纳结构阵列可以是非周期结构阵列，也可以是周期结构阵列。
基于上述实施例的内容，在本实施例中，所述微纳单元具有偏振无关特性。
在本实施例中，由于微纳单元具有偏振无关特性，因此使得光滤波器层对入射光的偏振不敏感，从而实现了对入射角、偏振均不敏感的光人工神经网络智能芯片。本申请实施例提供的光人工神经网络智能芯片对入射光的入射角以及偏振特性不敏感，即测量结果不会受到入射光的入射角度和偏振特性的影响，从而可以保证光谱测量性能的稳定性，进而可以保证智能处理的稳定性，如智能感知的稳定性、智能识别的稳定性、智能决策的稳定性等等。需要说明的是，微纳单元也可以具有偏振相关特性。
基于上述实施例的内容，在本实施例中，所述微纳单元具有四重旋转 对称性。
在本实施例中，需要说明的是，四重旋转对称性属于偏振无关特性中的一种具体情况，通过将微纳单元设计为具有四重旋转对称性的结构，可以满足偏振无关特性的要求。
下面结合图7所示例子进行举例说明，在本实施例中，如图7所示，光滤波器层1包含多个重复的微纳单元，如11、22、33、44、55、66，每个微纳单元由多组微纳结构阵列组成，多组微纳结构阵列对应的结构互不相同，微纳结构阵列为周期性结构，与上述实施例所不同的是，每组微纳结构阵列对应的结构可为圆、十字、正多边形、矩形等具有四重旋转对称性的结构，即结构旋转90°、180°、270°后，与原来的结构重合，从而使得结构具有偏振无关的特性，使得在不同偏振光入射都能取得相同的智能识别效果。
基于上述实施例的内容，在本实施例中，所述光滤波器层由一层或多层结构构成；
所述滤波器层是由半导体材料、金属材料、液晶、量子点材料、钙钛矿材料中的一种或多种制备；和/或，所述滤波器层是由光子晶体、超表面、随机结构、纳米结构、金属表面等离激元SPP(Surface Plasmon Polaritons，SPP)微纳结构、可调法布里-珀罗谐振腔(Fabry-perot Cavity，FP腔)中的一种或多种制备的滤波器层。
所述半导体材料包括硅、氧化硅、氮化硅、氧化钛、按照预设比例混合的复合材料以及直接带隙化合物半导体材料中的一种或多种；和/或，所述纳米结构包括纳米点二维材料、纳米柱二维材料和纳米线二维材料中的一种或多种。
其中，光子晶体，以及超表面与随机结构组合可采用CMOS工艺兼容，能够有较好的调制效果，微纳调制结构微孔中也可以填充其他材料做表面的平滑化处理；量子点与钙钛矿可以利用材料本身的谱调制特性使得单个调制结构的体积最小；SPP体积较小，可实现偏振相关的光调制；液晶可以用电压动态调控，提升空间分辨率；可调法布里-珀罗谐振腔可以动态调控，提升空间分辨率。
基于上述实施例的内容，在本实施例中，所述光滤波器层的厚度为0.1 λ～10λ，其中λ表示入射光的中心波长。
在本实施例中，需要说明的是，若光滤波器层的厚度远小于入射光的中心波长，就不能起到有效的频谱调制作用；若光滤波器层的厚度远大于入射光的中心波长，工艺上难以制备，并且会引入较大的光学损耗。因此，本实施例中为了减少光学损耗易于制备，并且保证有效的频谱调制作用，光滤波器层1中每个微纳单元的整体尺寸(面积)通常为λ 2～105λ 2，厚度通常为0.1λ～10λ(λ表示目标对象的入射光的中心波长)。如图5所示，每个微纳单元的整体尺寸为0.5μm 2～40000μm 2，光滤波器层1中的介质材料为多晶硅，厚度为50nm～2μm。
基于上述实施例的内容，在本实施例中，所述图像传感器为下述中的任意一项或多项：
CMOS图像传感器(Contact Image Sensor，CIS)、电荷耦合元件(Charge Coupled Device，CCD)、单光子雪崩二极管(Single Photon Avalanche Diode，SPAD)阵列和焦平面光电探测器阵列。
在本实施例中，需要说明的是，采用晶圆级别的CMOS图像传感器CIS，在晶圆级别实现单片集成，可以最大程度减小图像传感器与光滤波器层之间的距离，有利用于缩小单元的尺寸，降低器件体积和封装成本，SPAD可以用于弱光探测，CCD可以用于强光探测。
在本实施例中，光滤波器层和图像传感器可以由互补金属氧化物半导体(Complementary Metal Oxide Semiconductor，CMOS)集成工艺制造，有利于降低器件失效率，提高器件的成品率并降低成本。例如，可以通过在图像传感器上直接生长一层或多层介质材料，再进行刻蚀，在除去用于刻蚀的牺牲层之前，沉积金属材料，最后再移除牺牲层，进行制备得到光滤波器层。
基于上述实施例的内容，在本实施例中，所述人工神经网络的类型包括：前馈神经网络。
在本实施例中，前馈神经网络(Feedforward Neural Network，FNN)，又称作深度前馈网络(Deep Feedforward Network，DFN)、多层感知机(Multi-Layer Perceptron，MLP)，是一种最简单的神经网络，各神经元分层排列。每个神经元只与前一层的神经元相连。接收前一层的输出，并 输出给下一层，各层间没有反馈。前馈神经网络结构简单，易于在硬件上实现，应用广泛，能够以任意精度逼近任意连续函数及平方可积函数.而且可以精确实现任意有限训练样本集。前馈网络是一种静态非线性映射。通过简单非线性处理单元的复合映射，可获得复杂的非线性处理能力。
基于上述实施例的内容，在本实施例中，所述光滤波器层与所述图像传感器之前设置有透光介质层。
在本实施例中，需要说明的是，在所述光滤波器层与所述图像传感器之间设置透光介质层，可以有效将光滤波器层与图像传感器层分开，避免两者相互干扰。
基于上述实施例的内容，在本实施例中，所述图像传感器为前照式，包括：自上而下设置的金属线层和光探测层，所述光滤波器层集成在所述金属线层远离所述光探测层的一面；或，
所述图像传感器为背照式，包括：自上而下设置的光探测层和金属线层，所述光滤波器层集成在所述光探测层远离所述金属线层的一面。
在本实施例中，如图13所示为前照式的图像传感器，硅探测层21在金属线层22下方，光滤波器层1直接集成到金属线层22上。
在本实施例中，与图13不同的是，图14所示为背照式的图像传感器，硅探测层21在金属线层22上方，光滤波器层1直接集成到硅探测层21上。
需要说明的是，对于背照式的图像传感器，硅探测层21在金属线层22上方，可以减少金属线层对入射光的影响，从而可以提高器件的量子效率。
根据上面的内容可知，本实施例将光滤波器层作为人工神经网络的输入层和线性层，将图像传感器作为人工神经网络的非线性层的一部分(将图像传感器的平方检波响应作为人工神经网络的第一次非线性激活函数)，将光滤波器层对进入光滤波器层的入射光的滤波作用作为输入层到线性层的连接权重，本实施例提供的光人工神经网络智能芯片中的光滤波器层和图像传感器通过硬件的方式实现了人工神经网络中输入层、线性层以及部分非线性激活函数的相关功能，从而使得后续在使用该智能芯片进行智能处理时不需要再进行与输入层、线性层以及部分非线性激活函数对 应的复杂的信号处理和算法处理，这样可以大幅降低人工神经网络处理时的功耗和延时。此外，本实施例由于同时利用了目标对象空间不同点处的图像信息、光谱信息、入射光的角度信息以及入射光的相位信息，从而可以更加准确地实现对目标对象的智能处理。
由此可见，在本申请实施例中，将光滤波器层作为人工神经网络的输入层和线性层，图像传感器作为人工神经网络的非线性层的一部分，将物体的空间光谱信息投影到探测器的光电流响应中，并在处理器中实现电信号的全连接和二次非线性激活，实现了低功耗、低延时和高准确率的智能感知、识别和/或决策等功能。本申请实施例中基于光滤波器和图像传感器的光人工神经网智能芯片有以下效果：将人工神经网络部分嵌入包含各种光滤波器层的图像传感器中，实现快速准确的智能感知、识别和/或决策功能。此外，本申请实施例还可以在晶圆级别实现单片集成，从而可以最大程度减小传感器与光滤波器层之间的距离，有利于缩小单元的尺寸，降低器件体积和封装成本。
基于相同的发明构思，本申请另一实施例提供了一种智能设备，包括：如上面实施例所述的光人工神经网络智能芯片。所述智能设备包括智能手机、智能电脑、智能识别设备、智能感知设备和智能决策设备中的一种或多种。
由于本实施例提供的智能设备包括上述实施例所述的光人工神经网络智能芯片，因此，本实施例提供的智能设备具备上述实施例所述的光人工神经网络智能芯片的全部有益效果，由于上述实施例已经对此进行了较为详尽的描述，因此本实施例不再赘述。
基于相同的发明构思，本申请另一实施例提供了一种如上面所述实施例的光人工神经网络智能芯片的制备方法，如图15所示，具体包括如下步骤：
步骤1510、在所述图像传感器的表面制备包含有光调制结构的光滤波器层；
步骤1520、生成具备对信号进行全连接处理功能的处理器或生成具备对信号进行全连接处理以及第二次非线性激活处理功能的处理器；
步骤1530、连接所述图像传感器和所述处理器；
其中，所述光滤波器层用于通过所述光调制结构对进入至所述光调制结构不同位置点处的入射光分别进行不同的频谱调制，以在所述图像传感器的表面得到与不同位置点对应的入射光携带信息；所述入射光携带信息包括光强度分布信息、光谱信息、所述入射光的角度信息以及所述入射光的相位信息；
所述图像传感器通过平方检波响应将与不同位置点经光滤波器层调制后对应的入射光携带信息进行第一次非线性激活处理后转换为与不同位置点对应的电信号，并将与不同位置点对应的电信号发送给所述处理器；
所述处理器将与不同位置点对应的电信号进行全连接处理，或，所述处理器将与不同位置点对应的电信号进行全连接处理以及第二次非线性激活处理，得到人工神经网络的输出信号。
在本实施例中，还包括对所述光人工神经智能芯片的训练过程，具体包括：
利用与所述智能处理任务对应的输入训练样本以及输出训练样本，对包含不同的光调制结构、图像传感器和具有不同的全连接参数的处理器的光人工神经智能芯片进行训练得到满足训练收敛条件的光调制结构、图像传感器和处理器，并将满足训练收敛条件的光调制结构、图像传感器和处理器作为训练好的光调制结构、图像传感器和处理器；
或，利用与所述智能处理任务对应的输入训练样本以及输出训练样本，对包含不同的光调制结构、图像传感器和具有不同的全连接参数以及不同的第二次非线性激活参数的处理器的光人工神经智能芯片进行训练得到满足训练收敛条件的光调制结构、图像传感器和处理器，并将满足训练收敛条件的光调制结构、图像传感器和处理器作为训练好的光调制结构、图像传感器和处理器。
可以理解的是，在对包含不同的光调制结构、图像传感器和具有不同的全连接参数的处理器的光人工神经智能芯片进行训练，或，对包含不同的光调制结构、图像传感器和具有不同的全连接参数以及不同的第二次非线性激活参数的处理器的光人工神经智能芯片进行训练时，所述不同的光调制结构通过采用计算机光学仿真设计的方式设计实现。
在本实施例中，在所述图像传感器的感光区域的表面制备包含有光调制结构的光滤波器层，包括：
在所述图像传感器的表面生长一层或多层预设材料；
对所述一层或多层预设材料进行光调制结构图案的干法刻蚀，得到包含有光调制结构的光滤波器层；
或对所述一层或多层预设材料进行压印转移，得到包含有光调制结构的光滤波器层；
或通过对所述一层或多层预设材料进行外加动态调控，得到包含有光调制结构的光滤波器层；
或对所述一层或多层预设材料进行分区打印，得到包含有光调制结构的光滤波器层；
或对所述一层或多层预设材料进行分区材料生长，得到包含有光调制结构的光滤波器层；
或对所述一层或多层预设材料进行量子点转移，得到包含有光调制结构的光滤波器层。
当所述光人工神经网络智能芯片用于目标对象的智能处理任务时，利用与所述智能处理任务对应的输入训练样本和输出训练样本，对包含不同的光调制结构、图像传感器和具有不同的全连接参数的处理器的光人工神经网络智能芯片进行训练，得到满足训练收敛条件的光调制结构、图像传感器和处理器；或，对包含不同的光调制结构、图像传感器和具有不同的全连接参数以及不同的第二次非线性激活参数的处理器的光人工神经网络智能芯片进行训练，得到满足训练收敛条件的光调制结构、图像传感器和处理器。
在本实施例中，需要说明的是，如图1所示，光滤波器层1可以通过在图像传感器2上直接生长一层或多层介质材料，再进行刻蚀，在除去用于刻蚀的牺牲层之前，沉积金属材料，最后再移除牺牲层，进行制备得到。通过设计光调制结构的尺寸参数，各个单元能够对目标范围内不同波长的光有不同的调制作用，并且该调制作用对入射角度、偏振均不敏感。光滤波器层1中的每个单元对应图像传感器2上一个或多个像素。1是直接在2上制备的。
在本实施例中，需要说明的是，如图14所示，假设图像传感器2为背照式结构，则光滤波器层1可以在背照式的图像传感器的硅探测器层21上直接刻蚀，然后再沉积金属进行制备得到。
此外，需要说明的是，所述光滤波器层上的光调制结构可以通过对一层或多层预设材料进行光调制结构图案的干法刻蚀，干法刻蚀就是直接将图像传感器感光区域表面的一层或多层预设材料中不需要的部分去除，得到包含有光调制结构的光滤波器层；或对一层或多层预设材料进行压印转移，压印转移是在其它衬底上通过刻蚀制备所需的结构，再通过PDMS等材料将结构转移到图像传感器的感光区域，得到包含有光调制结构的光滤波器层；或通过对一层或多层预设材料进行外加动态调控，外加动态调控是采用有源材料，然后外加电极通过改变电压来调控相应区域的光调制特性，得到包含有光调制结构的光滤波器层；或对一层或多层预设材料进行分区打印，分区打印是分区采用打印的技术，得到包含有光调制结构的光滤波器层；或对一层或多层预设材料进行分区材料生长，得到包含有光调制结构的光滤波器层；或对一层或多层预设材料进行量子点转移，得到包含有光调制结构的光滤波器层。
此外，需要说明的是，由于本实施例提供的制备方法是上述实施例中的光人工神经网络智能芯片的制备方法，因此，关于一些原理和结构等方面的详细内容，可以参见上述实施例的介绍，本实施例对此不再赘述。
环境问题是我国可持续发展所面临的主要问题，环境检测工作可以实现对环境的有效保护和治理。环境检测包括空气、水质、土壤等的实时监测，为环境管理提供科学、准确的有效监测，并依此制定合理的解决对策。传统的环境污染监测是以湿式化学技术和吸气取样后的实验分析为基础。近年来分析仪器的快速发展能够满足许多环境污染监测的需要，但这些仪器通常只限于单点测量。相比而言，光学和光谱学技术以其大范围、多成分检测、时间分辨率佳、连续实时监测方式成为环境污染监测的理想工具。因此，实现大范围、高分辨率、小体积、低成本、安全可靠的实时环境污染检测芯片对环境治理和保护具有重要意义。
为此，基于前述实施例介绍的光人工神经网络智能芯片，本实施例提供一种用于多成分环境检测的新型光电芯片，该芯片由光滤波器层构成光 人工神经网络的输入层和线性层，由图像传感器构成光人工神经网络的第一次非线性激活函数，通过采集检测环境样本的图像信息和光谱信息，可以实现快速准确、安全可靠的空气、水、土壤等样本的环境污染物识别及定性分析。下面具体对本实施例的内容进行详细解释和说明。
本申请实施例还提供了一种光人工神经网络环保监测芯片，用于环保监测智能处理任务，包括：光滤波器层、图像传感器和处理器；所述光滤波器层对应人工神经网络的输入层、线性层以及所述输入层到所述线性层的连接权重，所述图像传感器的平方检波响应对应人工神经网络的非线性层中的第一次非线性激活函数；所述处理器对应人工神经网络的全连接以及输出层，或，所述处理器对应人工神经网络的全连接、非线性层中的第二次非线性激活函数以及输出层；
所述光滤波器层设置于所述图像传感器表面，所述光滤波器层包含有光调制结构，所述光调制结构用于对进入至所述光调制结构不同位置点处的入射光分别进行不同的频谱调制，以在所述图像传感器的表面得到与不同位置点对应的入射光携带信息；所述入射光携带信息包括光强度分布信息、光谱信息、所述入射光的角度信息以及所述入射光的相位信息；所述入射光包括环境污染物的反射光、透射光和/或辐射光；
所述图像传感器通过平方检波响应将与不同位置点经光滤波器层调制后对应的入射光携带信息进行第一次非线性激活处理后转换为与不同位置点对应的电信号，并将与不同位置点对应的电信号发送给所述处理器；
所述处理器将与不同位置点对应的电信号进行全连接处理，或，所述处理器将与不同位置点对应的电信号进行全连接处理以及第二次非线性激活处理，得到环保监测智能处理结果；
其中，所述环保监测智能处理任务包括环境污染物的识别和/或定性分析；所述环保监测智能处理结果包括环境污染物的识别结果和/或环境污染定性分析结果。
本实施例实现了一种能够实现人工神经网络功能的全新光人工神经网络环保监测芯片，用于环保监测任务，本申请实施例在硬件芯片上嵌入了人工神经网络，将硬件芯片上的光滤波器层作为人工神经网络的输入层 和线性层，将硬件芯片上的光滤波器层对入射光的滤波作用作为输入层到线性层的连接权重，将硬件芯片上的图像传感器的平方检波响应作为人工神经网络的非线性层中的第一次非线性激活函数，本申请实施例将环境污染物的空间光谱信息入射到预先训练好的硬件芯片中，通过硬件芯片对环境污染物空间不同点处的图像信息、光谱信息、入射光的角度信息以及入射光的相位信息进行人工神经网络分析进而得出环保监测结果，需要说明的是，本申请实施例实现了低功耗、安全可靠的快速准确环保监测。
可以理解的是，在该光人工神经网络环保监测芯片中，其上的硬件结构-光滤波器层对应人工神经网络的输入层和线性层，其上的硬件结构-图像传感器对应人工神经网络的非线性层的一部分；所述处理器对应人工神经网络的非线性层的另一部分以及输出层。具体地，光滤波器层设置于图像传感器的表面，光滤波器层包含有光调制结构，光调制结构用于对进入至光调制结构不同位置点处的入射光分别进行不同的频谱调制，以在图像传感器的表面得到与不同位置点对应的入射光携带信息，在本申请实施例中，光滤波器层上的光调制结构对入射光的调制作用相当于输入层到所述线性层的连接权重。同时，在本申请实施例中，图像传感器通过平方检波响应将与不同位置点经光滤波器层调制后对应的入射光携带信息进行第一次非线性激活处理后转换为与不同位置点对应的电信号，并将与不同位置点对应的电信号发送给所述处理器，处理器将与不同位置点对应的电信号进行全连接处理，或所述处理器将与不同位置点对应的电信号进行全连接处理以及第二次非线性激活处理，得到人工神经网络的输出信号，由此可见，在该光人工神经网络环保监测芯片中，以硬件方式实现的光滤波器层和图像传感器替代或实现了现有人工神经网络中的输入层、线性层和部分非线性激活函数的相关功能，也即本申请实施例将现有技术中采用软件实现的人工神经网络中的输入层、线性层以及一部分或全部的非线性激活函数进行了剥离，利用硬件的方式实现了人工神经网络中的输入层、线性层以及一部分或全部的非线性激活函数这些结构，从而使得后续在使用该光人工神经网络环保监测芯片进行人工神经网络智能处理时不需要再进行与输入层、线性层以及一部分或全部非线性激活函数对应的复杂的信号处理和算法处理，只需由光人工神经网络环保监测芯片中的处理器进行与 电信号全连接处理或全连接以及第二次非线性激活处理即可，这样可以大幅降低人工神经网络处理时的功耗和延时。由此可见，本申请实施例将光滤波器层作为人工神经网络的输入层、线性层以及所述输入层到所述线性层的连接权重，将图像传感器的平方检波响应作为人工神经网络的非线性层中的第一次非线性激活函数；将处理器作为人工神经网络的全连接以及输出层，或，将处理器对应人工神经网络的全连接、非线性层中的第二次非线性激活函数以及输出层，由此可见，本申请实施例不但能够省去现有技术中与输入层、线性层和一部分非线性激活函数对应的复杂的信号处理和算法处理，而且本申请实施例实际上同时利用了环境污染物的图像信息、光谱信息、入射光的角度和入射光的相位信息，即环境污染物的空间不同点处的入射光携带信息，由此可见，由于环境污染物的空间不同点处的入射光携带信息涵盖了环境污染物的图像、成分、形状、三维深度、结构等信息，从而在依据环境污染物的空间不同点处的入射光携带信息进行识别处理时，可以涵盖环境污染物的图像、成分、形状、三维深度、结构等多维度的信息，从而可以准确地测得环境污染物的信息，此外，由于光谱信息以其大范围、多成分检测、时间分辨率佳、能够连续实时监测等优势使得该芯片能够成为环境污染监测的理想工具，从而解决目前的环保监测仪器只适用于单点监测的问题，本实施例提供的芯片能够实现大范围、高分辨率、小体积、低成本、安全可靠的实时环境污染检测，这对环境治理和保护具有重要意义。
进一步地，所述光人工神经网络环保监测芯片包括训练好的光调制结构、图像传感器和处理器；
所述训练好的光调制结构、图像传感器和处理器是指利用与所述环保监测智能处理任务对应的输入训练样本以及输出训练样本，对包含不同的光调制结构、图像传感器和具有不同的全连接参数的处理器的光人工神经网络环保监测芯片进行训练得到的满足训练收敛条件的光调制结构、图像传感器和处理器；或，所述训练好的光调制结构、图像传感器和处理器是指利用与所述环保监测智能处理任务对应的输入训练样本以及输出训练样本，对包含不同的光调制结构、图像传感器和具有不同的全连接参数以及不同的第二次非线性激活参数的处理器的光人工神经网络环保监测芯 片进行训练得到的满足训练收敛条件的光调制结构、图像传感器和处理器；
其中，所述输入训练样本包括由不同环境污染物反射、透射和/或辐射的入射光；所述输出训练样本包括相应的环境污染物识别结果；和/或，所述输入训练样本包括由不同环境污染物反射、透射和/或辐射的入射光；所述输出训练样本包括相应的环境污染定性分析结果。
进一步地，在对包含不同的光调制结构、图像传感器和具有不同的全连接参数的处理器的光人工神经网络环保监测芯片进行训练，或，对包含不同的光调制结构、图像传感器和具有不同的全连接参数以及不同的第二次非线性激活参数的处理器的光人工神经网络环保监测芯片进行训练时，所述不同的光调制结构通过采用计算机光学仿真设计的方式设计实现。
在本实施例中，可以先对大量环境污染物样本进行采集，通过数据训练得到线性层的权重，即光滤波器层的系统函数，便可以逆向设计出所需的光滤波器层，将其集成在图像传感器上方。在实际环境检测过程中，再利用制作完成的光滤波器层的输出，对电信号全连接层的权重进一步训练并优化，便可实现高准确率的光人工神经网络，完成对环境污染物样本的快速准确识别和定性分析。由此可见，该芯片实际上同时利用了环境污染物空间不同点处的图像信息、光谱信息、入射光的角度信息以及入射光的相位信息，提高了环境污染检测的准确性和多样性，并在硬件上部分实现了人工神经网络，提高了环境污染检测的速度。此外，该芯片方案可以利用现有的CMOS工艺实现量产，降低了器件的体积、功耗和成本。
进一步地，所述光滤波器层中的光调制结构包含规则结构和/或不规则结构；和/或，所述光滤波器层中的光调制结构包含离散型结构和/或连续型结构。
进一步地，所述光滤波器层为单层结构或多层结构。
进一步地，所述光滤波器层中的光调制结构包含由多个微纳单元组成的单元阵列，每个微纳单元对应图像传感器上的一个或多个像素点；各个微纳单元的结构相同或不同。
进一步地，所述微纳单元包含规则结构和/或不规则结构；和/或，所述微纳单元包含离散型结构和/或连续型结构。
进一步地，所述微纳单元包含有多组微纳结构阵列，各组微纳结构阵列的结构相同或不同。
进一步地，各组微纳结构阵列具有宽带滤波或窄带滤波的作用。
进一步地，各组微纳结构阵列为周期结构阵列或非周期结构阵列。
进一步地，所述微纳单元包含的多组微纳结构阵列中有一组或多组空结构。
进一步地，所述微纳单元具有偏振无关特性。
进一步地，所述微纳单元具有四重旋转对称性。
进一步地，所述滤波器层是由半导体材料、金属材料、液晶、量子点材料、钙钛矿材料中的一种或多种制备；和/或，所述滤波器层是由光子晶体、超表面、随机结构、纳米结构、金属表面等离激元SPP微纳结构、可调法布里-珀罗谐振腔中的一种或多种制备的滤波器层。
进一步地，所述半导体材料包括硅、氧化硅、氮化硅、氧化钛、按照预设比例混合的复合材料以及直接带隙化合物半导体材料中的一种或多种；和/或，所述纳米结构包括纳米点二维材料、纳米柱二维材料和纳米线二维材料中的一种或多种。
进一步地，所述光滤波器层的厚度为0.1λ～10λ，其中λ表示入射光的中心波长。
本申请实施例还提供了一种环保监测设备，包括上面实施例所述的光人工神经网络环保监测芯片。该环保监测设备可以为环境状况探测仪、污染物含量分析仪等。
本申请实施例还提供了一种如上面所述的光人工神经网络环保监测芯片的制备方法，包括：
在所述图像传感器的表面制备包含有光调制结构的光滤波器层；
生成具备对信号进行全连接处理功能的处理器或生成具备对信号进行全连接处理以及第二次非线性激活处理功能的处理器；
连接所述图像传感器和所述处理器；
其中，所述光滤波器层用于通过所述光调制结构对进入至所述光调制结构不同位置点处的入射光分别进行不同的频谱调制，以在所述图像传感器的表面得到与不同位置点对应的入射光携带信息；所述入射光携带信息 包括光强度分布信息、光谱信息、所述入射光的角度信息以及所述入射光的相位信息；
所述图像传感器通过平方检波响应将与不同位置点经光滤波器层调制后对应的入射光携带信息进行第一次非线性激活处理后转换为与不同位置点对应的电信号，并将与不同位置点对应的电信号发送给所述处理器；
所述处理器将与不同位置点对应的电信号进行全连接处理，或，所述处理器将与不同位置点对应的电信号进行全连接处理以及第二次非线性激活处理，得到环保监测智能处理结果。
进一步地，所述光人工神经网络环保监测芯片的制备方法，还包括：对所述光人工神经网络环保监测芯片的训练过程，具体包括：
利用与所述环保监测智能处理任务对应的输入训练样本以及输出训练样本，对包含不同的光调制结构、图像传感器和具有不同的全连接参数的处理器的光人工神经网络环保监测芯片进行训练得到满足训练收敛条件的光调制结构、图像传感器和处理器，并将满足训练收敛条件的光调制结构、图像传感器和处理器作为训练好的光调制结构、图像传感器和处理器；
或，利用与所述环保监测智能处理任务对应的输入训练样本以及输出训练样本，对包含不同的光调制结构、图像传感器和具有不同的全连接参数以及不同的第二次非线性激活参数的处理器的光人工神经网络环保监测芯片进行训练得到满足训练收敛条件的光调制结构、图像传感器和处理器，并将满足训练收敛条件的光调制结构、图像传感器和处理器作为训练好的光调制结构、图像传感器和处理器。
进一步地，在对包含不同的光调制结构、图像传感器和具有不同的全连接参数的处理器的光人工神经网络环保监测芯片进行训练，或，对包含不同的光调制结构、图像传感器和具有不同的全连接参数以及不同的第二次非线性激活参数的处理器的光人工神经网络环保监测芯片进行训练时，所述不同的光调制结构通过采用计算机光学仿真设计的方式设计实现。
在本实施例中，需要说明的是，该芯片在图像传感器的感光区域表面上直接制备微纳调制结构，若干个分立或连续的微纳结构构成一个单元， 不同位置处的微纳调制结构对入射光具有不同的频谱调制作用，共同构成了光滤波器层。这些微纳调制结构对入射光不同波长成分的调制强度对应于人工神经网络的连接强度(线性层权重)。光滤波器层组成了人工神经网络的输入层和线性层以及输入层到线性层的连接权重，其中光滤波器层对输入信号在频谱上进行加权，再由图像传感器对加权后的信号转化为电信号(图像传感器处理的这部分相当于第一次非线性激活函数)，然后再通过处理器将不同位置图像传感器输出的电信号进行全连接，并经由第二次非线性激活函数实现完整的光人工神经网络。对于环境污染检测，可以先对大量环境污染物样本进行采集，通过数据训练得到线性层的权重，即光滤波器层的系统函数，便可以逆向设计出所需的光滤波器层，将其集成在图像传感器上方，在实际环境检测过程中，再利用制作完成的光滤波器层的输出，对电信号全连接层的权重进一步训练并优化，便可实现高准确率的光人工神经网络，完成对环境污染物样本的快速准确识别和定性分析。
可以理解的是，对于环境污染检测，通过在计算机上对微纳调制结构进行光学仿真，可以得到调制结构对入射光不同波长成分的调制强度(透射率)，将其作为人工神经网络输入层到线性层的连接权重，并在处理器中实现非线性激活函数，事先对大量环境污染物样本进行采集并进行数据训练，可以设计出所需的微纳调制结构并进行制备，在芯片上实现人工神经网络的输入层、线性层和一部分非线性激活函数。
此外，可以理解的是，参见图5，图像传感器2可以采用CIS晶圆，光滤波器层1直接在CIS晶圆上制备。光滤波器层1包含多个重复的调制单元，每个调制单元又包含4个不同的连续非周期结构阵列，非周期结构阵列的基本单元是通过前期采集大量不同的环境污染物样本，由人工神经网络数据训练设计得到的，通常是不规则形状的结构。各个非周期结构阵列对入射光具有不同的宽谱调制作用，每个调制单元的整体尺寸为0.5μm 2～40000μm 2。光滤波器层1中的介质材料为多晶硅，厚度为50nm～2μm。可以理解的是，CIS晶圆包括硅探测器层和金属线层，响应范围为可见到近红外波段；CIS晶圆是裸露的，未制备上拜尔滤光片阵列和微透镜阵列。每个调制单元对应CIS晶圆上的多个传感器单元。
对于环境检测芯片检测污染物的完整流程为：如图16所示，检测仪器下的光源照射到检测样本上，然后反射光由芯片采集，进行处理后可得到识别结果。其中，光滤波器层和图像传感器都可以由半导体CMOS集成工艺制造，在晶圆级别实现单片集成，有利于减小传感器与光滤波器层之间的距离，缩小器件的体积，降低封装成本，同时能够实现便携式检测。
本实施例提供的基于微纳调制结构和图像传感器的光人工神经网络的环保监测芯片具有如下效果：
A、将人工神经网络部分嵌入包含各种光滤波器层的图像传感器中，实现安全可靠、快速准确的环境污染检测；B、可检测样本包括但不限于空气、水、土壤，引入人工神经网训练识别污染物，检测范围大、样本丰富，且识别准确性高、定性分析精准；C、可以通过CMOS工艺一次流片完成对该芯片的制备，有利于降低器件失效率，提高器件的成品良率，并降低成本。D、在晶圆级别实现单片集成，可以最大程度减小传感器与光滤波器层之间的距离，有利于缩小单元的尺寸，降低器件体积和封装成本。
需要说明的是，关于本实施例提供的光人工神经网络环保监测芯片的详细结构类描述，可参见前述实施例中对于光人工神经网络芯片的介绍，为避免赘述，此处不再介绍。此外，关于光人工神经网络环保监测芯片制备方法的详细介绍，也可以参见前述实施例中对于光人工神经网络芯片制备方法的介绍，此处不再赘述。
可以理解的是，指纹识别技术是一种生物特征识别技术，广泛应用于智能手机解锁、门禁系统、银行密码验证等领域。指纹识别的主要过程包括指纹采集、指纹预处理、指纹特征提取与比对。指纹图像的采集是指纹识别的关键，获取指纹图像的方式主要有光学采集、电容传感器采集、热敏传感器采集、超声波采集等。其中，电容传感器通常安放在手机背部下方采集指纹，结合热敏传感器时，还能实现活体检测，但无法用于屏下指纹识别，这是因为屏幕模组的厚度限制了电容传感器的信号采集。现有的屏下指纹识别技术主要有光学采集和超声波采集两种方案，但这两种方案只采集了指纹的纹路图像信息，限制了指纹识别准确率，并且器件的体积和功耗仍然较大。此外，某些人群的指纹特征较少，不易识别；亲属之间指纹存在相似性，易导致识别错误；遗留在物体表面的指纹信息可能被盗 用，安全性不高。因此，需要将指纹信息与其他信息相结合以提高识别的准确性和安全性。综合来看，实现高准确率、小体积、低功耗、安全可靠的快速屏下指纹识别器件具有重要意义。
为此，基于前述实施例介绍的光人工神经网络智能芯片，本实施例提供一种用于指纹准确识别的新型光电芯片，该芯片由光滤波器层构成光人工神经网络的输入层和线性层，由图像传感器构成光人工神经网络的第一次非线性激活函数，通过采集待识别指纹的图像信息和光谱信息，可以实现快速准确、安全可靠的指纹识别。下面具体对本实施例的内容进行详细解释和说明。
本申请实施例还提供了一种光人工神经网络指纹识别芯片，用于指纹识别处理任务，包括：光滤波器层、图像传感器和处理器；所述光滤波器层对应人工神经网络的输入层、线性层以及所述输入层到所述线性层的连接权重，所述图像传感器的平方检波响应对应人工神经网络的非线性层中的第一次非线性激活函数；所述处理器对应人工神经网络的全连接以及输出层，或，所述处理器对应人工神经网络的全连接、非线性层中的第二次非线性激活函数以及输出层；
所述光滤波器层设置于所述图像传感器表面，所述光滤波器层包含有光调制结构，所述光调制结构用于对进入至所述光调制结构不同位置点处的入射光分别进行不同的频谱调制，以在所述图像传感器的表面得到与不同位置点对应的入射光携带信息；所述入射光携带信息包括光强度分布信息、光谱信息、所述入射光的角度信息以及所述入射光的相位信息；所述入射光包括用户指纹的反射光、透射光和/或辐射光；
所述图像传感器通过平方检波响应将与不同位置点经光滤波器层调制后对应的入射光携带信息进行第一次非线性激活处理后转换为与不同位置点对应的电信号，并将与不同位置点对应的电信号发送给所述处理器；
所述处理器将与不同位置点对应的电信号进行全连接处理，或，所述处理器将与不同位置点对应的电信号进行全连接处理以及第二次非线性激活处理，得到指纹识别处理结果。
由此可见，本实施例实现了一种能够实现人工神经网络功能的全新光 人工神经网络指纹识别芯片，用于指纹识别任务，本申请实施例在硬件芯片上嵌入了人工神经网络，将硬件芯片上的光滤波器层作为人工神经网络的输入层和线性层，将硬件芯片上的光滤波器层对入射光的滤波作用作为输入层到线性层的连接权重，将硬件芯片上的图像传感器的平方检波响应作为人工神经网络的非线性层中的第一次非线性激活函数，本申请实施例将指纹空间不同点处的图像信息、光谱信息、入射光的角度信息以及入射光的相位信息入射到预先训练好的硬件芯片中，通过硬件芯片对指纹的空间光谱信息进行人工神经网络分析进而得出指纹识别结果，需要说明的是，本申请实施例实现了低功耗、安全可靠的快速准确指纹识别。
可以理解的是，在该光人工神经网络指纹识别芯片中，其上的硬件结构-光滤波器层对应人工神经网络的输入层和线性层，其上的硬件结构-图像传感器对应人工神经网络的非线性层的一部分；所述处理器对应人工神经网络的非线性层的另一部分以及输出层。具体地，光滤波器层设置于图像传感器的表面，光滤波器层包含有光调制结构，光调制结构用于对进入至光调制结构不同位置点处的入射光分别进行不同的频谱调制，以在图像传感器的表面得到与不同位置点对应的入射光携带信息，在本申请实施例中，光滤波器层上的光调制结构对入射光的调制作用相当于输入层到所述线性层的连接权重。同时，在本申请实施例中，图像传感器通过平方检波响应将与不同位置点经光滤波器层调制后对应的入射光携带信息进行第一次非线性激活处理后转换为与不同位置点对应的电信号，并将与不同位置点对应的电信号发送给所述处理器，处理器将与不同位置点对应的电信号进行全连接处理，或所述处理器将与不同位置点对应的电信号进行全连接处理以及第二次非线性激活处理，得到人工神经网络的输出信号，由此可见，在该光人工神经网络指纹识别芯片中，以硬件方式实现的光滤波器层和图像传感器替代或实现了现有人工神经网络中的输入层、线性层和部分非线性激活函数的相关功能，也即本申请实施例将现有技术中采用软件实现的人工神经网络中的输入层、线性层以及一部分或全部的非线性激活函数进行了剥离，利用硬件的方式实现了人工神经网络中的输入层、线性层以及一部分或全部的非线性激活函数这些结构，从而使得后续在使用该光人工神经网络指纹识别芯片进行人工神经网络智能处理时不需要再进 行与输入层、线性层以及一部分或全部非线性激活函数对应的复杂的信号处理和算法处理，只需由光人工神经网络指纹识别芯片中的处理器进行与电信号全连接处理或全连接以及第二次非线性激活处理即可，这样可以大幅降低人工神经网络处理时的功耗和延时。由此可见，本申请实施例不但能够省去现有技术中与输入层、线性层和一部分非线性激活函数对应的复杂的信号处理和算法处理，而且本申请实施例实际上同时利用了指纹的图像信息、光谱信息、入射光的角度和入射光的相位信息，即指纹的空间不同点处的入射光携带信息，由此可见，由于指纹空间不同点处的入射光携带信息涵盖了指纹的图像、成分、形状、三维深度、结构等信息，从而在依据指纹空间不同点处的入射光携带信息进行识别处理时，可以涵盖指纹的图像、成分、形状、三维深度、结构等多维度的信息，从而可以准确地进行指纹识别。
进一步地，所述光人工神经网络指纹识别芯片包括训练好的光调制结构、图像传感器和处理器；
所述训练好的光调制结构、图像传感器和处理器是指利用与所述指纹识别处理任务对应的输入训练样本和输出训练样本，对包含不同的光调制结构、图像传感器和具有不同的全连接参数的处理器的光人工神经网络指纹识别芯片进行训练得到的满足训练收敛条件的光调制结构、图像传感器和处理器；或，所述训练好的光调制结构、图像传感器和处理器是指利用与所述指纹识别处理任务对应的输入训练样本和输出训练样本，对包含不同的光调制结构、图像传感器和具有不同的全连接参数以及不同的第二次非线性激活参数的处理器的光人工神经网络指纹识别芯片进行训练得到的满足训练收敛条件的光调制结构、图像传感器和处理器；
其中，所述输入训练样本包括由不同人体指纹反射、透射和/或辐射的入射光；所述输出训练样本包括相应的指纹识别结果。
在本实施例中，对于屏下指纹识别，可以先对大量人群的指纹进行采集，通过数据训练得到输入层到线性层的权重，即光滤波器层的系统函数，便可以逆向设计出所需的光滤波器层，将其集成在图像传感器上方。在实际使用时，用户录入指纹的过程中，再利用制作完成的光滤波器层的输出，对电信号全连接层的权重进一步训练并优化，便可实现高准确率的光人工 神经网络，完成对该用户指纹的快速准确识别。
可以理解的是，光滤波器层上的调制结构的具体调制图案是通过前期采集大量人群的指纹，由人工神经网络数据训练设计得到的，通常是不规则形状的结构，当然也有可能是规则形状的结构。
如图17所示，对于屏下指纹识别的完整流程为：手机屏幕下的光源照射到用户手指上，然后反射光由芯片采集，经内部处理后得到识别结果。
可以理解的是，该芯片实际上同时利用了指纹的图像信息和光谱信息，提高了指纹识别的准确性和安全性。同时该芯片在硬件上部分实现了人工神经网络，提高了指纹识别的速度。此外，该芯片方案可以利用现有的CMOS工艺实现量产，降低了器件的体积、功耗和成本。
进一步地，在对包含不同的光调制结构、图像传感器和具有不同的全连接参数的处理器的光人工神经网络指纹识别芯片进行训练，或，对包含不同的光调制结构、图像传感器和具有不同的全连接参数以及不同的第二次非线性激活参数的处理器的光人工神经网络指纹识别芯片进行训练时，所述不同的光调制结构通过采用计算机光学仿真设计的方式设计实现。
进一步地，所述光滤波器层中的光调制结构包含规则结构和/或不规则结构；和/或，所述光滤波器层中的光调制结构包含离散型结构和/或连续型结构。
进一步地，所述光滤波器层为单层结构或多层结构。
进一步地，所述光滤波器层中的光调制结构包含由多个微纳单元组成的单元阵列，每个微纳单元对应图像传感器上的一个或多个像素点；各个微纳单元的结构相同或不同。
进一步地，所述微纳单元包含规则结构和/或不规则结构；和/或，所述微纳单元包含离散型结构和/或连续型结构。
进一步地，所述微纳单元包含有多组微纳结构阵列，各组微纳结构阵列的结构相同或不同。
进一步地，各组微纳结构阵列具有宽带滤波或窄带滤波的作用。
进一步地，各组微纳结构阵列为周期结构阵列或非周期结构阵列。
进一步地，所述微纳单元包含的多组微纳结构阵列中有一组或多组空结构。
进一步地，所述微纳单元具有偏振无关特性。
进一步地，所述微纳单元具有四重旋转对称性。
进一步地，所述滤波器层是由半导体材料、金属材料、液晶、量子点材料、钙钛矿材料中的一种或多种制备；和/或，所述滤波器层是由光子晶体、超表面、随机结构、纳米结构、金属表面等离激元SPP微纳结构、可调法布里-珀罗谐振腔中的一种或多种制备的滤波器层。
进一步地，所述半导体材料包括硅、氧化硅、氮化硅、氧化钛、按照预设比例混合的复合材料以及直接带隙化合物半导体材料中的一种或多种；和/或，所述纳米结构包括纳米点二维材料、纳米柱二维材料和纳米线二维材料中的一种或多种。
进一步地，所述光滤波器层的厚度为0.1λ～10λ，其中λ表示入射光的中心波长。
本申请实施例还提供了一种指纹识别设备，包括上面实施例所述的光人工神经网络指纹识别芯片。该指纹识别设备可以为便携式的指纹识别设备，也可以是安装在固定位置的指纹识别设备。
本申请实施例还提供了一种如上面任一项所述的光人工神经网络指纹识别芯片的制备方法，包括：
在所述图像传感器的表面制备包含有光调制结构的光滤波器层；
生成具备对信号进行全连接处理功能的处理器或生成具备对信号进行全连接处理以及第二次非线性激活处理功能的处理器；
连接所述图像传感器和所述处理器；
其中，所述光滤波器层用于通过所述光调制结构对进入至所述光调制结构不同位置点处的入射光分别进行不同的频谱调制，以在所述图像传感器的表面得到与不同位置点对应的入射光携带信息；所述入射光携带信息包括光强度分布信息、光谱信息、所述入射光的角度信息以及所述入射光的相位信息；
所述图像传感器通过平方检波响应将与不同位置点经光滤波器层调制后对应的入射光携带信息进行第一次非线性激活处理后转换为与不同位置点对应的电信号，并将与不同位置点对应的电信号发送给所述处理器；
所述处理器将与不同位置点对应的电信号进行全连接处理，或，所述处理器将与不同位置点对应的电信号进行全连接处理以及第二次非线性激活处理，得到指纹识别处理结果。
进一步地，所述光人工神经网络指纹识别芯片的制备方法，还包括：对所述光人工神经网络指纹识别芯片的训练过程，具体包括：
利用与所述指纹识别处理任务对应的输入训练样本和输出训练样本，对包含不同的光调制结构、图像传感器和具有不同的全连接参数的处理器的光人工神经网络指纹识别芯片进行训练得到满足训练收敛条件的光调制结构、图像传感器和处理器，并将满足训练收敛条件的光调制结构、图像传感器和处理器作为训练好的光调制结构、图像传感器和处理器；
或，利用与所述指纹识别处理任务对应的输入训练样本和输出训练样本，对包含不同的光调制结构、图像传感器和具有不同的全连接参数以及不同的第二次非线性激活参数的处理器的光人工神经网络指纹识别芯片进行训练得到满足训练收敛条件的光调制结构、图像传感器和处理器，并将满足训练收敛条件的光调制结构、图像传感器和处理器作为训练好的光调制结构、图像传感器和处理器。
需要说明的是，本实施例提供的基于微纳调制结构和图像传感器的光人工神经网屏下指纹识别芯片有以下效果：A、将人工神经网络部分嵌入硬件芯片中，实现安全可靠、快速准确的屏下指纹识别。B、可以通过CMOS工艺一次流片完成对该芯片的制备，有利于降低器件失效率，提高器件的成品良率，并降低成本。C、在晶圆级别实现单片集成，可以最大程度减小传感器与光滤波器层之间的距离，有利于缩小单元的尺寸，降低器件体积和封装成本。
需要说明的是，关于本实施例提供的光人工神经网络指纹识别芯片的详细结构类描述，可参见前述实施例中对于光人工神经网络芯片的介绍，为避免赘述，此处不再介绍。此外，关于光人工神经网络指纹识别芯片制备方法的详细介绍，也可以参见前述实施例中对于光人工神经网络芯片制备方法的介绍，此处不再赘述。
可以理解的是，人脸识别技术是一种生物特征识别技术，广泛应用于门禁考勤系统、刑侦系统、电子商务等领域。人脸识别的主要过程包括人 脸图像的采集、预处理、特征提取、匹配与识别。然而，只利用人脸的图像信息难以保证识别的准确性，例如难以区分真实的人脸和人脸照片；即使结合深度信息，也难以区分人脸模型和真实人脸。因此，挖掘更多的人脸信息，实现更高准确率、安全可靠的快速人脸识别具有重要意义。
为此，基于前述实施例介绍的光人工神经网络智能芯片，本实施例提供一种用于人脸准确识别的新型光电芯片，该芯片由光滤波器层构成光人工神经网络的输入层和线性层，由图像传感器构成光人工神经网络的第一次非线性激活函数，通过采集待识别人脸空间不同点处的图像信息、光谱信息、入射光的角度信息以及入射光的相位信息，可以实现快速准确、安全可靠的人脸识别。下面具体对本实施例的内容进行详细解释和说明。
本申请实施例还提供了一种光人工神经网络人脸识别芯片，用于人脸识别处理任务，包括：光滤波器层、图像传感器和处理器；所述光滤波器层对应人工神经网络的输入层、线性层以及所述输入层到所述线性层的连接权重，所述图像传感器的平方检波响应对应人工神经网络的非线性层中的第一次非线性激活函数；所述处理器对应人工神经网络的全连接以及输出层，或，所述处理器对应人工神经网络的全连接、非线性层中的第二次非线性激活函数以及输出层；
所述光滤波器层设置于所述图像传感器表面，所述光滤波器层包含有光调制结构，所述光调制结构用于对进入至所述光调制结构不同位置点处的入射光分别进行不同的频谱调制，以在所述图像传感器的表面得到与不同位置点对应的入射光携带信息；所述入射光携带信息包括光强度分布信息、光谱信息、所述入射光的角度信息以及所述入射光的相位信息；所述入射光包括用户人脸的反射光、透射光和/或辐射光；
所述图像传感器通过平方检波响应将与不同位置点经光滤波器层调制后对应的入射光携带信息进行第一次非线性激活处理后转换为与不同位置点对应的电信号，并将与不同位置点对应的电信号发送给所述处理器；
所述处理器将与不同位置点对应的电信号进行全连接处理，或，所述处理器将与不同位置点对应的电信号进行全连接处理以及第二次非线性激活处理，得到人脸识别处理结果。
由此可见，本实施例实现了一种能够实现人工神经网络功能的全新光人工神经网络人脸识别芯片，用于人脸识别任务，本申请实施例在硬件芯片上嵌入了人工神经网络，将硬件芯片上的光滤波器层作为人工神经网络的输入层和线性层，将硬件芯片上的光滤波器层对入射光的滤波作用作为输入层到线性层的连接权重，将硬件芯片上的图像传感器的平方检波响应作为人工神经网络的非线性层中的第一次非线性激活函数，本申请实施例将人脸的空间光谱信息入射到预先训练好的硬件芯片中，通过硬件芯片对人脸空间不同点处的图像信息、光谱信息、入射光的角度信息以及入射光的相位信息进行人工神经网络分析进而得出人脸识别结果，需要说明的是，本申请实施例实现了低功耗、安全可靠的快速准确人脸识别。
可以理解的是，在该光人工神经网络人脸识别芯片中，其上的硬件结构-光滤波器层对应人工神经网络的输入层和线性层，其上的硬件结构-图像传感器对应人工神经网络的非线性层的一部分；所述处理器对应人工神经网络的非线性层的另一部分以及输出层。具体地，光滤波器层设置于图像传感器的表面，光滤波器层包含有光调制结构，光调制结构用于对进入至光调制结构不同位置点处的入射光分别进行不同的频谱调制，以在图像传感器的表面得到与不同位置点对应的入射光携带信息，在本申请实施例中，光滤波器层上的光调制结构对入射光的调制作用相当于输入层到所述线性层的连接权重。同时，在本申请实施例中，图像传感器通过平方检波响应将与不同位置点经光滤波器层调制后对应的入射光携带信息进行第一次非线性激活处理后转换为与不同位置点对应的电信号，并将与不同位置点对应的电信号发送给所述处理器，处理器将与不同位置点对应的电信号进行全连接处理，或所述处理器将与不同位置点对应的电信号进行全连接处理以及第二次非线性激活处理，得到人工神经网络的输出信号，由此可见，在该光人工神经网络人脸识别芯片中，以硬件方式实现的光滤波器层和图像传感器替代或实现了现有人工神经网络中的输入层、线性层和部分非线性激活函数的相关功能，也即本申请实施例将现有技术中采用软件实现的人工神经网络中的输入层、线性层以及一部分或全部的非线性激活函数进行了剥离，利用硬件的方式实现了人工神经网络中的输入层、线性层以及一部分或全部的非线性激活函数这些结构，从而使得后续在使用该 光人工神经网络人脸识别芯片进行人工神经网络智能处理时不需要再进行与输入层、线性层以及一部分或全部非线性激活函数对应的复杂的信号处理和算法处理，只需由光人工神经网络人脸识别芯片中的处理器进行与电信号全连接处理或全连接以及第二次非线性激活处理即可，这样可以大幅降低人工神经网络处理时的功耗和延时。由此可见，本申请实施例不但能够省去现有技术中与输入层、线性层和一部分非线性激活函数对应的复杂的信号处理和算法处理，而且本申请实施例实际上同时利用了人脸的图像信息、光谱信息、入射光的角度和入射光的相位信息，即人脸的空间不同点处的入射光携带信息，由此可见，由于人脸空间不同点处的入射光携带信息涵盖了人脸的图像、成分、形状、三维深度、结构等信息，从而在依据人脸空间不同点处的入射光携带信息进行识别处理时，可以涵盖人脸的图像、成分、形状、三维深度、结构等多维度的信息，从而可以准确地进行人脸识别。
进一步地，所述光人工神经网络人脸识别芯片包括训练好的光调制结构、图像传感器和处理器；
所述训练好的光调制结构、图像传感器和处理器是指利用与所述人脸识别处理任务对应的输入训练样本和输出训练样本，对包含不同的光调制结构、图像传感器和具有不同的全连接参数的处理器的光人工神经网络人脸识别芯片进行训练得到的满足训练收敛条件的光调制结构、图像传感器和处理器；或，所述训练好的光调制结构、图像传感器和处理器是指利用与所述人脸识别处理任务对应的输入训练样本和输出训练样本，对包含不同的光调制结构、图像传感器和具有不同的全连接参数以及不同的第二次非线性激活参数的处理器的光人工神经网络人脸识别芯片进行训练得到的满足训练收敛条件的光调制结构、图像传感器和处理器；
其中，所述输入训练样本包括由不同人脸反射、透射和/或辐射的入射光；所述输出训练样本包括相应的人脸识别结果。
进一步地，在对包含不同的光调制结构、图像传感器和具有不同的全连接参数的处理器的光人工神经网络人脸识别芯片进行训练，或，对包含不同的光调制结构、图像传感器和具有不同的全连接参数以及不同的第二次非线性激活参数的处理器的光人工神经网络人脸识别芯片进行训练时， 所述不同的光调制结构通过采用计算机光学仿真设计的方式设计实现。
在本实施例中，由于增加了光的频谱调制，可以实现输入为物体图像及其频谱的人工神经网光电芯片，可以实现快速准确、安全可靠的活体人脸识别。
在本实施例中，对于人脸识别，可以先对大量人群的人脸进行采集，通过数据训练得到输入层到线性层的权重，即光滤波器层的系统函数，便可以逆向设计出所需的光滤波器层，将其集成在图像传感器上方。在实际训练时，利用待识别的人脸样本，利用制作完成的光滤波器层的输出，对电信号全连接层的权重进一步训练并优化，便可实现高准确率的光人工神经网络，完成对用户人脸的快速准确识别。
可以理解的是，光滤波器层上的调制结构的具体调制图案是通过前期采集大量人群的人脸，由人工神经网络数据训练设计得到的，通常是不规则形状的结构，当然也有可能是规则形状的结构。
如图18所示，对于人脸识别的完整流程为：环境光或其他光源照射到用户人脸上，然后反射光由芯片采集，经内部处理后得到识别结果。
可以理解的是，该芯片实际上同时利用了人脸的图像信息和光谱信息，提高了人脸识别的准确性和安全性，尤其对于非活体的人脸模型也能够准确地将其排除在外。同时该芯片在硬件上部分实现了人工神经网络，提高了指纹识别的速度。此外，该芯片方案可以利用现有的CMOS工艺实现量产，降低了器件的体积、功耗和成本。
进一步地，所述光滤波器层中的光调制结构包含规则结构和/或不规则结构；和/或，所述光滤波器层中的光调制结构包含离散型结构和/或连续型结构。
进一步地，所述光滤波器层为单层结构或多层结构。
进一步地，所述光滤波器层中的光调制结构包含由多个微纳单元组成的单元阵列，每个微纳单元对应图像传感器上的一个或多个像素点；各个微纳单元的结构相同或不同。
进一步地，所述微纳单元包含规则结构和/或不规则结构；和/或，所述微纳单元包含离散型结构和/或连续型结构。
进一步地，所述微纳单元包含有多组微纳结构阵列，各组微纳结构阵 列的结构相同或不同。
进一步地，各组微纳结构阵列具有宽带滤波或窄带滤波的作用。
进一步地，各组微纳结构阵列为周期结构阵列或非周期结构阵列。
进一步地，所述微纳单元包含的多组微纳结构阵列中有一组或多组空结构。
进一步地，所述微纳单元具有偏振无关特性，特别地，所述微纳单元具有四重旋转对称性。
进一步地，所述滤波器层是由半导体材料、金属材料、液晶、量子点材料、钙钛矿材料中的一种或多种制备；和/或，所述滤波器层是由光子晶体、超表面、随机结构、纳米结构、金属表面等离激元SPP微纳结构、可调法布里-珀罗谐振腔中的一种或多种制备的滤波器层。
进一步地，所述半导体材料包括硅、氧化硅、氮化硅、氧化钛、按照预设比例混合的复合材料以及直接带隙化合物半导体材料中的一种或多种；和/或，所述纳米结构包括纳米点二维材料、纳米柱二维材料和纳米线二维材料中的一种或多种。
进一步地，所述光滤波器层的厚度为0.1λ～10λ，其中λ表示入射光的中心波长。
本申请实施例还提供了一种人脸识别设备，包括上面实施例所述的光人工神经网络人脸识别芯片。该人脸识别设备可以为便携式的人脸识别设备，也可以是安装在固定位置的人脸识别设备。由于该人脸识别设备具有和上述光人工神经网络人脸识别芯片类似的有益效果，故此处不再赘述。
本申请实施例还提供了一种如上面所述的光人工神经网络人脸识别芯片的制备方法，包括：
在所述图像传感器的表面制备包含有光调制结构的光滤波器层；
生成具备对信号进行全连接处理功能的处理器或生成具备对信号进行全连接处理以及第二次非线性激活处理功能的处理器；
连接所述图像传感器和所述处理器；
其中，所述光滤波器层用于通过所述光调制结构对进入至所述光调制结构不同位置点处的入射光分别进行不同的频谱调制，以在所述图像传感器的表面得到与不同位置点对应的入射光携带信息；所述入射光携带信息 包括光强度分布信息、光谱信息、所述入射光的角度信息以及所述入射光的相位信息；
所述图像传感器通过平方检波响应将与不同位置点经光滤波器层调制后对应的入射光携带信息进行第一次非线性激活处理后转换为与不同位置点对应的电信号，并将与不同位置点对应的电信号发送给所述处理器；
所述处理器将与不同位置点对应的电信号进行全连接处理，或，所述处理器将与不同位置点对应的电信号进行全连接处理以及第二次非线性激活处理，得到人脸识别处理结果。
进一步地，所述光人工神经网络人脸识别芯片的制备方法，还包括：对所述光人工神经网络人脸识别芯片的训练过程，具体包括：
利用与所述人脸识别处理任务对应的输入训练样本和输出训练样本，对包含不同的光调制结构、图像传感器和具有不同的全连接参数的处理器的光人工神经网络人脸识别芯片进行训练得到满足训练收敛条件的光调制结构、图像传感器和处理器，并将满足训练收敛条件的光调制结构、图像传感器和处理器作为训练好的光调制结构、图像传感器和处理器；
或，利用与所述人脸识别处理任务对应的输入训练样本和输出训练样本，对包含不同的光调制结构、图像传感器和具有不同的全连接参数以及不同的第二次非线性激活参数的处理器的光人工神经网络人脸识别芯片进行训练得到满足训练收敛条件的光调制结构、图像传感器和处理器，并将满足训练收敛条件的光调制结构、图像传感器和处理器作为训练好的光调制结构、图像传感器和处理器。
进一步地，在对包含不同的光调制结构、图像传感器和具有不同的全连接参数的处理器的光人工神经网络人脸识别芯片进行训练，或，对包含不同的光调制结构、图像传感器和具有不同的全连接参数以及不同的第二次非线性激活参数的处理器的光人工神经网络人脸识别芯片进行训练时，所述不同的光调制结构通过采用计算机光学仿真设计的方式设计实现。
需要说明的是，本实施例提供的基于微纳调制结构和图像传感器的光人工神经网络人脸识别芯片有以下效果：A、将人工神经网络部分嵌入硬件芯片中，实现安全可靠、快速准确的人脸识别。B、可以通过CMOS工 艺一次流片完成对该芯片的制备，有利于降低器件失效率，提高器件的成品良率，并降低成本。C、在晶圆级别实现单片集成，可以最大程度减小传感器与光滤波器层之间的距离，有利于缩小单元的尺寸，降低器件体积和封装成本。
需要说明的是，关于本实施例提供的光人工神经网络人脸识别芯片的详细结构类描述，可参见前述实施例中对于光人工神经网络芯片的介绍，为避免赘述，此处不再介绍。此外，关于光人工神经网络人脸识别芯片制备方法的详细介绍，也可以参见前述实施例中对于光人工神经网络芯片制备方法的介绍，此处不再赘述。
机器视觉技术是人工智能的一个分支技术，通过机器代替人眼进行观测和判断，广泛应用于工业生产、质量检测、快递分拣、无人驾驶等领域。一个典型的机器视觉系统包括成像系统、图像处理系统、通信及IO系统和联动机构。其中，成像系统负责采集目标物体的图像信息，是机器视觉技术的关键。目前机器视觉技术仅仅利用了物体的图像信息，在进行测量和识别时的准确性和可靠性还有待提升。因此，利用物体其他维度的信息，实现更高准确率和可靠性的增强机器视觉具有重要意义。
为此，基于前述实施例介绍的光人工神经网络智能芯片，本实施例提供一种用于增强机器视觉的新型光电芯片，该芯片由光滤波器层构成光人工神经网络的输入层和线性层，由图像传感器构成光人工神经网络的第一次非线性激活函数，通过采集待识别对象目标的图像信息和光谱信息，可以实现快速准确、安全可靠的对象目标识别，从而可以实现更高准确率和可靠性的增强机器视觉。下面具体对本实施例的内容进行详细解释和说明。
本申请实施例还提供了一种光人工神经网络机器视觉增强芯片，用于机器视觉智能处理任务，包括：光滤波器层、图像传感器和处理器；所述光滤波器层对应人工神经网络的输入层、线性层以及所述输入层到所述线性层的连接权重，所述图像传感器的平方检波响应对应人工神经网络的非线性层中的第一次非线性激活函数；所述处理器对应人工神经网络的全连接以及输出层，或，所述处理器对应人工神经网络的全连接、非线性层中的第二次非线性激活函数以及输出层；
所述光滤波器层设置于所述图像传感器表面，所述光滤波器层包含有光调制结构，所述光调制结构用于对进入至所述光调制结构不同位置点处的入射光分别进行不同的频谱调制，以在所述图像传感器的表面得到与不同位置点对应的入射光携带信息；所述入射光携带信息包括光强度分布信息、光谱信息、所述入射光的角度信息以及所述入射光的相位信息；所述入射光包括机器视觉场景下目标对象的反射光、透射光和/或辐射光；
所述图像传感器通过平方检波响应将与不同位置点经光滤波器层调制后对应的入射光携带信息进行第一次非线性激活处理后转换为与不同位置点对应的电信号，并将与不同位置点对应的电信号发送给所述处理器；
所述处理器将与不同位置点对应的电信号进行全连接处理，或，所述处理器将与不同位置点对应的电信号进行全连接处理以及第二次非线性激活处理，得到机器视觉智能处理结果；
其中，所述机器视觉智能处理任务包括机器视觉场景下目标对象的识别和/或定性分析；所述机器视觉智能处理结果包括机器视觉场景下目标对象的识别结果和/或定性分析结果。
由此可见，本实施例实现了一种能够实现人工神经网络功能的全新光人工神经网络增强机器视觉识别芯片，用于各种机器视觉应用场景下的目标对象识别任务，本申请实施例在硬件芯片上嵌入了人工神经网络，将硬件芯片上的光滤波器层作为人工神经网络的输入层和线性层，将硬件芯片上的光滤波器层对入射光的滤波作用作为输入层到线性层的连接权重，将硬件芯片上的图像传感器的平方检波响应作为人工神经网络的非线性层中的第一次非线性激活函数，本申请实施例将机器视觉应用场景下的目标对象空间不同点处的图像信息、光谱信息、入射光的角度信息以及入射光的相位信息入射到预先训练好的硬件芯片中，通过硬件芯片对目标对象空间不同点处的图像信息、光谱信息、入射光的角度信息以及入射光的相位信息进行人工神经网络分析进而得出目标对象的识别结果，需要说明的是，本申请实施例实现了低功耗、安全可靠的快速准确对象识别。
可以理解的是，在该光人工神经网络增强机器视觉芯片中，其上的硬件结构-光滤波器层对应人工神经网络的输入层和线性层，其上的硬件结构 -图像传感器对应人工神经网络的非线性层的一部分；所述处理器对应人工神经网络的非线性层的另一部分以及输出层。具体地，光滤波器层设置于图像传感器的表面，光滤波器层包含有光调制结构，光调制结构用于对进入至光调制结构不同位置点处的入射光携带信息分别进行不同的频谱调制，以在图像传感器的表面得到与不同位置点对应的入射光携带信息，在本申请实施例中，光滤波器层上的光调制结构对入射光的调制作用相当于输入层到所述线性层的连接权重。同时，在本申请实施例中，图像传感器通过平方检波响应将与不同位置点经光滤波器层调制后对应的入射光携带信息进行第一次非线性激活处理后转换为与不同位置点对应的电信号，并将与不同位置点对应的电信号发送给所述处理器，处理器将与不同位置点对应的电信号进行全连接处理，或所述处理器将与不同位置点对应的电信号进行全连接处理以及第二次非线性激活处理，得到人工神经网络的输出信号，由此可见，在该光人工神经网络增强机器视觉芯片中，以硬件方式实现的光滤波器层和图像传感器替代或实现了现有人工神经网络中的输入层、线性层和部分非线性激活函数的相关功能，也即本申请实施例将现有技术中采用软件实现的人工神经网络中的输入层、线性层以及一部分或全部的非线性激活函数进行了剥离，利用硬件的方式实现了人工神经网络中的输入层、线性层以及一部分或全部的非线性激活函数这些结构，从而使得后续在使用该芯片进行基于人工神经网络的机器视觉智能处理时不需要再进行与输入层、线性层以及一部分或全部非线性激活函数对应的复杂的信号处理和算法处理，只需由光人工神经网络增强机器视觉芯片中的处理器进行与电信号全连接处理或全连接以及第二次非线性激活处理即可，这样可以大幅降低人工神经网络处理时的功耗和延时。由此可见，本申请实施例不但能够省去现有技术中与输入层、线性层和一部分非线性激活函数对应的复杂的信号处理和算法处理，而且本申请实施例实际上同时利用了机器视觉应用场景下的目标对象的图像信息、光谱信息、入射光的角度和入射光的相位信息，即机器视觉应用场景下的目标对象的空间不同点处的入射光携带信息，由此可见，由于目标对象空间不同点处的入射光携带信息涵盖了目标对象的图像、成分、形状、三维深度、结构等信息，从而在依据目标对象空间不同点处的入射光携带信息进行识别处理时，可 以涵盖目标对象的图像、成分、形状、三维深度、结构等多维度的信息，从而可以准确地进行目标对象识别。
进一步地，所述光人工神经网络机器视觉增强芯片包括训练好的光调制结构、图像传感器和处理器；
所述训练好的光调制结构、图像传感器和处理器是指利用与所述机器视觉智能处理任务对应的输入训练样本以及输出训练样本，对包含不同的光调制结构、图像传感器和具有不同的全连接参数的处理器的光人工神经网络机器视觉增强芯片进行训练得到的满足训练收敛条件的光调制结构、图像传感器和处理器；或，所述训练好的光调制结构、图像传感器和处理器是指利用与所述机器视觉智能处理任务对应的输入训练样本以及输出训练样本，对包含不同的光调制结构、图像传感器和具有不同的全连接参数以及不同的第二次非线性激活参数的处理器的光人工神经网络机器视觉增强芯片进行训练得到的满足训练收敛条件的光调制结构、图像传感器和处理器；
其中，所述输入训练样本包括由特定机器视觉场景下的目标对象反射、透射和/或辐射的入射光，所述输出训练样本包括特定机器视觉场景下的目标对象识别结果；和/或，所述输入训练样本包括由特定机器视觉场景下的目标对象反射、透射和/或辐射的入射光，所述输出训练样本包括特定机器视觉场景下的目标对象定性分析结果。
在本实施例中，由于增加了光的频谱调制，可以实现输入为物体图像及其频谱的人工神经网光电芯片，可以实现准确、安全可靠的机器视觉目标对象的识别。
在本实施例中，对于特定的机器视觉应用场景，可以通过数据训练，得到输入层到线性层的连接权重，即光滤波器层的系统函数，便可以逆向设计出所需的光滤波器层，将其集成在图像传感器上方，进而可以制备得到在机器视觉应用场景下能够快速准确识别判断的光人工神经网络增强机器视觉芯片。
可以理解的是，光滤波器层上的调制结构的具体调制图案是通过前期采集相应机器视觉应用场景大量目标对象，由人工神经网络数据训练设计得到的，通常是不规则形状的结构，当然也有可能是规则形状的结构。
如图19所示，对于增强机器视觉应用的完整流程为：光源照射到被检测物体上，然后反射光由芯片采集，再由处理器进行算法处理，即可得到识别结果，最后由控制机构做出相应操作。
可以理解的是，该芯片实际上同时利用了被检测物体的图像信息和光谱信息，提高了被检测物体识别的准确性和安全性，同时该芯片在硬件上部分实现了人工神经网络，提高了被检测物体识别的速度。此外，该芯片方案可以利用现有的CMOS工艺实现量产，降低了器件的体积、功耗和成本。
进一步地，在对包含不同的光调制结构、图像传感器和具有不同的全连接参数的处理器的光人工神经网络机器视觉增强芯片进行训练，或，对包含不同的光调制结构、图像传感器和具有不同的全连接参数以及不同的第二次非线性激活参数的处理器的光人工神经网络机器视觉增强芯片进行训练时，所述不同的光调制结构通过采用计算机光学仿真设计的方式设计实现。
进一步地，所述光滤波器层中的光调制结构包含规则结构和/或不规则结构；和/或，所述光滤波器层中的光调制结构包含离散型结构和/或连续型结构。
进一步地，所述光滤波器层为单层结构或多层结构。
进一步地，所述光滤波器层中的光调制结构包含由多个微纳单元组成的单元阵列，每个微纳单元对应图像传感器上的一个或多个像素点；各个微纳单元的结构相同或不同。
进一步地，所述微纳单元包含规则结构和/或不规则结构；和/或，所述微纳单元包含离散型结构和/或连续型结构。
进一步地，所述微纳单元包含有多组微纳结构阵列，各组微纳结构阵列的结构相同或不同。
进一步地，各组微纳结构阵列具有宽带滤波或窄带滤波的作用。
进一步地，各组微纳结构阵列为周期结构阵列或非周期结构阵列。
进一步地，所述微纳单元包含的多组微纳结构阵列中有一组或多组空结构。
进一步地，所述微纳单元具有偏振无关特性。
进一步地，所述微纳单元具有四重旋转对称性。
进一步地，所述滤波器层是由半导体材料、金属材料、液晶、量子点材料、钙钛矿材料中的一种或多种制备；和/或，所述滤波器层是由光子晶体、超表面、随机结构、纳米结构、金属表面等离激元SPP微纳结构、可调法布里-珀罗谐振腔中的一种或多种制备的滤波器层。
进一步地，所述半导体材料包括硅、氧化硅、氮化硅、氧化钛、按照预设比例混合的复合材料以及直接带隙化合物半导体材料中的一种或多种；和/或，所述纳米结构包括纳米点二维材料、纳米柱二维材料和纳米线二维材料中的一种或多种。
进一步地，所述光滤波器层的厚度为0.1λ～10λ，其中λ表示入射光的中心波长。
本申请实施例还提供了一种增强机器视觉系统，包括控制机构以及如上面实施例所述的光人工神经网络增强机器视觉芯片。其中，控制机构与光人工神经网络增强机器视觉芯片相连，控制机构根据人工神经网络增强机器视觉芯片的识别结果进行相应的控制，从而完成机器视觉场景下的应用目标。这里，控制机构可能是机械手，机械臂，也可以是智能控制按钮等，本实施例对此不作限定。控制结构可以根据光人工神经网络增强机器视觉芯片的识别结果按照预定的控制逻辑进行相应的控制。
可以理解的是，机器视觉技术是人工智能的一个分支技术，通过机器代替人眼进行观测和判断，广泛应用于工业生产、质量检测、快递分拣、无人驾驶等领域。一个典型的机器视觉系统包括成像系统、图像处理系统、通信及IO系统和联动机构。其中，成像系统负责采集目标物体的图像信息，是机器视觉技术的关键。目前机器视觉技术仅仅利用了物体的图像信息，在进行测量和识别时的准确性和可靠性还有待提升。而本实施例综合利用了物体光谱信息，从而可以实现更高准确率和可靠性的增强机器视觉系统。
本申请实施例还提供了一种如上面任一项所述的光人工神经网络机器视觉增强芯片的制备方法，包括：
在所述图像传感器的表面制备包含有光调制结构的光滤波器层；
生成具备对信号进行全连接处理功能的处理器或生成具备对信号进 行全连接处理以及第二次非线性激活处理功能的处理器；
连接所述图像传感器和所述处理器；
其中，所述光滤波器层用于通过所述光调制结构对进入至所述光调制结构不同位置点处的入射光分别进行不同的频谱调制，以在所述图像传感器的表面得到与不同位置点对应的入射光携带信息；所述入射光携带信息包括光强度分布信息、光谱信息、所述入射光的角度信息以及所述入射光的相位信息；
所述图像传感器通过平方检波响应将与不同位置点经光滤波器层调制后对应的入射光携带信息进行第一次非线性激活处理后转换为与不同位置点对应的电信号，并将与不同位置点对应的电信号发送给所述处理器；
所述处理器将与不同位置点对应的电信号进行全连接处理，或，所述处理器将与不同位置点对应的电信号进行全连接处理以及第二次非线性激活处理，得到机器视觉智能处理结果。
进一步地，所述光人工神经网络机器视觉增强芯片的制备方法，还包括：对所述光人工神经网络机器视觉增强芯片的训练过程，具体包括：
利用与所述机器视觉智能处理任务对应的输入训练样本以及输出训练样本，对包含不同的光调制结构、图像传感器和具有不同的全连接参数的处理器的光人工神经网络机器视觉增强芯片进行训练得到满足训练收敛条件的光调制结构、图像传感器和处理器，并将满足训练收敛条件的光调制结构、图像传感器和处理器作为训练好的光调制结构、图像传感器和处理器；
或，利用与所述机器视觉智能处理任务对应的输入训练样本以及输出训练样本，对包含不同的光调制结构、图像传感器和具有不同的全连接参数以及不同的第二次非线性激活参数的处理器的光人工神经网络机器视觉增强芯片进行训练得到满足训练收敛条件的光调制结构、图像传感器和处理器，并将满足训练收敛条件的光调制结构、图像传感器和处理器作为训练好的光调制结构、图像传感器和处理器。
进一步地，在对包含不同的光调制结构、图像传感器和具有不同的全连接参数的处理器的光人工神经网络机器视觉增强芯片进行训练，或，对 包含不同的光调制结构、图像传感器和具有不同的全连接参数以及不同的第二次非线性激活参数的处理器的光人工神经网络机器视觉增强芯片进行训练时，所述不同的光调制结构通过采用计算机光学仿真设计的方式设计实现。
举例来说，对于无人驾驶这一机器视觉系统，需要自动识别车辆前方的障碍物的类型，进而进行准确快速的自动控制。又如，对于质量检测这一机器视觉系统，需要能够准确快速识别出目标对象的质量瑕疵，从而保证质量检测的质量，避免出现漏检或错检。又如，对于手术导航这一机器视觉系统，对于实时性的要求是非常高的，而采用本实施例提供的芯片能够很好地解决实时性这个问题。
可以理解的是，在对芯片上的光滤波器层的结构进行设计时，需要先对相应机器视觉应用场景下的大量目标对象进行采集，通过数据训练得到线性层的权重，即光滤波器层的系统函数，便可以逆向设计出所需的光滤波器层，将其集成在图像传感器上方。
需要说明的是，本实施例提供的基于微纳调制结构和图像传感器的光人工神经网络增强机器视觉芯片有以下效果：A、将人工神经网络部分嵌入硬件芯片中，提升了机器视觉相关应用的实时性和可靠性。B、可以通过CMOS工艺一次流片完成对该芯片的制备，有利于降低器件失效率，提高器件的成品良率，并降低成本。C、在晶圆级别实现单片集成，可以最大程度减小传感器与光滤波器层之间的距离，有利于缩小单元的尺寸，降低器件体积和封装成本。
需要说明的是，关于本实施例提供的光人工神经网络增强机器视觉芯片的详细结构类描述，可参见前述实施例中对于光人工神经网络芯片的介绍，为避免赘述，此处不再介绍。此外，关于光人工神经网络增强机器视觉芯片制备方法的详细介绍，也可以参见前述实施例中对于光人工神经网络芯片制备方法的介绍，此处不再赘述。
最后应说明的是：以上实施例仅用以说明本申请的技术方案，而非对其限制；尽管参照前述实施例对本申请进行了详细的说明，本领域的普通技术人员应当理解：其依然可以对前述各实施例所记载的技术方案进行修改，或者对其中部分技术特征进行等同替换；而这些修改或者替换，并不使相应技术方案的本质脱离本申请各实施例技术方案的精神和范围。

Claims (22)

  1. 一种光人工神经网络智能芯片，其特征在于，包括：光滤波器层、图像传感器和处理器；所述光滤波器层对应人工神经网络的输入层、线性层以及所述输入层到所述线性层的连接权重，所述图像传感器的平方检波响应对应人工神经网络的非线性层中的第一次非线性激活函数；所述处理器对应人工神经网络的全连接以及输出层，或，所述处理器对应人工神经网络的全连接、非线性层中的第二次非线性激活函数以及输出层；
    所述光滤波器层设置于所述图像传感器表面，所述光滤波器层包含有光调制结构，所述光调制结构用于对进入至所述光调制结构不同位置点处的入射光分别进行不同的频谱调制，以在所述图像传感器的表面得到与不同位置点对应的入射光携带信息；所述入射光携带信息包括光强度分布信息、光谱信息、所述入射光的角度信息以及所述入射光的相位信息；
    所述图像传感器通过平方检波响应将与不同位置点经光滤波器层调制后对应的入射光携带信息进行第一次非线性激活处理后转换为与不同位置点对应的电信号，并将与不同位置点对应的电信号发送给所述处理器；
    所述处理器将与不同位置点对应的电信号进行全连接处理，或，所述处理器将与不同位置点对应的电信号进行全连接处理以及第二次非线性激活处理，得到人工神经网络的输出信号。
  2. 根据权利要求1所述的光人工神经网络智能芯片，其特征在于，所述光人工神经网络智能芯片用于目标对象的智能处理任务；所述智能处理任务至少包括智能感知、智能识别和智能决策任务中的一种或多种；
    目标对象的反射光、透射光和/或辐射光进入至训练好的光人工神经网络智能芯片中，得到所述目标对象的智能处理结果；所述智能处理结果至少包括智能感知结果、智能识别结果和/或智能决策结果中的一种或多种；
    其中，训练好的光人工神经网络智能芯片是指包括训练好的光调制结构、图像传感器和处理器的光人工神经网络智能芯片；
    所述训练好的光调制结构、图像传感器和处理器是指利用与所述智能处理任务对应的输入训练样本和输出训练样本，对包含不同的光调制结构、图像传感器和具有不同的全连接参数的处理器的光人工神经网络智能 芯片进行训练得到的满足训练收敛条件的光调制结构、图像传感器和处理器；或，所述训练好的光调制结构、图像传感器和处理器是指利用与所述智能处理任务对应的输入训练样本和输出训练样本，对包含不同的光调制结构、图像传感器和具有不同的全连接参数以及不同的第二次非线性激活参数的处理器的光人工神经网络智能芯片进行训练得到的满足训练收敛条件的光调制结构、图像传感器和处理器。
  3. 根据权利要求2所述的光人工神经网络智能芯片，其特征在于，在对包含不同的光调制结构、图像传感器和具有不同的全连接参数的处理器的光人工神经网络智能芯片进行训练，或，对包含不同的光调制结构、图像传感器和具有不同的全连接参数以及不同的第二次非线性激活参数的处理器的光人工神经网络智能芯片进行训练时，所述不同的光调制结构通过采用计算机光学仿真设计的方式设计实现。
  4. 根据权利要求1～3任一项所述的光人工神经网络智能芯片，其特征在于，所述光滤波器层中的光调制结构包含规则结构和/或不规则结构；和/或，所述光滤波器层中的光调制结构包含离散型结构和/或连续型结构。
  5. 根据权利要求1～3任一项所述的光人工神经网络智能芯片，其特征在于，所述光滤波器层为单层结构或多层结构。
  6. 根据权利要求1～3任一项所述的光人工神经网络智能芯片，其特征在于，所述光滤波器层中的光调制结构包含由多个微纳单元组成的单元阵列，每个微纳单元对应图像传感器上的一个或多个像素点；各个微纳单元的结构相同或不同。
  7. 根据权利要求6所述的光人工神经网络智能芯片，其特征在于，所述微纳单元包含规则结构和/或不规则结构；和/或，所述微纳单元包含离散型结构和/或连续型结构。
  8. 根据权利要求6所述的光人工神经网络智能芯片，其特征在于，所述微纳单元包含有多组微纳结构阵列，各组微纳结构阵列的结构相同或不同。
  9. 根据权利要求8所述的光人工神经网络智能芯片，其特征在于，各组微纳结构阵列具有宽带滤波或窄带滤波的作用。
  10. 根据权利要求8所述的光人工神经网络智能芯片，其特征在于， 各组微纳结构阵列为周期结构阵列或非周期结构阵列。
  11. 根据权利要求8所述的光人工神经网络智能芯片，其特征在于，所述微纳单元包含的多组微纳结构阵列中有一组或多组空结构。
  12. 根据权利要求8所述的光人工神经网络智能芯片，其特征在于，所述微纳单元具有偏振无关特性。
  13. 根据权利要求12所述的光人工神经网络智能芯片，其特征在于，所述微纳单元具有四重旋转对称性。
  14. 根据权利要求1所述的光人工神经网络智能芯片，其特征在于，所述光滤波器层由一层或多层结构构成；
    所述滤波器层是由半导体材料、金属材料、液晶、量子点材料、钙钛矿材料中的一种或多种制备；和/或，所述滤波器层是由光子晶体、超表面、随机结构、纳米结构、金属表面等离激元SPP微纳结构、可调法布里-珀罗谐振腔中的一种或多种制备的滤波器层。
  15. 根据权利要求14所述的光人工神经网络智能芯片，其特征在于，所述半导体材料包括硅、氧化硅、氮化硅、氧化钛、按照预设比例混合的复合材料以及直接带隙化合物半导体材料中的一种或多种；和/或，所述纳米结构包括纳米点二维材料、纳米柱二维材料和纳米线二维材料中的一种或多种。
  16. 根据权利要求1所述的光人工神经网络智能芯片，其特征在于，所述光滤波器层的厚度为0.1λ～10λ，其中λ表示入射光的中心波长。
  17. 根据权利要求1所述的光人工神经网络智能芯片，其特征在于，所述图像传感器为下述中的任意一项或多项：
    CMOS图像传感器CIS、电荷耦合元件CCD、单光子雪崩二极管SPAD阵列和焦平面光电探测器阵列。
  18. 一种智能设备，其特征在于，包括：如权利要求1～17任一项所述的光人工神经网络智能芯片。
  19. 根据权利要求18所述的智能设备，其特征在于，所述智能设备包括智能手机、智能电脑、智能识别设备、智能感知设备和智能决策设备中的一种或多种。
  20. 一种如权利要求1～17任一项所述的光人工神经网络智能芯片的 制备方法，其特征在于，包括：
    在所述图像传感器的表面制备包含有光调制结构的光滤波器层；
    生成具备对信号进行全连接处理功能的处理器或生成具备对信号进行全连接处理以及第二次非线性激活处理功能的处理器；
    连接所述图像传感器和所述处理器；
    其中，所述光滤波器层用于通过所述光调制结构对进入至所述光调制结构不同位置点处的入射光分别进行不同的频谱调制，以在所述图像传感器的表面得到与不同位置点对应的入射光携带信息；所述入射光携带信息包括光强度分布信息、光谱信息、所述入射光的角度信息以及所述入射光的相位信息；
    所述图像传感器通过平方检波响应将与不同位置点经光滤波器层调制后对应的入射光携带信息进行第一次非线性激活处理后转换为与不同位置点对应的电信号，并将与不同位置点对应的电信号发送给所述处理器；
    所述处理器将与不同位置点对应的电信号进行全连接处理，或，所述处理器将与不同位置点对应的电信号进行全连接处理以及第二次非线性激活处理，得到人工神经网络的输出信号。
  21. 根据权利要求20所述的光人工神经网络智能芯片的制备方法，其特征在于，在所述图像传感器的表面制备包含有光调制结构的光滤波器层，包括：
    在所述图像传感器的表面生长一层或多层预设材料；
    对所述一层或多层预设材料进行光调制结构图案的刻蚀，得到包含有光调制结构的光滤波器层；
    或对所述一层或多层预设材料进行压印转移，得到包含有光调制结构的光滤波器层；
    或通过对所述一层或多层预设材料进行外加动态调制，得到包含有光调制结构的光滤波器层；
    或对所述一层或多层预设材料进行分区打印，得到包含有光调制结构的光滤波器层；
    或对所述一层或多层预设材料进行分区生长，得到包含有光调制结构 的光滤波器层；
    或对所述一层或多层预设材料进行量子点转移，得到包含有光调制结构的光滤波器层。
  22. 根据权利要求20所述的光人工神经网络智能芯片的制备方法，其特征在于，当所述光人工神经网络智能芯片用于目标对象的智能处理任务时，利用与所述智能处理任务对应的输入训练样本和输出训练样本，对包含不同的光调制结构、图像传感器和具有不同的全连接参数的处理器的光人工神经网络智能芯片进行训练，得到满足训练收敛条件的光调制结构、图像传感器和处理器；或，对包含不同的光调制结构、图像传感器和具有不同的全连接参数以及不同的第二次非线性激活参数的处理器的光人工神经网络智能芯片进行训练，得到满足训练收敛条件的光调制结构、图像传感器和处理器。
PCT/CN2021/115966 2021-02-08 2021-09-01 光人工神经网络智能芯片及制备方法 WO2022166189A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202110172825.1A CN114912598A (zh) 2021-02-08 2021-02-08 光人工神经网络智能芯片及制备方法
CN202110172825.1 2021-02-08

Publications (1)

Publication Number Publication Date
WO2022166189A1 true WO2022166189A1 (zh) 2022-08-11

Family

ID=82740811

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/115966 WO2022166189A1 (zh) 2021-02-08 2021-09-01 光人工神经网络智能芯片及制备方法

Country Status (2)

Country Link
CN (1) CN114912598A (zh)
WO (1) WO2022166189A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116027608A (zh) * 2023-03-27 2023-04-28 清华大学 全光非线性调制的超表面衍射光计算器件及其制备方法

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116164841B (zh) * 2023-04-26 2023-07-07 中国科学院长春光学精密机械与物理研究所 基于计算增强型像元分光光谱成像芯片的光谱重构方法
CN116242484B (zh) * 2023-05-10 2023-08-01 中国科学院长春光学精密机械与物理研究所 一种光谱成像芯片与光谱重构算法协同设计方法

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170351293A1 (en) * 2016-06-02 2017-12-07 Jacques Johannes Carolan Apparatus and Methods for Optical Neural Network
US20190019100A1 (en) * 2017-07-11 2019-01-17 Charles ROQUES-CARMES Optical Ising Machines and Optical Convolutional Neural Networks
CN111386538A (zh) * 2017-09-20 2020-07-07 视觉动力公司 光子神经网络系统
US20200257958A1 (en) * 2019-02-07 2020-08-13 Samsung Electronics Co., Ltd. Optical device and optical neural network apparatus including the same
CN111832721A (zh) * 2019-04-15 2020-10-27 香港科技大学 全光学神经网络
CN112041857A (zh) * 2018-03-27 2020-12-04 巴伊兰大学 光学神经网络单元以及光学神经网络配置

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170351293A1 (en) * 2016-06-02 2017-12-07 Jacques Johannes Carolan Apparatus and Methods for Optical Neural Network
US20190019100A1 (en) * 2017-07-11 2019-01-17 Charles ROQUES-CARMES Optical Ising Machines and Optical Convolutional Neural Networks
CN111386538A (zh) * 2017-09-20 2020-07-07 视觉动力公司 光子神经网络系统
CN112041857A (zh) * 2018-03-27 2020-12-04 巴伊兰大学 光学神经网络单元以及光学神经网络配置
US20200257958A1 (en) * 2019-02-07 2020-08-13 Samsung Electronics Co., Ltd. Optical device and optical neural network apparatus including the same
CN111832721A (zh) * 2019-04-15 2020-10-27 香港科技大学 全光学神经网络

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
SHEN YICHEN; HARRIS NICHOLAS C.; SKIRLO SCOTT; ENGLUND DIRK; SOLJACIC MARIN: "Deep learning with coherent nanophotonic circuits", 2017 IEEE PHOTONICS SOCIETY SUMMER TOPICAL MEETING SERIES (SUM), 10 July 2017 (2017-07-10), pages 189 - 190, XP033143799, DOI: 10.1109/PHOSST.2017.8012714 *
SUN JIE, ZHAI HONGCHEN, ZHANG YANXIN, LIU YUE, TANG YILIANG: "Integrated Opto-electronic Hybrid Neural Network for Pattern Recognition Controlled by Single Microprocessor", ACTA SCIENTIARUM NATURALIUM UNIVERSITATIS NANKAIENSIS, vol. 33, no. 1, 30 March 2000 (2000-03-30), pages 106 - 109+118, XP055956148 *

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116027608A (zh) * 2023-03-27 2023-04-28 清华大学 全光非线性调制的超表面衍射光计算器件及其制备方法
CN116027608B (zh) * 2023-03-27 2023-07-25 清华大学 全光非线性调制的超表面衍射光计算器件及其制备方法

Also Published As

Publication number Publication date
CN114912598A (zh) 2022-08-16

Similar Documents

Publication Publication Date Title
WO2022166189A1 (zh) 光人工神经网络智能芯片及制备方法
Gao et al. Computational spectrometers enabled by nanophotonics and deep learning
Gruev et al. CCD polarization imaging sensor with aluminum nanowire optical filters
CN111490060A (zh) 光谱成像芯片及光谱识别设备
TWI790639B (zh) 一種圖像採集晶片、物體成像識別設備及物體成像識別方法
WO2022166188A1 (zh) 光人工神经网络智能芯片、智能处理设备及制备方法
Cadusch et al. Visible to long-wave infrared chip-scale spectrometers based on photodetectors with tailored responsivities and multispectral filters
US11620849B2 (en) Spectral imaging chip and apparatus, information processing method, fingerprint living body identification device and fingerprint module
Sun et al. Research on polarization performance of InGaAs focal plane array integrated with superpixel-structured subwavelength grating
CN211828773U (zh) 光谱成像芯片及光谱识别设备
WO2012088117A1 (en) Spectrometer including three-dimensional photonic crystal
Sarwar et al. Miniaturizing a chip-scale spectrometer using local strain engineering and total-variation regularized reconstruction
CN210112118U (zh) 一种图像采集芯片及物体成像识别设备
CN114913553A (zh) 光人工神经网络指纹识别芯片、指纹识别装置及制备方法
Juhl et al. Polarimetry with disordered photonic structures
CN112287594B (zh) 一种用于纳米光子学器件的神经网络自学习系统及其运用
CN209639829U (zh) 一种推扫式偏振光谱成像微系统
Zhou et al. Emerging Computational Micro‐Spectrometers—From Complex System Integration to Simple In Situ Modulation
CN114912601A (zh) 光人工神经网络环保监测芯片及制备方法
Wu et al. Miniaturized Computational Spectrometer
CN114912597A (zh) 农业控制芯片、智能农业控制设备及制备方法
Ahamed et al. Unique Hyperspectral Response Design Enabled by Periodic Surface Textures in Photodiodes
CN118158507A (zh) 图像采集芯片、物体成像识别设备及物体成像识别方法
Kumar et al. Novel design of photonic filter chip for micro-spectrometer
Hu et al. A Computational Microspectrometer Based on Binary-Image-Generated Freeform Metasurfaces for Spectral Sensing

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21924188

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21924188

Country of ref document: EP

Kind code of ref document: A1