WO2022262760A1 - 一种基于微光学器件的端到端光电检测系统及方法 - Google Patents

一种基于微光学器件的端到端光电检测系统及方法 Download PDF

Info

Publication number
WO2022262760A1
WO2022262760A1 PCT/CN2022/098893 CN2022098893W WO2022262760A1 WO 2022262760 A1 WO2022262760 A1 WO 2022262760A1 CN 2022098893 W CN2022098893 W CN 2022098893W WO 2022262760 A1 WO2022262760 A1 WO 2022262760A1
Authority
WO
WIPO (PCT)
Prior art keywords
micro
optical
optical device
image
photoelectric detection
Prior art date
Application number
PCT/CN2022/098893
Other languages
English (en)
French (fr)
Inventor
田宜彬
陈伟
邓弘
Original Assignee
浙江光仑科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 浙江光仑科技有限公司 filed Critical 浙江光仑科技有限公司
Publication of WO2022262760A1 publication Critical patent/WO2022262760A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
    • G06F17/10—Complex mathematical operations
    • G06F17/14—Fourier, Walsh or analogous domain transformations, e.g. Laplace, Hilbert, Karhunen-Loeve, transforms
    • G06F17/141—Discrete Fourier transforms
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks

Definitions

  • the invention relates to the technical field of photoelectric detection, in particular to an end-to-end photoelectric detection system and method based on micro-optical devices.
  • Micro-optical devices can be coded apertures, diffractive optical elements, Fresnel mirrors, micro-lens arrays, and optical homogenizers, etc.
  • Figure 6 and Figure 7 show the difference between traditional lens-based imaging and micro-optics-based imaging.
  • micro-optical devices are simple in structure, small in size and light in weight, which can greatly simplify the structure of visual sensors and have better stability (temperature changes and vibration shocks have less impact on the optical properties of the device).
  • a point on the target object corresponds to a point on the vision sensor (light is refracted and focused); while in an ideal micro-optical imaging vision sensor, the point on the target object A point corresponds to several points on the visual sensor (light is scattered or diffracted), so the original image captured by the micro-optical imaging visual sensor is sometimes unable to recognize its specific content.
  • the optical transfer function (also known as the point spread function) of the lens imaging vision sensor is approximated by a Delta function of one-to-one mapping:
  • the optical transfer function of the micro-optical imaging vision sensor is a complex function of one-to-many mapping:
  • optical transfer function of an actual imaging system is usually not a discrete function expressed by formulas 1 and 2, so the above two formulas are approximate descriptions.
  • the image acquired on the micro-optical imaging vision sensor can be expressed as:
  • z1 and z2 are the depth boundaries of the target object
  • C is a system constant
  • w(x, y, z) is the surface feature (which can include texture/brightness and depth) on the target object at (x, y, z) position
  • g(x, y, z) is the optical transfer function in formula (2)
  • n(u, v) is the imaging noise at the visual sensor pixel (u, v)
  • i(u, v) is the visual sensor pixel ( The value obtained at u, v).
  • the application scenario needs to provide the image acquired by the vision sensor to the human eye or use a traditional image-based processing method, we can reconstruct the image acquired by the micro-optical vision sensor.
  • the reconstructed image can be further image analyzed and processed by traditional machine vision algorithms to obtain the desired results.
  • micro-optical devices to perform specific processing on the optical information reflected or emitted by the target object, and directly analyze the information required for image extraction obtained by the micro-optical vision sensor, without image reconstruction, and to achieve image analysis directly extracted from the sensor Results of an end-to-end photodetection system.
  • the present invention aims to overcome the deficiency in the prior art that image analysis results need to be obtained through image reconstruction, and provides an end-to-end photoelectric detection based on micro-optical devices that can directly extract image analysis results from visual sensors without image reconstruction systems and methods.
  • An end-to-end photoelectric detection method based on a micro-optical device comprising the following steps:
  • Step 1 the target object emits or reflects optical information
  • Step 2 the micro-optical device performs preliminary signal processing on the optical information
  • Step 3 the visual sensor collects the image preliminarily processed by the micro-optical device, and directly analyzes to obtain the required photoelectric detection result.
  • a point on the target object corresponds to a point on the vision sensor (light is refracted and focused); while in an ideal micro-optical imaging vision sensor, a point on the target object corresponds to a point on the vision sensor Several points on the sensor (light is scattered and diffracted), so the original image captured by the micro-optical vision sensor cannot be recognized by the human eye.
  • the application scenario needs to provide the image acquired by the vision sensor to the human eye or use a traditional image-based processing method, the image acquired by the micro-optical vision sensor needs to be reconstructed.
  • the workflow of image reconstruction on the data of the micro-optical vision sensor and then using the traditional image and video processing method is shown in Figure 8.
  • the main disadvantage of this two-step independent data processing method is that image reconstruction and downstream image and video processing require a lot of calculations. If the reconstructed image does not need to be processed and judged by human eyes, some of the calculations are unnecessary.
  • the present invention adopts end-to-end processing method, specifically, utilizes micro-optical device to carry out certain preliminary signal processing to the optical information (initial end) that target object emits or reflects, and the information collected by visual sensor passes through micro-optics
  • the image processed by the device is directly analyzed to obtain the required detection result (end end), and there is no image reconstruction link between the start end and the end end, so that the visual sensor has intelligent analysis and processing capabilities, and at the same time reduces the photoelectric
  • the amount of calculation required for detection achieves the purpose of directly extracting image analysis results from the vision sensor without image reconstruction.
  • the micro-optic device adopts the following design:
  • Step a first use traditional methods to extract image features, and then confirm and screen through experts to obtain an ideal image feature set
  • Step b given an optical simulation image
  • Step c micro-optical device signal processing simulation simulation image
  • Step d then image preprocessing is performed on the simulated image by the visual sensor, and the image feature set of the micro-optical device is obtained by the LBP local binary mode;
  • step e the distance between the image feature set of the micro-optics device and the ideal image feature set is calculated to obtain the objective function:
  • Obj(Feature set) ⁇ *GeometricDistance+ ⁇ *Brightness Distance, where Geometric Distance is the XYZ geometric distance between the feature center of the micro-optical device image feature set and the ideal image feature set, and Brightness Distance is the value of the feature center pixel.
  • the micro-optical device that realizes the above-mentioned end-to-end photodetection can be realized through the most basic inverse Fourier transform (IFFT, Inverse Fast Fourier Transform).
  • IFFT Inverse Fast Fourier Transform
  • the advantage of this design method is that it is simple to implement, but the optical signal processing capability of the obtained micro-optical device is relatively limited, and the obtained micro-optical device essentially performs Fourier transform on the optical signal emitted or reflected by the target object. Therefore, the present invention adopts the closed-loop design method in FIG. 2 to design the micro-optical device, so that the micro-optic device can realize more complex optical signal processing, such as specific feature extraction. Therefore, through the combination of optical simulation and manual or expert sample labeling, the objective function of the weighted combination of multiple indicators is realized, and by optimizing such an objective function, the designed micro-optical device has the required optical signal processing capability.
  • IFFT Inverse Fast Fourier Transform
  • the traditional method for extracting image features is corner detection of two-dimensional images, edge detection, LBP local binary mode, MSER maximum stable external region, corner detection of three-dimensional images and line detection. one. Since the output of these feature extraction methods is heavily dependent on parameters, the ideal image feature set can be obtained by confirming and screening the output manually or by experts.
  • the visual sensor is provided with a neural network input layer, a preprocessing subnetwork, a detection and classification subnetwork, and a neural network output layer, and the neural network input layer passes through the preprocessing subnetwork, detection and classification subnetwork and Neural network output layer electrical connections.
  • the photoelectric detection results are directly extracted from the images of the micro-optic vision sensor through a multi-layer neural network.
  • step A the visual sensor collects the original image
  • Step B the collected original image is input through the neural network input layer, then processed through the preprocessing sub-network, and then detected and analyzed through the detection and classification sub-network;
  • step C the photoelectric detection and analysis results are output through the neural network output layer.
  • the original image acquired on the visual sensor has been processed by the specific optical information of the micro-optical device.
  • the photoelectric detection result can be directly extracted from the visual sensor image through a multi-layer neural network.
  • the present invention also provides an end-to-end photoelectric detection system based on a micro-optical device, which includes a PCB substrate, a support tube and a micro-optic device, the PCB substrate is provided with a computer system on chip and a visual sensor, and the visual sensor passes through
  • the PCB substrate is electrically connected to the computer on-chip system, one end of the support tube is fixedly connected to the PCB substrate, the micro-optical device is installed on the other end of the support tube, the visual sensor is located in the support tube, and the top of the micro-optic device is located Outside the support tube, the bottom of the micro-optical device is located in the support tube and is provided with a filter layer, and the filter layer is fixedly connected with the micro-optic device.
  • the micro-optical device performs a certain preliminary signal processing on the optical information (starting end) emitted or reflected by the target object, and selects or shields specific wavelengths through the filter layer to extract specific image features.
  • the image processed by the device is directly analyzed under the control of the computer on-chip system to obtain the required detection results (end end).
  • the purpose of the visual sensor to directly extract the image analysis results.
  • an optical protection screen is provided on the top of the micro-optical device, and the optical protection screen is fixedly connected with the micro-optic device.
  • Optical protective screens help protect optics from damage.
  • the micro-optical device includes a glass substrate, one side of the glass substrate is provided with a micro-optic layer, the glass substrate is fixedly connected to the optical protection screen through the micro-optic layer, and the filter layer corresponds to the glass substrate The other side is fixedly connected.
  • the optical protective screen is beneficial to protect the micro-optic layer on the surface of the glass substrate from damage; the micro-optic layer adopts a micro-optical structure to facilitate a certain preliminary signal processing of the optical information emitted or reflected by the target object.
  • one side of the micro-optic layer is provided with an AR coating
  • the micro-optic layer is fixedly connected to the optical protection screen through the AR coating
  • the other side corresponding to the micro-optic layer is fixedly connected to the glass substrate.
  • AR coatings are beneficial for micro-optics surfaces with lower reflectance.
  • the beneficial effects of the present invention are: to enable the visual sensor to have intelligent analysis and processing capabilities, reduce the calculation amount of photoelectric detection, and achieve the purpose of directly extracting the image analysis results from the visual sensor without image reconstruction; realize the above-mentioned specific optical
  • Fig. 1 is the flowchart of photoelectric detection of micro-optical device in the present invention
  • Fig. 2 is the flow chart of the closed-loop design of the micro-optical device
  • Figure 3 is a flow chart of the end-to-end image analysis and processing of the visual sensor
  • Fig. 4 is a system architecture diagram of the photoelectric detection of the micro-optical device
  • Fig. 5 is a structural schematic diagram of a micro-optical device
  • Figure 6 is a system architecture diagram based on lens imaging
  • Figure 7 is a system architecture diagram based on micro-optical device imaging
  • Fig. 8 is a flow chart of performing image reconstruction on the data of the visual sensor first and then using the traditional image and video processing method.
  • PCB substrate 1. Support tube, 3. Micro optics device, 4. Computer on-chip system, 5. Vision sensor, 6. Filter layer, 7. Optical protection screen, 8. Glass substrate, 9. Micro Optical layer, 10. AR coating, 11. Lens, 12. Neural network input layer, 13. Preprocessing subnetwork, 14. Detection and classification subnetwork, 15. Neural network output layer, 16. Target object.
  • a kind of end-to-end photoelectric detection method based on micro-optical device comprises the following steps:
  • Step 1 the target object 16 emits or reflects optical information
  • Step 2 the micro-optical device 3 performs preliminary signal processing on the optical information
  • Step 3 the visual sensor 5 collects the image preliminarily processed by the micro-optical device 3, and directly analyzes to obtain the required photoelectric detection result.
  • the micro-optical device 3 adopts the following design:
  • Step a first use traditional methods to extract image features, and then confirm and screen through experts to obtain an ideal image feature set
  • Step b given an optical simulation image
  • Step c the signal processing of the micro-optical device 3 simulates the simulated image
  • Step d then perform image preprocessing on the simulated image through the visual sensor 5, and obtain the image feature set of the micro-optical device 3 through the LBP local binary mode;
  • step e the distance between the image feature set of the micro-optical device 3 and the ideal image feature set is calculated to obtain the objective function:
  • Obj(Feature set) ⁇ *GeometricDistance+ ⁇ *Brightness Distance, wherein Geometric Distance is the XYZ geometric distance between the micro-optical device 3 image feature set and the feature center in the ideal image feature set, and Brightness Distance is the value of the feature center pixel.
  • step a the traditional method for extracting image features is one of two-dimensional image corner detection, edge detection, LBP local binary mode, MSER maximum stable outer region, three-dimensional image corner detection and line detection.
  • the visual sensor 5 is provided with a neural network input layer 12, a preprocessing subnetwork 13, a detection and classification subnetwork 14, a neural network output layer 15, and the neural network input layer 12 passes through the preprocessing subnetwork 13,
  • the detection and classification sub-network 14 is electrically connected to the neural network output layer 15 .
  • Step A the visual sensor 5 collects the original image
  • Step B the collected original image is input through the neural network input layer 12, then processed through the preprocessing sub-network 13, and then detected and analyzed through the detection and classification sub-network 14;
  • step C the photoelectric detection and analysis results are output through the neural network output layer 15 .
  • the present invention also provides an end-to-end photoelectric detection system based on micro-optical devices, including a PCB substrate 1, a support tube 2 and a micro-optical device 3, and the PCB substrate 1 is provided with a computer on-chip system 4 and A visual sensor 5, the visual sensor 5 is electrically connected to the computer system on chip 4 through the PCB substrate 1, one end of the support tube 2 is fixedly connected to the PCB substrate 1, the micro-optical device 3 is installed on the other end of the support tube 2, and the visual sensor 5 is located in the support tube 2, the top of the micro-optical device 3 is located outside the support tube 2, the bottom of the micro-optic device 3 is located in the support tube 2 and is provided with a filter layer 6, and the filter layer 6 is fixedly connected to the micro-optic device 3.
  • the top of the micro-optical device 3 is provided with an optical protection screen 7 , and the optical protection screen 7 is fixedly connected with the micro-optic device 3 .
  • the micro-optic device 3 includes a glass substrate 8, one side of the glass substrate 8 is provided with a micro-optic layer 9, the glass substrate 8 is fixedly connected with the optical protective screen 7 through the micro-optic layer 9, and the filter layer 6 is fixedly connected to the other side corresponding to the glass substrate 8 .
  • one side of the micro-optic layer 9 is provided with an AR coating 10
  • the micro-optic layer 9 is fixedly connected with the optical protective screen 7 through the AR coating 10
  • the other side corresponding to the micro-optic layer 9 is connected to the glass.
  • the substrate 8 is fixedly connected.
  • a point on the target object corresponds to a point on the visual sensor 5 (light is refracted and focused); and in an ideal micro-optical imaging visual sensor 5, a point on the target object Corresponding to several points on the visual sensor 5 (the light is scattered and diffracted), so the original image acquired by the micro-optic visual sensor 5 cannot be recognized by human eyes.
  • the application scenario needs to provide the image acquired by the visual sensor 5 to human eyes or use a traditional image-based processing method, the image acquired by the micro-optic visual sensor 5 needs to be reconstructed.
  • the workflow of performing image reconstruction on the data of the micro-optical vision sensor 5 and then using the traditional image and video processing method is shown in FIG. 8 .
  • the main disadvantage of this two-step independent data processing method is that image reconstruction and downstream image and video processing require a lot of calculations. If the reconstructed image does not need to be processed and judged by human eyes, some of the calculations are unnecessary.
  • the present invention adopts end-to-end processing method, specifically, utilizes micro-optical device 3 to carry out certain preliminary signal processing to the optical information (starting end) that target object sends or reflects, and the process that visual sensor 5 gathers
  • the image processed by the micro-optical device 3 is directly analyzed to obtain the required detection result (end end), and there is no longer an image reconstruction link between the start end and the end end, so that the visual sensor 5 has intelligent analysis and processing capabilities,
  • the calculation amount of the photoelectric detection is reduced, and the purpose of directly extracting the image analysis result from the visual sensor 5 is achieved without image reconstruction.
  • the micro-optical device 3 that realizes the above-mentioned end-to-end photoelectric detection can be realized through the most basic inverse Fourier transform (IFFT, Inverse Fast Fourier Transform).
  • IFFT Inverse Fast Fourier Transform
  • the advantage of this design method is that it is simple to implement, but the optical signal processing capability of the obtained micro-optical device 3 is relatively limited, and the obtained micro-optical device 3 essentially performs Fourier transformation on the optical signal emitted or reflected by the target object. transform. Therefore, the present invention adopts the design method shown in FIG. 2 to design the micro-optical device 3, so that the micro-optic device 3 can realize more complex optical signal processing, such as specific feature extraction. Therefore, through the combination of optical simulation and manual or expert sample labeling, the objective function of the weighted combination of multiple indicators is realized, and by optimizing such an objective function, the designed micro-optical device 3 has the required optical signal processing capability.
  • IFFT Inverse Fast Fourier Transform
  • the original image acquired on the visual sensor 5 has been processed by the specific optical information of the micro-optical device 3, and such an image is further input through the neural network input layer 12, and then processed through the preprocessing sub-network 13, and then Detection and analysis are performed through the detection and classification sub-network 14 , and finally the photoelectric detection analysis result is output through the neural network output layer 15 , which realizes the direct extraction of the photoelectric detection result from the image of the visual sensor 5 .

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Software Systems (AREA)
  • Computational Mathematics (AREA)
  • Mathematical Optimization (AREA)
  • Pure & Applied Mathematics (AREA)
  • Mathematical Analysis (AREA)
  • General Engineering & Computer Science (AREA)
  • Artificial Intelligence (AREA)
  • Biophysics (AREA)
  • Discrete Mathematics (AREA)
  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Algebra (AREA)
  • Biomedical Technology (AREA)
  • Databases & Information Systems (AREA)
  • Computational Linguistics (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • Length Measuring Devices By Optical Means (AREA)
  • Image Input (AREA)

Abstract

本发明公开了一种基于微光学器件的端到端光电检测系统及方法，旨在提供一种无需通过图像重建便能从视觉传感器直接提取图像分析结果的基于微光学器件的端到端光电检测系统及方法。它包括以下步骤：步骤一，目标物体发出或反射出光学信息；步骤二，微光学器件对光学信息进行初步的信号处理；步骤三，视觉传感器采集经微光学器件初步处理的图像，并直接分析得到所需要的光电检测结果。本发明的有益效果是：使视觉传感器具备智能化的分析处理能力，同时减小光电检测的计算量，达到了无需通过图像重建便能从视觉传感器直接提取图像分析结果的目的。

Description

一种基于微光学器件的端到端光电检测系统及方法 技术领域
本发明涉及光电检测相关技术领域，尤其是指一种基于微光学器件的端到端光电检测系统及方法。
背景技术
随着半导体处理器的计算能力的持续提高，近几年人们提出了各种计算成像方法来简化视觉传感器的结构和增加复杂的成像功能。其中一个方法是利用单个微光学器件(Micro Optics Element,MOE)来替换传统的由多个镜片组成的镜头，并结合相应的计算方法来重建2D或3D图像。微光学器件可以是编码光圈、衍射光学元件、菲涅尔镜、微透镜阵列和光学匀光片等。
如图6和图7展示了传统的基于镜头的成像和基于微光学器件的成像的区别。宏观上，微光学器件结构上简单，体积小、重量轻，可以大大简化视觉传感器的结构，并具有更好的稳定性(温度变化和振动冲击对器件的光学特性的影响更小)。需要指出的是，在理想的镜头成像视觉传感器里，目标物体上的一个点对应到视觉传感器上的一个点(光被折射聚焦)；而在理想的微光学成像视觉传感器里，目标物体上的一个点对应到视觉传感器上的若干个点(光被散射或是衍射)，因此微光学成像视觉传感器获取的原始图像人眼有时无法识别其具体内容。
用简化的模型来表述，镜头成像视觉传感器的光学传递函数(也称为点扩散函数)近似于一个一对一映射的Delta函数：
Figure PCTCN2022098893-appb-000001
而微光学成像视觉传感器的光学传递函数是一个一对多映射的复杂函数：
Figure PCTCN2022098893-appb-000002
实际成像系统光学传递函数通常不是公式1和2表述的离散函数，因此以上两个公式是近似的描述。
微光学成像视觉传感器上获取的图像可以表达为：
Figure PCTCN2022098893-appb-000003
其中z1和z2为目标物体的深度边界，C为系统常数，w(x，y，z)是目标物体上在(x，y，z)位置的表面特征(可以包括纹理/亮度和深度)，g(x，y，z)是公式(2)中的光学传递函数，n(u，v)为视觉传感器像素(u，v)处的成像噪声，i(u，v)为视觉传感器像素(u，v)处获取的数值。
如果应用场景需要把视觉传感器获取的图像提供给人眼看或是使用基于传统图像的处理方法，我们可以将微光学视觉传感器获取的图像进行重建。重建的图像可以经过传统的机器视觉算法进行进一步图像分析和处理后获得所需的结果。
在物联网和智能制造的应用中，大部分时候视觉传感器获取的图像是直接给机器看(即由计算机进行图像的分析并决策)而不是给人看。因此我们也可以利用微光学器件对目标物体反射或发出的光学信息进行特定处理，并直接分析微光学视觉传感器获取的图像提取所需的信息，不再进行图像重建，实现从传感器直接提取图像分析结果的端到端光电检测系统。
发明内容
本发明是为了克服现有技术中需要通过图像重建才能获取图像分析结果的不足，提供了一种无需通过图像重建便能从视觉传感器直接提取图像分析结果的基于微光学器件的端到端光电检测系统及方法。
为了实现上述目的，本发明采用以下技术方案：
一种基于微光学器件的端到端光电检测方法，包括以下步骤：
步骤一，目标物体发出或反射出光学信息；
步骤二，微光学器件对光学信息进行初步的信号处理；
步骤三，视觉传感器采集经微光学器件初步处理的图像，并直接分析得到所需要的光电检测结果。
在理想的镜头成像视觉传感器里，目标物体上的一个点对应到视觉传感器上的一个点(光被折射聚焦)；而在理想的微光学成像视觉传感器里，目标物体上的一个点对应到视觉传感器上的若干个点(光被散射和衍射)，因此微光学视觉传感器获取的原始图像人眼无法识别其具体内容。如上所述，如果应用场景需要把视觉传感器获取的图像提供给人眼看或是使用基于传统图像的处理方法，需要将微光学视觉传感器获取的图像进行重建。先对微光学视觉传感器的数据进行图像重建再使用传统的图像视频处理方法的工作流程如图8所示。这样分两步独立进行数据处理方式的主要缺点是图像重建和下游图像和视频处理计算量大，如果重建的图像不需要经过人眼进行处理和判断，其中部分运算是不必要的。
如图1所示，本发明采用端到端的处理方法，具体的说，利用微光学器件对目标物体发出或反射的光学信息(起始端)进行一定的初步信号处理，视觉传感器采集的经过微光学器件处理的图像进行直接分析得到所需要的检测结果(结束端)，在起始端和结束端之间不再有图像重建这一环节，使视觉传感器具备智能化的分析处理能力，同时减小光电检测的计算量，达到了无需通过图像重建便能从视觉传感器直接提取图像分析结果的目的。
作为优选，在步骤二中，为了让微光学器件实现更复杂的光学信号处理，微光学器件采用了如下设计：
步骤a，先使用传统方法提取图像特征，然后通过专家进行确认和筛选以得到理想的图像特征集；
步骤b，给定光学仿真模拟图像；
步骤c，微光学器件信号处理仿真模拟图像；
步骤d，然后通过视觉传感器对仿真模拟图像进行图像预处理，并通过LBP局部二值模式得到微光学器件图像特征集；
步骤e，微光学器件图像特征集与理想图像特征集之间的距离计算得到目标函数：
Obj(Feature set)＝α*GeometricDistance+β*Brightness Distance，其中Geometric Distance为微光学器件图像特征集与理想图像特征集里特征中心的XYZ几何距离，Brightness Distance为特征中心像素的值。
实现上述端到端光电检测的微光学器件可以通过最基本的反向傅里叶变换(IFFT,Inverse Fast Fourier Transform)实现。这一设计方法的优点是实现简单，但是所得到的微光学器件的光学信号处理能力相对有限，这样得到的微光学器件本质上是对目标物体发射或反射的光学信号进行了傅里叶变换。故本发明采用图2的闭环设计方法进行微光学器件的设计，使得微光学器件能够实现更复杂的光学信号处理，比如特定的特征提取。故通过光学仿真模拟与人工或专家样本标注结合，实现多种指标加权结合的目标函数，并通过优化这样的目标函数使得设计的微光学器件具备所需的光学信号处理能力。
作为优选，在步骤a中，提取图像特征的传统方法是二维图像的角点检测、边缘检测、LBP局部二值模式、MSER最大稳定外部区域、三维图像的角点检测和直线检测中的其中之一。由于这些特征提取方法的输出严重依赖于参数，对其输出采用人工或专家进行确认和筛选能够得到理想的图像特征集。
作为优选，所述视觉传感器内设有神经网络输入层、预处理子网络、检测和分类子网络、神经网络输出层，所述神经网络输入层依次通过预处理子网络、检测和分类子网络与神经网络输出层电连接。通过多层神经网络实现从微光学视觉传感器图像直接提取光电检测结果。
作为优选，所述视觉传感器分析获取所需光电检测结果的具体流程如下：步骤A，视觉传感器采集原始图像；
步骤B，采集的原始图像通过神经网络输入层输入，再通过预处理子网络进行处理，然后通过检测和分类子网络进行检测分析；
步骤C，光电检测分析结果通过神经网络输出层输出。
视觉传感器上获取的原始图像已经经过微光学器件的特定光学信息处理，对这样的图像可以进一步通过多层神经网络实现从视觉传感器图像直接提取光电检测结果。
本发明还提供了一种基于微光学器件的端到端光电检测系统，它包括PCB基板、支撑管和微光学器件，所述PCB基板上设有计算机片上系统和视觉传感器，所述视觉传感器通过PCB基板与计算机片上系统电连接，所述支撑管的一端与PCB基板固定连接，所述微光学器件安装于支撑管的另一端，所述视觉传感器位于支撑管内，所述微光学器件的顶部位于支撑管外，所述微光学器件的底部位于支撑管内且设有滤光层，所述滤光层与微光学器件固定连接。微光学器件对目标物体发出或反射的光学信息(起始端)进行一定的初步信号处理，并通过滤光层对特定波长进行选通或屏蔽，提取特定的图像特征，视觉传感器采集的经过微光学器件处理的图像在计算机片上系统的控制下直接分析得到所需要的检测结果(结束端)，在起始端和结束端之间不再有图像重建这一环节，达到了无需通过图像重建便能从视觉传感器直接提取图像分析结果的目的。
作为优选，所述微光学器件的顶部设有光学保护屏，所述光学保护屏与微光学器件固定连接。光学保护屏有利于保护光学器件不受损。
作为优选，所述微光学器件包括玻璃基板，所述玻璃基板的一侧设有微光学层，所述玻璃基板通过微光学层与光学保护屏固定连接，所述滤光层与玻璃基板相对应的另一侧固定连接。更具体地，光学保护屏有利于保护位于玻璃基板表面的微光学层不受损伤；微光学层采用微光学结构便于对目标物体发出或反射的光学信息进行一定的初步信号处理。
作为优选，所述微光学层的一侧设有AR镀膜，所述微光学层通过AR镀膜与光学保护屏固定连接，所述微光学层相对应的另一侧与玻璃基板固定连接。AR镀膜有利于微光学器件表面具有较低的反射比。
本发明的有益效果是：使视觉传感器具备智能化的分析处理能力，同时减小光电检测的计算量，达到了无需通过图像重建便能从视觉传感器直接提取图像分析结果的目的；实现上述特定光学信号处理的微光学器件的设计方法，通过光学仿真模拟与人工或专家样本标注结合，实现多种指标加权结合的目标函数，并通过优化这样的目标函数使得设计的微光学器件具备所需的光学信号处理能力；通过多层神经网络实现从视觉传感器图像直接提取光电检测结果；有利于保护位于玻璃基板表面的微光学层不受损伤；有利于微光学器件表面具有较低的反射比。
附图说明
图1是本发明中微光学器件光电检测的流程图；
图2是微光学器件的闭环设计流程图；
图3是视觉传感器端到端图像分析和处理流程图；
图4是微光学器件光电检测的系统架构图；
图5是微光学器件的结构示意图；
图6是基于镜头成像的系统架构图；
图7是基于微光学器件成像的系统架构图；
图8是先对视觉传感器的数据进行图像重建再使用传统的图像视频处理方法的流程图。
图中：1.PCB基板，2.支撑管，3.微光学器件，4.计算机片上系统，5.视觉传感器，6.滤光层，7.光学保护屏，8.玻璃基板，9.微光学层，10.AR镀膜，11.镜头，12.神经网络输入层，13.预处理子网络，14.检测和分类子网络，15.神经网络输出层，16.目标物体。
具体实施方式
下面结合附图和具体实施方式对本发明做进一步的描述。
如图1所述的实施例中，一种基于微光学器件的端到端光电检测方法，包括以下步骤：
步骤一，目标物体16发出或反射出光学信息；
步骤二，微光学器件3对光学信息进行初步的信号处理；
步骤三，视觉传感器5采集经微光学器件3初步处理的图像，并直接分析得到所需要的光电检测结果。
如图2所示，在步骤二中，为了让微光学器件3实现更复杂的光学信号处理，微光学器件3采用了如下设计：
步骤a，先使用传统方法提取图像特征，然后通过专家进行确认和筛选以得到理想的图像特征集；
步骤b，给定光学仿真模拟图像；
步骤c，微光学器件3信号处理仿真模拟图像；
步骤d，然后通过视觉传感器5对仿真模拟图像进行图像预处理，并通过LBP局部二值模式得到微光学器件3图像特征集；
步骤e，微光学器件3图像特征集与理想图像特征集之间的距离计算得到目标函数：
Obj(Feature set)＝α*GeometricDistance+β*Brightness Distance，其中Geometric Distance为微光学器件3图像特征集与理想图像特征集里特征中心的XYZ几何距离，Brightness Distance为特征中心像素的值。
在步骤a中，提取图像特征的传统方法是二维图像的角点检测、边缘检测、LBP局部二值模式、MSER最大稳定外部区域、三维图像的角点检测和直线检测中的其中之一。
如图3所示，视觉传感器5内设有神经网络输入层12、预处理子网络13、检测和分类子网络14、神经网络输出层15，神经网络输入层12依次通过预处理子网络13、检测和分类子网络14与神经网络输出层15电连接。
如图3所示，视觉传感器5分析获取所需光电检测结果的具体流程如下
步骤A，视觉传感器5采集原始图像；
步骤B，采集的原始图像通过神经网络输入层12输入，再通过预处理子网络13进行处理，然后通过检测和分类子网络14进行检测分析；
步骤C，光电检测分析结果通过神经网络输出层15输出。
如图4所示，本发明还提供了一种基于微光学器件的端到端光电检测系统，包括PCB基板1、支撑管2和微光学器件3，PCB基板1上设有计算机片上系统4和视觉传感器5，视觉传感器5通过PCB基板1与计算机片上系统4电连接，支撑管2的一端与PCB基板1固定连接，微光学器件3安装于支撑管2的另一端，视觉传感器5位于支撑管2内，微光学器件3的顶部位于支撑管2外，微光学器件3的底部位于支撑管2内且设有滤光层6，滤光层6与微光学器件3固定连接。微光学器件3的顶部设有光学保护屏7，光学保护屏7与微光学器件3固定连接。
如图4和图5所示，微光学器件3包括玻璃基板8，玻璃基板8的一侧设有微光学层9，玻璃基板8通过微光学层9与光学保护屏7固定连接，滤光 层6与玻璃基板8相对应的另一侧固定连接。
如图4和图5所示，微光学层9的一侧设有AR镀膜10，微光学层9通过AR镀膜10与光学保护屏7固定连接，微光学层9相对应的另一侧与玻璃基板8固定连接。
在理想的镜头成像视觉传感器5里，目标物体上的一个点对应到视觉传感器5上的一个点(光被折射聚焦)；而在理想的微光学成像视觉传感器5里，目标物体上的一个点对应到视觉传感器5上的若干个点(光被散射和衍射)，因此微光学视觉传感器5获取的原始图像人眼无法识别其具体内容。如上所述，如果应用场景需要把视觉传感器5获取的图像提供给人眼看或是使用基于传统图像的处理方法，需要将微光学视觉传感器5获取的图像进行重建。先对微光学视觉传感器5的数据进行图像重建再使用传统的图像视频处理方法的工作流程如图8所示。这样分两步独立进行数据处理方式的主要缺点是图像重建和下游图像和视频处理计算量大，如果重建的图像不需要经过人眼进行处理和判断，其中部分运算是不必要的。
如图1所示，本发明采用端到端的处理方法，具体的说，利用微光学器件3对目标物体发出或反射的光学信息(起始端)进行一定的初步信号处理，视觉传感器5采集的经过微光学器件3处理的图像进行直接分析得到所需要的检测结果(结束端)，在起始端和结束端之间不再有图像重建这一环节，使视觉传感器5具备智能化的分析处理能力，同时减小光电检测的计算量，达到了无需通过图像重建便能从视觉传感器5直接提取图像分析结果的目的。
实现上述端到端光电检测的微光学器件3可以通过最基本的反向傅里叶变换(IFFT,Inverse Fast Fourier Transform)实现。这一设计方法的优点是实现简单，但是所得到的微光学器件3的光学信号处理能力相对有限，这样得到的微光学器件3本质上是对目标物体发射或反射的光学信号进行了傅里叶变换。故本发明采用如图2所示的设计方法进行微光学器件3的设计，使得微光学器件3能够实现更复杂的光学信号处理，比如特定的特征提取。故通过光学仿真模拟与人工或专家样本标注结合，实现多种指标加权结合的目标函数，并通过优化这样的目标函数使得设计的微光学器件3具备所需的光学信号处理能力。
如图3所示，视觉传感器5上获取的原始图像已经经过微光学器件3的特定光学信息处理，对这样的图像进一步通过神经网络输入层12输入，再通过预处理子网络13进行处理，然后通过检测和分类子网络14进行检测分析，最后光电检测分析结果通过神经网络输出层15输出，实现了从视觉传感器5图像直接提取光电检测结果。

Claims (9)

  1. 一种基于微光学器件的端到端光电检测方法，其特征是，
    包括以下步骤：
    步骤一，目标物体(16)发出或反射出光学信息；
    步骤二，微光学器件(3)对光学信息进行初步的信号处理；
    步骤三，视觉传感器(5)采集经微光学器件(3)初步处理的图像，并直接分析得到所需要的光电检测结果。
  2. 根据权利要求1所述的一种基于微光学器件的端到端光电检测方法，其特征是，在步骤二中，为了让微光学器件(3)实现更复杂的光学信号处理，微光学器件(3)采用了如下设计：
    步骤a，先使用传统方法提取图像特征，然后通过专家进行确认和筛选以得到理想的图像特征集；
    步骤b，给定光学仿真模拟图像；
    步骤c，微光学器件(3)信号处理仿真模拟图像；
    步骤d，然后通过视觉传感器(5)对仿真模拟图像进行图像预处理，并通过LBP局部二值模式得到微光学器件(3)图像特征集；
    步骤e，微光学器件(3)图像特征集与理想图像特征集之间的距离计算得到目标函数：Obj(Feature set)＝α*GeometricDistance+β*BrightnessDistance，
    其中GeometricDistance为微光学器件(3)图像特征集与理想图像特征集里特征中心的XYZ几何距离，Brightness Distance为特征中心像素的值。
  3. 根据权利要求2所述的一种基于微光学器件的端到端光电检测方法，其特征是，在步骤a中，提取图像特征的传统方法是二维图像的角点检测、边缘检测、LBP局部二值模式、MSER最大稳定外部区域、三维图像的角点检测和直线检测中的其中之一。
  4. 根据权利要求2所述的一种基于微光学器件的端到端光电检测方法，其特征是，所述视觉传感器(5)内设有神经网络输入层(12)、预处理子网络(13)、 检测和分类子网络(14)、神经网络输出层(15)，所述神经网络输入层(12)依次通过预处理子网络(13)、检测和分类子网络(14)与神经网络输出层(15)电连接。
  5. 根据权利要求4所述的一种基于微光学器件的端到端光电检测方法，其特征是，所述视觉传感器(5)分析获取所需光电检测结果的具体流程如下：
    步骤A，视觉传感器(5)采集原始图像；
    步骤B，采集的原始图像通过神经网络输入层(12)输入，再通过预处理子网络(13)进行处理，然后通过检测和分类子网络(14)进行检测分析；
    步骤C，光电检测分析结果通过神经网络输出层(15)输出。
  6. 一种基于微光学器件的端到端光电检测系统，其特征是，包括PCB基板(1)、支撑管(2)和微光学器件(3)，所述PCB基板(1)上设有计算机片上系统(4)和视觉传感器(5)，所述视觉传感器(5)通过PCB基板(1)与计算机片上系统(4)电连接，所述支撑管(2)的一端与PCB基板(1)固定连接，所述微光学器件(3)安装于支撑管(2)的另一端，所述视觉传感器(5)位于支撑管(2)内，所述微光学器件(3)的顶部位于支撑管(2)外，所述微光学器件(3)的底部位于支撑管(2)内且设有滤光层(6)，所述滤光层(6)与微光学器件(3)固定连接。
  7. 根据权利要求6所述的一种基于微光学器件的端到端光电检测系统，其特征是，所述微光学器件(3)的顶部设有光学保护屏(7)，所述光学保护屏(7)与微光学器件(3)固定连接。
  8. 根据权利要求7所述的一种基于微光学器件的端到端光电检测系统，其特征是，所述微光学器件(3)包括玻璃基板(8)，所述玻璃基板(8)的一侧设有微光学层(9)，所述玻璃基板(8)通过微光学层(9)与光学保护屏(7)固定连接，所述滤光层(6)与玻璃基板(8)相对应的另一侧固定连接。
  9. 根据权利要求8所述的一种基于微光学器件的端到端光电检测系统，其特征是，所述微光学层(9)的一侧设有AR镀膜(10)，所述微光学层(9)通过AR镀膜(10)与光学保护屏(7)固定连接，所述微光学层(9)相对应的另一侧与玻璃基板(8)固定连接。
PCT/CN2022/098893 2021-06-17 2022-06-15 一种基于微光学器件的端到端光电检测系统及方法 WO2022262760A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202110672129.7A CN113408545B (zh) 2021-06-17 2021-06-17 一种基于微光学器件的端到端光电检测系统及方法
CN202110672129.7 2021-06-17

Publications (1)

Publication Number Publication Date
WO2022262760A1 true WO2022262760A1 (zh) 2022-12-22

Family

ID=77684690

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/098893 WO2022262760A1 (zh) 2021-06-17 2022-06-15 一种基于微光学器件的端到端光电检测系统及方法

Country Status (2)

Country Link
CN (1) CN113408545B (zh)
WO (1) WO2022262760A1 (zh)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113408545B (zh) * 2021-06-17 2024-03-01 浙江光仑科技有限公司 一种基于微光学器件的端到端光电检测系统及方法

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5953452A (en) * 1992-11-05 1999-09-14 The Johns Hopkins University Optical-digital method and processor for pattern recognition
US20200097005A1 (en) * 2018-09-26 2020-03-26 Toyota Jidosha Kabushiki Kaisha Object detection device, object detection method, and vehicle controller
WO2020247828A1 (en) * 2019-06-07 2020-12-10 The Regents Of The University Of California Diffractive deep neural networks with differential and class-specific detection
CN112839149A (zh) * 2020-12-31 2021-05-25 光沦科技(深圳)有限公司 异构微光学成像模块及其图像重建方法和装置
CN113408545A (zh) * 2021-06-17 2021-09-17 浙江光仑科技有限公司 一种基于微光学器件的端到端光电检测系统及方法

Family Cites Families (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
GB2332545B (en) * 1997-12-17 2002-03-13 Marconi Gec Ltd Radar return signal signature analysis
JP5067885B2 (ja) * 2007-09-14 2012-11-07 株式会社リコー 画像入力装置及び個人認証装置
JP2010015024A (ja) * 2008-07-04 2010-01-21 Canon Inc 撮像装置、その制御方法、プログラム及び記憶媒体
FR2989472B1 (fr) * 2012-04-13 2015-09-25 Bioaxial Procede et dispositif optique
FR2997518B1 (fr) * 2012-10-30 2014-12-05 Centre Nat Rech Scient Systeme d'imagerie holographique auto-reference
CN108351289B (zh) * 2015-10-28 2021-11-26 国立大学法人东京大学 分析装置
CN106535384B (zh) * 2016-12-23 2023-10-13 成都光数云联科技有限公司 具有分析系统的微波炉
CN112200726B (zh) * 2020-10-29 2023-04-07 陈根生 基于无透镜显微成像的尿沉渣有形成分检测方法及系统

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5953452A (en) * 1992-11-05 1999-09-14 The Johns Hopkins University Optical-digital method and processor for pattern recognition
US20200097005A1 (en) * 2018-09-26 2020-03-26 Toyota Jidosha Kabushiki Kaisha Object detection device, object detection method, and vehicle controller
WO2020247828A1 (en) * 2019-06-07 2020-12-10 The Regents Of The University Of California Diffractive deep neural networks with differential and class-specific detection
CN112839149A (zh) * 2020-12-31 2021-05-25 光沦科技(深圳)有限公司 异构微光学成像模块及其图像重建方法和装置
CN113408545A (zh) * 2021-06-17 2021-09-17 浙江光仑科技有限公司 一种基于微光学器件的端到端光电检测系统及方法

Also Published As

Publication number Publication date
CN113408545A (zh) 2021-09-17
CN113408545B (zh) 2024-03-01

Similar Documents

Publication Publication Date Title
CN111046703B (zh) 人脸防伪检测方法、装置及多目相机
DE112020004810T5 (de) Systeme und verfahren zur erfassung von oberflächennormalen mit polarisation
CN109034184B (zh) 一种基于深度学习的均压环检测识别方法
JP6305171B2 (ja) シーン内の物体を検出する方法
CN112288663A (zh) 一种红外与可见光图像融合方法及系统
CN109086675B (zh) 一种基于光场成像技术的人脸识别及攻击检测方法及其装置
JP2022509034A (ja) ニューラルネットワークを使用した輝点除去
CN104978724B (zh) 基于多尺度变换与脉冲耦合神经网络的红外偏振融合方法
CN107677216A (zh) 一种基于光度立体视觉的多个磨粒三维形貌同步获取方法
WO2022262760A1 (zh) 一种基于微光学器件的端到端光电检测系统及方法
CN108632506A (zh) 一种微透镜阵列成像系统
CN114782298B (zh) 一种具有区域注意力的红外与可见光图像融合方法
Asmare et al. Image enhancement by fusion in contourlet transform
CN113095358A (zh) 一种图像融合方法及系统
WO2022143571A1 (zh) 异构微光学成像模块及其图像重建方法和装置
CN115311186A (zh) 一种红外与可见光图像跨尺度注意力对抗融合方法及终端
US20190107627A1 (en) Optoelectronic Systems
CN114298151A (zh) 一种基于点云数据与图像数据融合的3d目标检测方法
WO2020163742A1 (en) Integrated spatial phase imaging
Xu Accurate measurement of structural vibration based on digital image processing technology
CN110197178A (zh) 一种图谱特征融合深度网络的大米种类快速鉴别检测装置及其检测方法
CN115950534A (zh) 分焦平面式高光通量仿生多光谱偏振成像探测器
CN114170668A (zh) 一种高光谱人脸识别方法及系统
Sun et al. A vision system based on TOF 3D imaging technology applied to robotic citrus harvesting
CN114821239A (zh) 一种有雾环境下的病虫害检测方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22824236

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE