WO2020147574A1 - Deep-learning-based stereo matching method for binocular dynamic vision sensor - Google Patents

Deep-learning-based stereo matching method for binocular dynamic vision sensor Download PDF

Info

Publication number
WO2020147574A1
WO2020147574A1 PCT/CN2019/130224 CN2019130224W WO2020147574A1 WO 2020147574 A1 WO2020147574 A1 WO 2020147574A1 CN 2019130224 W CN2019130224 W CN 2019130224W WO 2020147574 A1 WO2020147574 A1 WO 2020147574A1
Authority
WO
WIPO (PCT)
Prior art keywords
event
dynamic vision
stereo matching
vision sensor
point
Prior art date
Application number
PCT/CN2019/130224
Other languages
French (fr)
Chinese (zh)
Inventor
陈广
刘佩根
沈律宇
宁翔宇
唐笠轩
Original Assignee
同济大学
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 同济大学 filed Critical 同济大学
Publication of WO2020147574A1 publication Critical patent/WO2020147574A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration

Definitions

  • the invention relates to the technical field of image matching, in particular to a stereo matching method for binocular dynamic vision sensors based on deep learning.
  • the dynamic vision sensor outputs a stream of events by detecting changes in the logarithmic intensity of image brightness, where each event has location, polarity, and time stamp information. Compared with traditional cameras, it has the advantages of low latency, high time resolution and large dynamic range.
  • binocular stereo matching technology is an important way to obtain image depth information.
  • the application of traditional binocular stereo matching technology on the mobile terminal is greatly restricted.
  • the purpose of the present invention is to provide a stereo matching method for binocular dynamic vision sensor based on deep learning in order to overcome the defects of the prior art.
  • a stereo matching method for binocular dynamic vision sensor based on deep learning including the following steps:
  • the event training point pair is characterized, and sent to the twin neural network for training, and stereo matching is performed according to the training result.
  • the step 1) specifically includes the following steps:
  • the calculation formula of the position coordinates (x R , y R ) of the point of interest in the right sensor is:
  • (x L , y L ) are the position coordinates of the point of interest in the left sensor
  • d is the parallax value
  • z is the corresponding depth information
  • b and f are the baseline distance and focal length of the binocular dynamic vision sensor.
  • the characterization method of the construction event is specifically:
  • mi is the normalized value
  • c max is the maximum number of events in each small square counted in different time intervals ⁇ t;
  • using the twin neural network training event training point pair specifically includes the following steps:
  • the number of representations of matched and unmatched event point pairs are sent to the twin neural network in equal numbers.
  • the present invention has the following advantages:
  • the present invention can effectively solve the problem of stereo matching for dynamic vision sensors. It directly processes data on the generated event stream, which can effectively reduce the amount of calculation, reduce the required computing resources, and improve the matching speed, which is easy to implement on the mobile terminal.
  • the present invention uses event distribution information around the point of interest to characterize the point of interest, and the used information is rich and stable. And apply a large amount of data to train the neural network, and perform stereo matching based on deep learning, which can make the matching method more robust and improve the matching accuracy.
  • Figure 1 is a flow chart of the stereo matching of the present invention.
  • Figure 2 is a schematic plan view of the characterization method.
  • Figure 3 is a partial representation diagram.
  • Figure 4 is a schematic diagram of the twin neural network.
  • the present invention provides a stereo matching method for binocular dynamic vision sensors based on deep learning.
  • the method can characterize the event stream output by the left and right dynamic vision sensors, and perform matching through a trained neural network to improve the matching accuracy. At the same time improve the matching speed.
  • the method includes the following steps:
  • step (1) the method of generating event training point pairs is as follows:
  • d is the parallax value
  • calculation formula is:
  • z is the depth information corresponding to the event point
  • b and f are the baseline distance and focal length of the binocular dynamic vision sensor, which are known quantities.
  • step (2) the method of constructing event representation is as follows:
  • mi is the normalized value
  • c max is the maximum number of events in each small square counted in different time intervals ⁇ t.
  • step (3) the training method for the representation is as follows:
  • step (1) Using the method described in step (1), take multiple different time points on the existing binocular event camera data set, and generate multiple event point pairs at different locations at each time point.
  • An event point is characterized separately to obtain an N*N*S-dimensional representation vector, which is sent to the twin neural network, and an M-dimensional description vector is output.
  • the neural network is shown in Figure 4.
  • (4-2) Calculate the Euclidean distance between the M-dimensional description vectors generated by the corresponding point pair, and adjust the neural network parameters to reduce the distance value.
  • a representation is established and sent to the trained neural network to generate a description vector.
  • characterize all the positions on the same epipolar line in the right sensor in turn, send them to the neural network to generate description vectors, calculate and compare the Euclidean distance between the description vectors generated by the characterization on both sides, and take the smallest distance , And use the position corresponding to the description vector on the right as the matching point.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Image Analysis (AREA)

Abstract

A deep-learning-based stereo matching method for a binocular dynamic vision sensor, the method comprising the following steps: 1) generating training point pairs according to depth information in a data set of a binocular event camera; 2) constructing a representation mode suitable for events in an event flow of a dynamic vision sensor; and 3) representing the event training point pairs according to the representation mode, and transmitting same to twin neural networks for training, and performing stereo matching according to a training result. Compared with the prior art, the method has the advantages of high matching precision and a fast matching speed.

Description

一种基于深度学习的双目动态视觉传感器立体匹配方法A stereo matching method for binocular dynamic vision sensor based on deep learning 技术领域Technical field
本发明涉及图像匹配技术领域，尤其是涉及一种基于深度学习的双目动态视觉传感器立体匹配方法。The invention relates to the technical field of image matching, in particular to a stereo matching method for binocular dynamic vision sensors based on deep learning.
背景技术Background technique
动态视觉传感器通过检测图像亮度的对数强度的变化来输出事件流，其中每个事件都具有位置、极性和时间戳信息。与传统相机相比，其具有延迟低，时间分辨率高，动态范围大等优势。The dynamic vision sensor outputs a stream of events by detecting changes in the logarithmic intensity of image brightness, where each event has location, polarity, and time stamp information. Compared with traditional cameras, it has the advantages of low latency, high time resolution and large dynamic range.
在传统图像处理技术中，双目立体匹配技术是获得图像深度信息的重要途径。但因传统视觉传感器输出数据量大，耗费资源高，因此传统双目立体匹配技术在移动端的应用受到很大的限制。In traditional image processing technology, binocular stereo matching technology is an important way to obtain image depth information. However, due to the large amount of output data of traditional vision sensors and high resource consumption, the application of traditional binocular stereo matching technology on the mobile terminal is greatly restricted.
发明内容Summary of the invention
本发明的目的就是为了克服上述现有技术存在的缺陷而提供一种基于深度学习的双目动态视觉传感器立体匹配方法。The purpose of the present invention is to provide a stereo matching method for binocular dynamic vision sensor based on deep learning in order to overcome the defects of the prior art.
本发明的目的可以通过以下技术方案来实现：The object of the present invention can be achieved by the following technical solutions:
一种基于深度学习的双目动态视觉传感器立体匹配方法，包括以下步骤：A stereo matching method for binocular dynamic vision sensor based on deep learning, including the following steps:
1)根据双目事件相机数据集中的深度信息生成训练点对；1) Generate training point pairs based on the depth information in the binocular event camera data set;
2)构建适用于动态视觉传感器事件流中事件的表征方式；2) Construct a representation method suitable for events in the event stream of dynamic vision sensors;
3)根据表征方式对事件训练点对进行表征，并送入孪生神经网络进行训练，并根据训练结果进行立体匹配。3) According to the representation method, the event training point pair is characterized, and sent to the twin neural network for training, and stereo matching is performed according to the training result.
所述的步骤1)具体包括以下步骤：The step 1) specifically includes the following steps:
11)在左侧动态视觉传感器视域范围内，随机选取一个事件作为兴趣点；11) Randomly select an event as a point of interest within the field of view of the dynamic vision sensor on the left;
12)根据该兴趣点在左侧传感器内的位置信息及真实深度信息，以极线为限制，将其投影到右侧动态视觉传感器上，获得该兴趣点在右侧传感器内的位置坐标信息，形成训练点对。12) According to the position information of the point of interest in the left sensor and the real depth information, using the epipolar line as the limit, project it onto the right dynamic vision sensor to obtain the position coordinate information of the point of interest in the right sensor, Form training point pairs.
所述的步骤12)中，兴趣点在右侧传感器内的位置坐标(x R，y R)的计算式为： In the step 12), the calculation formula of the position coordinates (x R , y R ) of the point of interest in the right sensor is:
Figure PCTCN2019130224-appb-000001
Figure PCTCN2019130224-appb-000001
Figure PCTCN2019130224-appb-000002
Figure PCTCN2019130224-appb-000002
其中，(x L，y L)为兴趣点在左侧传感器内的位置坐标，d为视差值，z为对应的深度信息，b和f为该双目动态视觉传感器的基线距离与焦距。 Among them, (x L , y L ) are the position coordinates of the point of interest in the left sensor, d is the parallax value, z is the corresponding depth information, and b and f are the baseline distance and focal length of the binocular dynamic vision sensor.
所述的步骤2)中，构建事件的表征方式具体为：In the step 2), the characterization method of the construction event is specifically:
21)以表征点为几何中心，建立边长为L并与传感器视角对齐的正方形区域，将此正方形区域划分为相等的N*N个正方形小区域，N为奇数；21) Taking the characterization point as the geometric center, establish a square area with side length L and aligned with the sensor viewing angle, and divide the square area into equal N*N square small areas, where N is an odd number;
22)选取连续S(S为偶数)个时间间隔Δt，使得表征点的事件时间戳位于
Figure PCTCN2019130224-appb-000003
处，统计每个时间间隔Δt内，各小正方形区域内产生的事件数目c i； 22) Select consecutive S (S is an even number) time interval Δt, so that the event timestamp of the characteristic point is located at
Figure PCTCN2019130224-appb-000003
Count the number of events c i generated in each small square area in each time interval Δt;
23)将不同时间间隔Δt内，每个小正方形内的事件数进行归一化，作为该小正方形的值，则有：23) Normalize the number of events in each small square in different time intervals Δt, as the value of the small square, then:
Figure PCTCN2019130224-appb-000004
Figure PCTCN2019130224-appb-000004
c max＝max(c i) c max ＝max(c i )
其中，m i为归一化后的值，c max为在不同时间间隔Δt内，统计的各小正方形内事件数最大值； Among them, mi is the normalized value, and c max is the maximum number of events in each small square counted in different time intervals Δt;
24)将归一化后的值m i从小至大的排序，形成N*N*S维表征向量。 24) Sort the normalized value mi from small to large to form an N*N*S-dimensional representation vector.
所述的步骤3)中，采用孪生神经网络训练事件训练点对具体包括以下步骤：In the step 3), using the twin neural network training event training point pair specifically includes the following steps:
31)将匹配的训练点对的表征向量送入孪生神经网络，输出其各自的M维描述向量；31) Send the representation vector of the matched training point pair to the twin neural network, and output its respective M-dimensional description vector;
32)计算生成的M维描述向量间的欧几里得距离，并调整孪生神经网络的参数，缩小距离值；32) Calculate the Euclidean distance between the generated M-dimensional description vectors, and adjust the parameters of the twin neural network to reduce the distance value;
33)将不匹配的两个事件点的表征向量送入调整参数后的孪生神经网络，输出各自的M维描述向量；33) Send the representation vectors of the two unmatched event points to the twin neural network after adjusting the parameters, and output their respective M-dimensional description vectors;
34)计算不匹配的两个事件点生成的M维描述向量间的欧几里得距离，调整神经网络参数，扩大其距离值；34) Calculate the Euclidean distance between the M-dimensional description vectors generated by the two unmatched event points, adjust the neural network parameters, and expand the distance value;
35)进行立体匹配。35) Perform stereo matching.
所述的步骤4)中，匹配与不匹配事件点对的表征送入孪生神经网络的数量相等。In the step 4), the number of representations of matched and unmatched event point pairs are sent to the twin neural network in equal numbers.
与现有技术相比，本发明具有以下优点：Compared with the prior art, the present invention has the following advantages:
一、本发明能够有效解决针对动态视觉传感器立体匹配的问题，其直接对生成的事件流进行数据处理，可以有效减少计算量，降低所需计算资源，提升匹配速度，易于在移动端进行实现。1. The present invention can effectively solve the problem of stereo matching for dynamic vision sensors. It directly processes data on the generated event stream, which can effectively reduce the amount of calculation, reduce the required computing resources, and improve the matching speed, which is easy to implement on the mobile terminal.
二、本发明使用兴趣点周围事件分布信息对兴趣点进行表征，所用信息丰富，稳定性好。并应用大量数据对神经网络进行训练，以基于深度学习的方式进行立体匹配，可以使匹配方法具有较强的鲁棒性，提升匹配准确度。2. The present invention uses event distribution information around the point of interest to characterize the point of interest, and the used information is rich and stable. And apply a large amount of data to train the neural network, and perform stereo matching based on deep learning, which can make the matching method more robust and improve the matching accuracy.
附图说明BRIEF DESCRIPTION
图1为本发明的立体匹配流程图。Figure 1 is a flow chart of the stereo matching of the present invention.
图2为表征方法平面示意图。Figure 2 is a schematic plan view of the characterization method.
图3为局部表征示意图。Figure 3 is a partial representation diagram.
图4为孪生神经网络示意图。Figure 4 is a schematic diagram of the twin neural network.
具体实施方式detailed description
下面结合附图和具体实施例对本发明进行详细说明。The present invention will be described in detail below with reference to the drawings and specific embodiments.
实施例Examples
本发明提供了一种基于深度学习的双目动态视觉传感器立体匹配的方法，该方法能够对左右动态视觉传感器输出的事件流进行表征，并通过训练好的神经网络进行匹配，提高匹配准确率的同时提升匹配速度。该方法包括如下步骤：The present invention provides a stereo matching method for binocular dynamic vision sensors based on deep learning. The method can characterize the event stream output by the left and right dynamic vision sensors, and perform matching through a trained neural network to improve the matching accuracy. At the same time improve the matching speed. The method includes the following steps:
(1)根据已有双目事件相机的数据集，以其提供的深度信息为基础，生成训练点对；(1) Generate training point pairs based on the data set of the existing binocular event camera and the depth information provided by it;
(2)构建适用于动态视觉传感器事件流事件的表征方法；(2) Construct a representation method suitable for event stream events of dynamic vision sensors;
(3)用构建的表征方法对事件训练点对进行表征，并送入神经网络进行训练。(3) Use the constructed characterization method to characterize the event training point pair and send it to the neural network for training.
在步骤(1)中，事件训练点对的生成方式如下：In step (1), the method of generating event training point pairs is as follows:
(2-1)在左侧动态视觉传感器视域范围内，随机选取一个事件作为兴趣点。(2-1) Randomly select an event as a point of interest within the field of view of the dynamic vision sensor on the left.
(2-2)以传感器左上角顶点为原点，以正右与正下方向分别为x，y正半轴，记录该兴趣点的位置信息(x L，y L)。根据双目相机投影原理，其右侧对应点的坐标(x R，y R)应满足： (2-2) Taking the vertex of the upper left corner of the sensor as the origin, and taking the positive and right directions as the x and y positive semi-axes respectively, record the position information (x L , y L ) of the point of interest. According to the projection principle of the binocular camera, the coordinates (x R , y R ) of the corresponding point on the right side should satisfy:
Figure PCTCN2019130224-appb-000005
Figure PCTCN2019130224-appb-000005
其中，d为视差值，计算公式为：Among them, d is the parallax value, and the calculation formula is:
Figure PCTCN2019130224-appb-000006
Figure PCTCN2019130224-appb-000006
其中z为该事件点对应的深度信息，b和f为该双目动态视觉传感器的基线距离与焦距，为已知量。Where z is the depth information corresponding to the event point, b and f are the baseline distance and focal length of the binocular dynamic vision sensor, which are known quantities.
在步骤(2)中，事件的表征构建方法如下：In step (2), the method of constructing event representation is as follows:
(3-1)以表征点为几何中心，建立边长为L并与传感器视角对齐的正方形，将此正方形划分为相等的N*N个正方形小区域，如图2所示。此实施例中，边长L取33个像素值，N取值为11，即存在121个小正方形，每个小正方形的边长为3个像素值。(3-1) Taking the characterizing point as the geometric center, establish a square with side length L and aligned with the sensor's viewing angle, and divide this square into equal N*N square small areas, as shown in Figure 2. In this embodiment, the side length L is 33 pixels, and N is 11, that is, there are 121 small squares, and the side length of each small square is 3 pixels.
(3-2)取连续S个时间间隔Δt，使所选取的事件时间戳位于
Figure PCTCN2019130224-appb-000007
处，统计每个时间间隔Δt内，各小正方形区域内产生的事件数目c i，示意图如图3所示。 (3-2) Take S consecutive time intervals Δt so that the selected event timestamp is at
Figure PCTCN2019130224-appb-000007
Count the number of events c i generated in each small square area in each time interval Δt, as shown in Figure 3.
(3-3)将不同时间间隔Δt内，每个小正方形内的事件数进行归一化，作为该小正方形的值。归一化公式为：(3-3) Normalize the number of events in each small square in different time intervals Δt as the value of the small square. The normalization formula is:
Figure PCTCN2019130224-appb-000008
Figure PCTCN2019130224-appb-000008
c max＝max(c i) c max ＝max(c i )
其中，m i为归一化后的值，c max为在不同时间间隔Δt内，统计的各小正方形内事件数最大值。 Among them, mi is the normalized value, and c max is the maximum number of events in each small square counted in different time intervals Δt.
(3-4)将m i从小至大的排序，形成一个N*N*S维表征向量。 (3-4) Sort mi from small to large to form an N*N*S-dimensional representation vector.
在步骤(3)中，对表征的训练方法如下：In step (3), the training method for the representation is as follows:
(4-1)运用步骤(1)所述方法，在现有双目事件相机数据集上，取多个不同时间点，并在每一个时间点不同位置处生成多个事件点对，对每一个事件点分别进行表征，获得N*N*S维表征向量，送入到孪生神经网络中，输出M维描述向量。本实施例中，神经网络如图4所示。(4-1) Using the method described in step (1), take multiple different time points on the existing binocular event camera data set, and generate multiple event point pairs at different locations at each time point. An event point is characterized separately to obtain an N*N*S-dimensional representation vector, which is sent to the twin neural network, and an M-dimensional description vector is output. In this embodiment, the neural network is shown in Figure 4.
(4-2)计算对应点对所生成的M维描述向量间的欧几里得距离，调整神经网络参 数，使得其距离值缩小。(4-2) Calculate the Euclidean distance between the M-dimensional description vectors generated by the corresponding point pair, and adjust the neural network parameters to reduce the distance value.
(4-3)同理，将不匹配的两个事件点的表征送入上述神经网络，输出各自的M维描述向量。(4-3) Similarly, the representations of the two unmatched event points are sent to the aforementioned neural network, and their respective M-dimensional description vectors are output.
(4-4)计算不匹配点对两个向量间的欧几里得距离，调整神经网络参数，扩大其距离值，训练过程中，匹配与不匹配事件点对的表征送入孪生神经网络的数量相等。(4-4) Calculate the Euclidean distance between the two vectors of the mismatched point pair, adjust the neural network parameters, and expand the distance value. During the training process, the representations of the matched and mismatched event point pairs are sent to the twin neural network. The number is equal.
(4-5)进行立体匹配。(4-5) Perform stereo matching.
针对左侧动态视觉传感器每一个新生成的事件，建立表征，并送入训练好的神经网络生成描述向量。同时，对右侧传感器中同一极线上的所有位置依次建立表征，送入神经网络生成描述向量，计算两侧表征所生成描述向量间的欧几里得距离并进行比较，取其距离最小者，将该右侧描述向量所对应的位置作为匹配点。For each new event generated by the dynamic vision sensor on the left, a representation is established and sent to the trained neural network to generate a description vector. At the same time, characterize all the positions on the same epipolar line in the right sensor in turn, send them to the neural network to generate description vectors, calculate and compare the Euclidean distance between the description vectors generated by the characterization on both sides, and take the smallest distance , And use the position corresponding to the description vector on the right as the matching point.
上述的对实施例的描述是为便于该技术领域的普通技术人员能理解和应用本发明。熟悉本领域技术的人员显然可以容易地对这些实施例做出各种修改，并把在此说明的一般原理应用到其他实施例中而不必经过创造性的劳动。因此，本发明不限于这里的实施例，本领域技术人员根据本发明的揭示，不脱离本发明范畴所做出的改进和修改都应该在本发明的保护范围之内。The above description of the embodiments is to facilitate those of ordinary skill in the art to understand and apply the present invention. Those skilled in the art can obviously easily make various modifications to these embodiments, and apply the general principles described herein to other embodiments without creative work. Therefore, the present invention is not limited to the embodiments herein. Based on the disclosure of the present invention, those skilled in the art should make improvements and modifications without departing from the scope of the present invention within the protection scope of the present invention.

Claims (6)

  1. 一种基于深度学习的双目动态视觉传感器立体匹配方法，其特征在于，包括以下步骤：A stereo matching method for binocular dynamic vision sensor based on deep learning, characterized in that it comprises the following steps:
    1)根据双目事件相机数据集中的深度信息生成训练点对；1) Generate training point pairs based on the depth information in the binocular event camera data set;
    2)构建适用于动态视觉传感器事件流中事件的表征方式；2) Construct a representation method suitable for events in the event stream of dynamic vision sensors;
    3)根据表征方式对事件训练点对进行表征，并送入孪生神经网络进行训练，并根据训练结果进行立体匹配。3) According to the representation method, the event training point pair is characterized, and sent to the twin neural network for training, and stereo matching is performed according to the training result.
  2. 根据权利要求1所述的一种基于深度学习的双目动态视觉传感器立体匹配方法，其特征在于，所述的步骤1)具体包括以下步骤：The stereo matching method of binocular dynamic vision sensor based on deep learning according to claim 1, wherein said step 1) specifically includes the following steps:
    11)在左侧动态视觉传感器视域范围内，随机选取一个事件作为兴趣点；11) Randomly select an event as a point of interest within the field of view of the dynamic vision sensor on the left;
    12)根据该兴趣点在左侧传感器内的位置信息及真实深度信息，以极线为限制，将其投影到右侧动态视觉传感器上，获得该兴趣点在右侧传感器内的位置坐标信息，形成训练点对。12) According to the position information of the point of interest in the left sensor and the real depth information, using the epipolar line as the limit, project it onto the right dynamic vision sensor to obtain the position coordinate information of the point of interest in the right sensor, Form training point pairs.
  3. 根据权利要求2所述的一种基于深度学习的双目动态视觉传感器立体匹配方法，其特征在于，所述的步骤12)中，兴趣点在右侧传感器内的位置坐标(x R,y R)的计算式为： The stereo matching method of binocular dynamic vision sensor based on deep learning according to claim 2, characterized in that, in said step 12), the position coordinates (x R , y R ) Is calculated as:
    Figure PCTCN2019130224-appb-100001
    Figure PCTCN2019130224-appb-100001
    Figure PCTCN2019130224-appb-100002
    Figure PCTCN2019130224-appb-100002
    其中，(x L,y L)为兴趣点在左侧传感器内的位置坐标，d为视差值，z为对应的深度信息，b和f为该双目动态视觉传感器的基线距离与焦距。 Among them, (x L , y L ) are the position coordinates of the point of interest in the left sensor, d is the parallax value, z is the corresponding depth information, and b and f are the baseline distance and focal length of the binocular dynamic vision sensor.
  4. 根据权利要求1所述的一种基于深度学习的双目动态视觉传感器立体匹配方法，其特征在于，所述的步骤2)中，构建事件的表征方式具体为：The stereo matching method of binocular dynamic vision sensor based on deep learning according to claim 1, wherein in said step 2), the representation method of constructing event is specifically:
    21)以表征点为几何中心，建立边长为L并与传感器视角对齐的正方形区域，将此正方形区域划分为相等的N*N个正方形小区域；21) Taking the characterization point as the geometric center, establish a square area with side length L and aligned with the sensor's viewing angle, and divide this square area into equal N*N square small areas;
    22)选取连续S个时间间隔Δt，使得表征点的事件时间戳位于
    Figure PCTCN2019130224-appb-100003
    处，统计每个时间间隔Δt内，各小正方形区域内产生的事件数目c i；     22) Select consecutive S time intervals Δt, so that the event timestamp of the characteristic point is located at
    Figure PCTCN2019130224-appb-100003
    Count the number of events c i generated in each small square area in each time interval Δt;
    23)将不同时间间隔Δt内，每个小正方形内的事件数进行归一化，作为该小正方形的值，则有：23) Normalize the number of events in each small square in different time intervals Δt, as the value of the small square, then:
    Figure PCTCN2019130224-appb-100004
    Figure PCTCN2019130224-appb-100004
    c max＝max(c i) c max ＝max(c i )
    其中，m i为归一化后的值，c max为在不同时间间隔Δt内，统计的各小正方形内事件数最大值； Among them, mi is the normalized value, and c max is the maximum number of events in each small square counted in different time intervals Δt;
    24)将归一化后的值m i从小至大的排序，形成N*N*S维表征向量。 24) Sort the normalized value mi from small to large to form an N*N*S-dimensional representation vector.
  5. 根据权利要求1所述的一种基于深度学习的双目动态视觉传感器立体匹配方法，其特征在于，所述的步骤3)中，采用孪生神经网络训练事件训练点对具体包括以下步骤：The stereo matching method of binocular dynamic vision sensor based on deep learning according to claim 1, characterized in that, in said step 3), using twin neural network training event training point pair specifically includes the following steps:
    31)将匹配的训练点对的表征向量送入孪生神经网络，输出其各自的M维描述向量；31) Send the representation vector of the matched training point pair to the twin neural network, and output its respective M-dimensional description vector;
    32)计算生成的M维描述向量间的欧几里得距离，并调整孪生神经网络的参数，缩小距 离值；32) Calculate the Euclidean distance between the generated M-dimensional description vectors, and adjust the parameters of the twin neural network to reduce the distance value;
    33)将不匹配的两个事件点的表征向量送入调整参数后的孪生神经网络，输出各自的M维描述向量；33) Send the representation vectors of the two unmatched event points to the twin neural network after adjusting the parameters, and output their respective M-dimensional description vectors;
    34)计算不匹配的两个事件点生成的M维描述向量间的欧几里得距离，调整神经网络参数，扩大其距离值；34) Calculate the Euclidean distance between the M-dimensional description vectors generated by the two unmatched event points, adjust the neural network parameters, and expand the distance value;
    35)进行立体匹配。35) Perform stereo matching.
  6. 根据权利要求5所述的一种基于深度学习的双目动态视觉传感器立体匹配方法，其特征在于，所述的步骤4)中，匹配与不匹配事件点对的表征送入孪生神经网络的数量相等。The stereo matching method of binocular dynamic vision sensor based on deep learning according to claim 5, wherein in said step 4), the number of the representations of matched and unmatched event point pairs sent to the twin neural network equal.
PCT/CN2019/130224 2019-01-17 2019-12-31 Deep-learning-based stereo matching method for binocular dynamic vision sensor WO2020147574A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910044239.1 2019-01-17
CN201910044239.1A CN109801314B (en) 2019-01-17 2019-01-17 Binocular dynamic vision sensor stereo matching method based on deep learning

Publications (1)

Publication Number Publication Date
WO2020147574A1 true WO2020147574A1 (en) 2020-07-23

Family

ID=66559622

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/130224 WO2020147574A1 (en) 2019-01-17 2019-12-31 Deep-learning-based stereo matching method for binocular dynamic vision sensor

Country Status (2)

Country Link
CN (1) CN109801314B (en)
WO (1) WO2020147574A1 (en)

Families Citing this family (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109801314B (en) * 2019-01-17 2020-10-02 同济大学 Binocular dynamic vision sensor stereo matching method based on deep learning
CN110414558B (en) * 2019-06-24 2021-07-20 武汉大学 Feature point matching method based on event camera
CN110399908B (en) * 2019-07-04 2021-06-08 西北工业大学 Event-based camera classification method and apparatus, storage medium, and electronic apparatus
CN110599414B (en) * 2019-08-28 2021-08-03 武汉大学 Event camera data processing-oriented time-space normalization method
CN111127532B (en) * 2019-12-31 2020-12-22 成都信息工程大学 Medical image deformation registration method and system based on deep learning characteristic optical flow
CN111696143B (en) * 2020-06-16 2022-11-04 清华大学 Event data registration method and system
CN113222945B (en) * 2021-05-19 2024-02-09 西安电子科技大学 Depth information measuring method based on binocular event camera

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160203614A1 (en) * 2015-01-14 2016-07-14 Samsung Electronics Co., Ltd. Method and apparatus of detecting object using event-based sensor
CN107403154A (en) * 2017-07-20 2017-11-28 四川大学 A kind of gait recognition method based on dynamic visual sensor
WO2018219931A1 (en) * 2017-05-29 2018-12-06 Universität Zürich Block-matching optical flow and stereo vision for dynamic vision sensors
CN109801314A (en) * 2019-01-17 2019-05-24 同济大学 A kind of binocular dynamic visual sensor solid matching method based on deep learning

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP3323084A4 (en) * 2015-07-16 2019-07-10 Blast Motion Inc. Multi-sensor event detection and tagging system
US9934557B2 (en) * 2016-03-22 2018-04-03 Samsung Electronics Co., Ltd Method and apparatus of image representation and processing for dynamic vision sensor
CN108073929B (en) * 2016-11-15 2023-11-24 北京三星通信技术研究有限公司 Object detection method and device based on dynamic vision sensor
CN108986223A (en) * 2017-05-31 2018-12-11 上海拆名晃信息科技有限公司 A kind of method and apparatus of 3 D scene rebuilding
CN107180215B (en) * 2017-05-31 2020-01-31 同济大学 Parking lot automatic mapping and high-precision positioning method based on library position and two-dimensional code

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160203614A1 (en) * 2015-01-14 2016-07-14 Samsung Electronics Co., Ltd. Method and apparatus of detecting object using event-based sensor
WO2018219931A1 (en) * 2017-05-29 2018-12-06 Universität Zürich Block-matching optical flow and stereo vision for dynamic vision sensors
CN107403154A (en) * 2017-07-20 2017-11-28 四川大学 A kind of gait recognition method based on dynamic visual sensor
CN109801314A (en) * 2019-01-17 2019-05-24 同济大学 A kind of binocular dynamic visual sensor solid matching method based on deep learning

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
PAUL ROGISTER ; RYAD BENOSMAN ; SIO-HOI IENG ; PATRICK LICHTSTEINER ; TOBI DELBRUCK: "Asynchronous Event-Based Binocular Stereo Matching", IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, vol. 23, no. 2, 29 February 2012 (2012-02-29), pages 347 - 353, XP011406609, ISSN: 2162-237X, DOI: 10.1109/TNNLS.2011.2180025 *
ZHANG, XINYU ET AL: "Overview of Deep Learning Intelligent Driving Methods", JOURNAL OF TSINGHUA UNIVERSITY SCIENCE AND TECHNOLOGY, vol. 58, no. 4, 31 December 2018 (2018-12-31), ISSN: 1000-0054 *

Also Published As

Publication number Publication date
CN109801314A (en) 2019-05-24
CN109801314B (en) 2020-10-02

Similar Documents

Publication Publication Date Title
WO2020147574A1 (en) Deep-learning-based stereo matching method for binocular dynamic vision sensor
CN110991266B (en) Binocular face living body detection method and device
CN105279372B (en) A kind of method and apparatus of determining depth of building
WO2017107524A1 (en) Imaging distortion test method and apparatus for virtual reality helmet
CN109544628B (en) Accurate reading identification system and method for pointer instrument
WO2017121058A1 (en) All-optical information acquisition system
CN109163657A (en) A kind of circular target position and posture detection method rebuild based on binocular vision 3 D
CN103617615B (en) Radial distortion parameter acquisition methods and acquisition device
WO2021179772A1 (en) Calibration method, position determination method and apparatus, electronic device and storage medium
CN108833912A (en) A kind of measurement method and system of video camera machine core optical axis center and field angle
CN106780389A (en) A kind of fisheye image correcting method and device based on Coordinate Conversion
CN104123725B (en) A kind of computational methods of single line array camera homography matrix H
CN109656033A (en) A kind of method and device for distinguishing liquid crystal display dust and defect
CN107016697A (en) A kind of height measurement method and device
CN105139336B (en) A kind of method of multichannel full-view image conversion ball curtain flake film
CN106033614A (en) Moving object detection method of mobile camera under high parallax
WO2020143499A1 (en) Corner detection method based on dynamic vision sensor
CN103440638A (en) Method for solving camera inner parameters by utilizing bimirror device and circular point characteristics
CN110441737B (en) Sound source positioning method and equipment adopting fisheye lens
CN106558081A (en) The method for demarcating the circular cone catadioptric video camera of optical resonator system
CN116310131A (en) Three-dimensional reconstruction method considering multi-view fusion strategy
CN109241822A (en) A kind of multi-faceted method for detecting human face and system based on MTCNN
CN106595595B (en) A kind of Indoor Robot orientation method based on depth transducer
CN103278104B (en) A kind of scaling board of the dual camera systems for DIC measurement and scaling method thereof
CN106296797A (en) A kind of spatial digitizer characteristic point modeling data processing method

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19910146

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19910146

Country of ref document: EP

Kind code of ref document: A1

122 Ep: pct application non-entry in european phase

Ref document number: 19910146

Country of ref document: EP

Kind code of ref document: A1