JP7376839B2 - 3D information estimation method, 3D information estimation device and program - Google Patents

Description

The present invention relates to a three-dimensional information estimation method, a three-dimensional information estimation device, and a program technique.

There is an event-based vision sensor that asynchronously detects a change in brightness of each pixel and outputs, as event data, the coordinates, time, and sign of the amount of change only for pixels whose brightness value has changed. Below, a camera equipped with a normal image sensor may be referred to as a frame camera, and a camera equipped with an event-based vision sensor may be referred to as an event camera.

The frame camera outputs the luminance value integrated into each pixel for each frame. In a frame camera, the integration time, that is, the exposure time, and the dynamic range of the signal are the same for all pixels. Therefore, when very bright pixels and very dark pixels coexist, blown out highlights, blocked up shadows, quantization errors, etc. occur. Furthermore, if the brightness of the scene changes drastically due to the influence of lighting, the exposure time may not be able to be adjusted, resulting in blown-out highlights or blown-out shadows.

On the other hand, an event camera outputs event data asynchronously every time the luminance change of each pixel exceeds a certain threshold, so problems such as blown-out highlights, blocked-up shadows, and quantization errors do not occur. Furthermore, the event data output from the event camera is much sparser than the image output from the frame camera. Therefore, the memory for storing event data and the transmission band for transmitting event data can be reduced, and extremely high time resolution can be achieved. For the same reason, the amount of calculation and power consumption required for photographing and data processing can be kept very low compared to frame cameras.

As described above, by using an event camera, machine vision can be performed stably, with low power, and with extremely high temporal resolution even in situations where the amount of light is low or where the lighting changes rapidly.

As a three-dimensional position estimation method using an event camera, there is a SLAM (Simultaneous Localization and Mapping) method (see Non-Patent Document 1). In the SLAM method, a stationary subject is photographed while moving a camera, and the correspondence between frames is determined for the local feature points found in each frame, thereby estimating the amount of camera movement and the three-dimensional position of the feature points.

When performing the SLAM method using a frame camera, a feature amount is extracted for each pixel of an image and correspondence between frames is estimated for all feature amounts, which requires very high calculation cost. Further, since many erroneous estimates of correspondence occur, many calculations are required to estimate the three-dimensional position taking this into consideration.

When performing the SLAM method using an event camera, the number of data to be processed is small and the time resolution is high, so the range in which matching points are searched can be extremely narrowed. Therefore, when performing the SLAM method using an event camera, three-dimensional information can be estimated more quickly and stably than when performing the SLAM method using a frame camera.

Free encyclopedia "Wikipedia", "SLAM" [searched on December 2, 2020], Internet (URL: https://ja.wikipedia.org/wiki/SLAM)

However, the SLAM method cannot estimate three-dimensional information for a moving object. When estimating three-dimensional information for a moving object, a stereo method may be used. Since event cameras do not have a shutter function, it is difficult to synchronize different event cameras. Furthermore, the event vision sensor is provided with a dormant period during which measurement cannot be performed at that pixel for a certain period of time after an event is output. Therefore, if different event cameras do not start measurement at the same time, the event data patterns will be different. In this case, it is difficult to find correspondence between patterns.

As described above, there has been a problem in that it is difficult to estimate three-dimensional information of a moving object using an event camera.

In view of the above circumstances, the present invention aims to provide a technique for estimating three-dimensional information of a moving object.

One aspect of the present invention is a three-dimensional information estimation method in a three-dimensional information estimation device that focuses light rays emitted from one point on a subject surface onto a plurality of positions on an event-based vision sensor, the method comprising: Among the pixels of the vision sensor, when the amount of change in brightness value exceeds a certain value, the position of the pixel where the amount of change in brightness value exceeds the certain value, the time when the amount of change in brightness value exceeds the certain value, and and a corresponding step of associating two of the event information based on the time included in the event information among the plurality of event information acquired in the event information acquisition step. and an estimating step of estimating a distance to a subject based on the position of a pixel included in each of the two pieces of event information associated in the corresponding step.

One aspect of the present invention is a three-dimensional information estimation device that focuses light rays emitted from one point on the surface of a subject onto a plurality of positions on an event-based vision sensor, the device comprising: , when the amount of change in brightness value exceeds a certain value, obtain event information including the position of the pixel where the amount of change in brightness value exceeds the certain value and the time when the amount of change in brightness value exceeds the certain value. an event information acquisition unit that associates two pieces of event information based on the time included in the event information among the plurality of event information acquired by the event information acquisition unit; The three-dimensional information estimating device includes an estimating unit that estimates a distance to a subject based on the position of a pixel included in each of the two attached event information.

One aspect of the present invention is a program for causing a computer to execute a three-dimensional information estimation method.

According to the present invention, it becomes possible to estimate three-dimensional information of a moving object.

FIG. 2 is a block diagram showing the configuration of a three-dimensional information estimation device. FIG. 2 is a diagram showing a schematic configuration of an imaging unit. FIG. 3 is a diagram of the mask section viewed from the optical axis direction. It is a flowchart which shows the flow of three-dimensional information estimation processing. It is a flowchart which shows the flow of three-dimensional information estimation processing.

Embodiments of the present invention will be described in detail with reference to the drawings.

FIG. 1 is a block diagram showing the configuration of a three-dimensional information estimation device 100 in an embodiment. The three-dimensional information estimation device 100 includes an imaging section 110 and an event processing section 120. The imaging unit 110 outputs event data to the event processing unit 120 when an event occurs. Here, an event indicates that the amount of change in the brightness value of a pixel of an event-based vision sensor (hereinafter referred to as "event sensor") exceeds a certain value. In this embodiment, the amount of change per predetermined time is used as the amount of change.

The event data also includes the position of the pixel where the event occurred, the time when the amount of change in the brightness value exceeds a certain value (hereinafter also referred to as a "time stamp"), and code information indicating an increase or decrease in the brightness value. The code information indicating an increase/decrease in the brightness value indicates a positive value when the brightness value increases, and a negative value when the brightness value decreases. In a typical subject, events occur without synchronization at each pixel, so event data is also output each time.

The event processing section 120 includes an event information acquisition section 121, a correspondence section 122, an estimation section 123, and an event information storage section 124. The event information acquisition unit 121 acquires event information output from the imaging unit 110. The event information acquisition unit 121 stores the acquired event information in the event information storage unit 124. Note that the event information acquisition unit 121 may perform arbitrary filter processing to remove noise.

The correspondence unit 122 associates two pieces of event information among the event information stored in the event information storage unit 124 by the event information acquisition unit 121 based on time stamps. The associated event information is also expressed as an event set. The method of association will be described later.

The estimating unit 123 estimates the distance from the three-dimensional information estimation device 100 to the subject based on the position of the pixel included in each of the two pieces of associated event information. Based on this estimated distance, three-dimensional information of the subject can be estimated. The three-dimensional information estimation device 100 may output the estimated distance to a host device such as a PC (Personal Computer).

FIG. 2 is a diagram showing a schematic configuration of the imaging section 110. The imaging section 110 includes a lens 111, a mask section 112, and an event sensor 113. Further, FIG. 2 shows a subject 115 and a focusing plane 116. Note that in FIG. 2, the lens 111 and the mask section 112 are drawn apart for easy understanding, but the actual distance between the lens 111 and the mask section 112 is approximately zero. Further, D, D', C, L, L', and R shown in FIG. 2 will be used in the later explanation.

The mask portion 112 is provided with a plurality of (two in FIG. 2) openings 112A and 112B. FIG. 3 is a diagram of the mask portion 112 viewed from the optical axis direction. FIG. 3 shows the mask portion 112 and the openings 112A and 112B. Note that the openings 112A and 112B are drawn larger than their actual sizes. That is, in reality, the diameter of the openings 112A, 112B relative to the diameter of the mask portion 112 is much smaller than that shown in FIG.

In FIG. 2, reflected light from the surface of a subject 115 passes through a lens 111 and openings 112A and 112B, and is focused on an event sensor 113. In this way, the imaging unit 110 focuses the light rays emitted from one point on the subject surface onto a plurality of positions on the event sensor 113 using the openings 112A and 112B. Note that the light rays emitted from one point on the subject's surface include not only the light rays emitted by the subject itself but also the light rays reflected on the subject's surface. The position of a pixel included in the event information is expressed by XY coordinates with the event sensor 113 regarded as two-dimensional coordinates. For example, the coordinates of a pixel located five pixels to the right and three pixels upward from the pixel corresponding to the origin are (5, 3). Given the pixel pitch p, the distance from the origin (0, 0) to, for example, (7, 0) is 7p.

Here, a method of association by the correspondence unit 122 will be explained. As shown in FIG. 2, a light beam emitted from one point on the subject passes through either the opening 112A or 112B and reaches two points on the event sensor 113. If the amount of change in luminance value exceeds a certain value at pixels corresponding to the two points on the event sensor 113 reached at this time, events will occur at the same time or approximately the same time. The correspondence unit 122 associates these two pieces of event information. The timing for associating event information may be the timing when new event information is stored in the event information storage unit 124, or may be the timing that arrives at regular intervals.

The correspondence unit 122 groups event information that includes the same code information among the event information that occurred at a certain time t from the event information, and performs matching to associate event information from these. Since there are two pieces of code information, they may be grouped into two groups, plus and minus. This group is also expressed as a candidate event group.

Note that the event information that occurred at time t may be only event information whose timestamp completely matches time t, or only event information whose timestamps have a difference within a predetermined range. Note that the difference between the timestamps is considered to be within a predetermined range, for example, when it falls within an appropriate time threshold range around time t. In this case, if the time threshold is T, for example, the time around time t is from time t-T to time t+T, so the difference in time stamps is within 2T.

Depending on the specifications of the event sensor 113, event information is output for each horizontal or vertical line on the sensor. As a result, timestamps may deviate between events that occur on different lines. You can set a time threshold to account for these shifts, or you can arrange the openings on a horizontal or vertical line so that event information for corresponding events is output at the same timing. good.

In this way, event information satisfies the time-related conditions included in the event information (hereinafter also referred to as "time conditions"), and is further grouped according to code information-related conditions (hereinafter also referred to as "code conditions"). . Therefore, when there is a plurality of event information that satisfies the time condition and there is a plurality of event information that belongs to the same group by grouping based on the sign condition, a candidate event group is obtained.

Next, the correspondence unit 122 performs matching in order to determine an event set among the candidate event group. Event sets may be determined sequentially for each candidate event group, or optimization may be performed such that all event sets are determined simultaneously.

Any matching method may be used. The simplest matching method is to arbitrarily select two pieces of event information from a group of candidate events and create an event set. Furthermore, for example, the candidate event group may be further narrowed down by another method. As an example of narrowing down, when performing matching on certain event information, an epipolar line is defined based on the arrangement of openings, and only event information that occurs on the epipolar line is selected as candidates. In this case, since it is only necessary to process event information that occurs on the epipolar line, the amount of processing can be reduced. Other matching methods include template matching performed in general image processing, cross-correlation methods, and methods using neural networks.

Note that event information that occurs at a time before or after the timestamp included in the event information at the same location as the location included in the event information belonging to the candidate event group or a location near the location may be used for matching. Constraint conditions may be given to event information used for matching based on an event set determined using event information that occurred at previous and subsequent times or already estimated three-dimensional information. Further, assuming that there is a correspondence between an event set whose sign information is positive and an event set whose sign information is negative, matching may be performed using two event sets, a positive event candidate group and a negative event candidate group. For example, matching may be performed using event information with positive code information and event information with negative code information with matching time stamps.

Note that when the light emitted from points A and B on the subject reaches the event sensor 113, matching can also be performed in consideration of the case where either of the arrival points overlaps. Let the overlapping point be X, and the non-overlapping points be X_A and X_B. In X, the brightness value changes depending on the total amount of light of the two lights. Therefore, event information may be output from X at a different timing from when event information is output from X_A and X_B. At this time, the number of event data output within a certain period of time from the timing at which event information is output from X and the change pattern of the brightness value are considered to be related to the combination of the two light change patterns. Thereby, it is possible to associate X_A and X_B with which the above relationship is established. Since the sign information may be different between the change in brightness value corresponding to A and the change in brightness value corresponding to B, matching is performed using two candidate events: a positive event candidate group and a negative event candidate group. Good too.

A method for estimating the distance from the three-dimensional information estimation device 100 to the subject will be described using FIG. 2 described above. First, in FIG. 2, the distance between the lens 111 and the opening 112A is set to 0, and similarly the distance between the lens 111 and the opening 112B is set to 0. Let f be the focal length of the lens 111, and L be the distance between the lens 111 and the event sensor 113. Let D be the distance from the lens 111 to the focal position. Let A be the distance between the opening 112A and the opening 112B. Let the distance between the subject 115 and the lens 111 be D'.

Let L' be the distance between the lens and the plane where the light rays that have passed through the openings 112A and 112B intersect, and C be the distance between the points where the respective light rays reach the event sensor 113. Note that, as described above, since the sizes of the openings 112A and 112B are extremely small, the blurring of the points reaching the event sensor 113 is extremely small, and it is assumed that they do not overlap with each other. At this time, the following (1) holds true based on the lens imaging formula.

Furthermore, the following (2) holds true between R, C, L', and L.

From (1) and (2) above, D' can be found as shown in (3) below.

Therefore, if the pixel pitch is p and the distance on the coordinates between the points where the light rays reach the event sensor 113 is Δx, the above (3) becomes the following (4).

Using the formula shown in (4), the three-dimensional information estimation device 100 estimates the distance from the three-dimensional information estimation device 100 to the subject.

The flow of processing executed by the three-dimensional information estimating device 100 described above will be explained using a flowchart. 4 and 5 are flowcharts showing the flow of three-dimensional information estimation processing. Note that the flowcharts shown in FIGS. 4 and 5 show the flow of processing when the timing for associating event information is the timing at which new event information is stored in the event information storage unit 124.

In FIG. 4, when an event occurs (step S101: YES), the event information acquisition unit 121 acquires event information output from the imaging unit 110 (step S102). The event information acquisition unit 121 stores the acquired event information in the event information storage unit 124 (step S103). The event information acquisition unit 121 transmits an event occurrence MSG (message) indicating that an event has occurred to the corresponding unit 122 (step S104), and returns to the process of step S101.

Next, the flowchart of FIG. 5 will be explained. In addition, in the flowchart of FIG. 5, the presence of a plurality of pieces of event information that satisfies the above-described time condition is expressed as the presence of time condition satisfaction information. Furthermore, the existence of a plurality of pieces of event information that belong to the same group due to grouping based on code conditions is expressed as the presence of code condition satisfaction information.

In FIG. 5, when the handling unit 122 receives the event occurrence MSG from the event information acquisition unit 121 (step S201: YES), the handling unit 122 determines whether time condition satisfaction information exists (step S202). . If the time condition satisfaction information does not exist (step S202: NO), the correspondence unit 122 ends the process.

If time condition satisfaction information exists (step S202: YES), the correspondence unit 122 determines whether code condition satisfaction information exists (step S203). If the code condition satisfaction information does not exist (step S203: NO), the correspondence unit 122 ends the process. If code condition satisfaction information exists (step S203: YES), the correspondence unit 122 performs the above-mentioned matching (step S204). The estimation unit 123 estimates the distance from the three-dimensional information estimation device 100 to the subject using the above (4) (step S205), and ends the process.

By executing the three-dimensional information estimation process described above every time an event occurs, the distance from the three-dimensional information estimation device 100 to the subject is estimated, so that the three-dimensional information of the subject can be estimated.

In the embodiment described above, the mask section 112 having the openings 112A and 112B may be located between the lens 111 and the subject. Furthermore, when the lens 111 is composed of a plurality of lenses, the mask section 112 may be located between the lenses. Furthermore, the subject may be in front of the focal plane.

Furthermore, a laser scanning projector may be used as the illumination for illuminating the subject. Laser scanning projectors move the laser irradiation position at high speed, so even if it appears to the naked eye that a wide area is illuminated, in reality only one point is illuminated at any given time. Therefore, at any time t, there are only two pieces of event information at most, and these pieces of event information are determined as they are as an event set.

Further, an arbitrary projector may be used to irradiate the subject with discrete dot pattern illumination. By designing the arrangement of the apertures and the dot pattern so that the distance between the set of points that reach the event sensor from one of the dots through the aperture is sufficiently smaller than the distance between adjacent dots, Matching can be performed without considering the fact that the arrival points overlap. Furthermore, by switching such lighting at high speed, more event information can be obtained, so more detailed three-dimensional information can be obtained.

Further, in this embodiment, the mask section 112 has two openings 112A and 112B, but it may have three or more openings. An example of the arrangement position of the openings is a position where the centers of the openings are all on the same straight line. When there are three openings, the openings can be arranged at the vertices of the triangle.

Further, instead of the aperture, a microlens array or other optical element may be used to cause a plurality of images to reach the event sensor.

Note that when a camera equipped with a normal image sensor (hereinafter referred to as a "frame camera") is provided with an opening similar to this embodiment, two images are observed superimposed on the image sensor, so each opening It is impossible to estimate three-dimensional information unless the images are separated by making them correspond to different color channels.

On the other hand, similarly to the three-dimensional information estimation device 100 of this embodiment, a camera equipped with an event sensor (hereinafter referred to as "event camera") observes the amount of change in brightness that occurs when the subject, lighting, and event camera move. Therefore, since events occur only in a limited area, there is a low possibility that events corresponding to two images will overlap.

Furthermore, when matching is performed using the output image of a frame camera, it is basically difficult to match using luminance information of one pixel, so block matching is performed that also uses luminance information of spatially adjacent pixels. Event cameras observe changes in brightness for each pixel with very high resolution, so by using time-series data that represents the change pattern, matching can be performed without using information about adjacent pixels. Therefore, with the event sensor, even if block matching is impossible due to the occurrence positions of events corresponding to two images being located very close together, the search is easy.

For the above reasons, by providing a plurality of apertures in the event camera, three-dimensional information can be estimated using a single channel, which is not possible with a similar configuration of a frame camera.

The event processing unit 120 may be configured using a processor such as a CPU (Central Processing Unit) and a memory. In this case, the event processing unit 120 functions as the event processing unit 120 when the processor executes the program. Note that all or part of each function of the event processing unit 120 may be realized using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). . The above program may be recorded on a computer-readable recording medium. Computer-readable recording media include portable media such as flexible disks, magneto-optical disks, ROMs, CD-ROMs, semiconductor storage devices (such as SSDs: Solid State Drives), and hard disks and semiconductor storages built into computer systems. It is a storage device such as a device. The above program may be transmitted via a telecommunications line.

Although the embodiments of the present invention have been described above in detail with reference to the drawings, the specific configuration is not limited to these embodiments, and includes designs within the scope of the gist of the present invention.

The present invention is applicable to a three-dimensional information estimation device equipped with an event-based vision sensor.

DESCRIPTION OF SYMBOLS 100... Three-dimensional information estimation device, 110... Imaging unit, 111... Lens, 112... Mask unit, 112A, 112B... Opening part, 113... Event sensor, 115... Subject, 116... Focus plane, 120... Event processing unit, 121...Event information acquisition section, 122...Corresponding section, 123...Estimation section, 124...Event information storage section

Claims (6)

A 3D information estimation method in a 3D information estimation device that focuses light rays emitted from one point on a subject surface to multiple positions on an event-based vision sensor, the method comprising:
Among the pixels of the event-based vision sensor, when the amount of change in brightness value exceeds a certain value, the position of the pixel where the amount of change in brightness value exceeds the certain value and the amount of change in brightness value exceeds the certain value. an event information acquisition step of acquiring event information including the time when the event occurred;
a matching step of associating two pieces of event information among the plurality of pieces of event information acquired in the event information acquisition step based on the time included in the event information;
an estimating step of estimating a distance to the subject based on the position of a pixel included in each of the two pieces of event information associated in the corresponding step;
A three-dimensional information estimation method. The event information includes code information indicating an increase or decrease in brightness value,
2. The three-dimensional information estimation method according to claim 1, wherein the matching step matches two pieces of event information that include the same code information. 3. The three-dimensional method according to claim 1, wherein the matching step matches two pieces of event information in which the times included in the event information are the same or the difference in time included in the event information is within a predetermined range. Information estimation method. The three-dimensional information estimation method according to any one of claims 1 to 3, wherein the estimating step estimates the distance to the subject using the distance between the positions of pixels included in each of the two pieces of event information. . A three-dimensional information estimation device that focuses light rays emitted from one point on a subject surface to multiple positions on an event-based vision sensor,
Among the pixels of the event-based vision sensor, when the amount of change in brightness value exceeds a certain value, the position of the pixel where the amount of change in brightness value exceeds the certain value and the amount of change in brightness value exceeds the certain value. an event information acquisition unit that acquires event information including the time when the event occurred;
a correspondence unit that associates two pieces of event information among the plurality of pieces of event information acquired by the event information acquisition unit based on times included in the event information;
an estimating unit that estimates a distance to the subject based on the position of a pixel included in each of the two pieces of event information correlated by the corresponding unit;
A three-dimensional information estimation device equipped with A program for causing a computer to execute the three-dimensional information estimation method according to any one of claims 1 to 4.

Images