JP7243671B2 - Parallax calculation system, information processing device, information processing method and program - Google Patents

Description

The present invention relates to a parallax calculation system, an information processing apparatus, an information processing method, and a program for performing parallax calculation.

2. Description of the Related Art Conventionally, there has been known a distance measurement technique for calculating the distance to an object to be photographed by performing parallax calculation on images (stereo images) photographed by a plurality of imaging units such as stereo cameras (for example, See Patent Documents 1 to 4, etc.).

According to the distance measurement technique, the distance from the imaging unit can be calculated for all objects drawn as object images in the captured image (that is, all objects to be captured).

However, an object drawn as an object image in a captured image usually includes a plurality of objects at different distances from each other, ranging from objects near the imaging unit to objects far from the imaging unit.

For this reason, if an attempt is made to uniformly calculate parallax for all pixels in the captured image regardless of the distance to the object, incorrect parallax candidates will be extracted during parallax search, resulting in reduced accuracy of parallax calculation. can be the cause.

SUMMARY OF THE INVENTION The present invention has been made in view of the above problems, and an object of the present invention is to avoid extraction of erroneous parallax candidates and improve the accuracy of parallax calculation.

An information processing apparatus according to an embodiment of the present invention has the following configuration. i.e.
An information processing device that performs parallax calculation based on captured images captured by a plurality of imaging units,
acquisition means for acquiring distance information indicating a distance to an object in the imaging direction calculated based on reflected waves when electromagnetic waves are emitted toward the imaging direction of the plurality of imaging units;
Pixels of interest for which the parallax calculation is performed are pixels of the object image other than pixels corresponding to the irradiation range, among the pixels of the object image in the captured image in which the object irradiated with the electromagnetic wave is drawn. Determination means for determining whether or not
if the determining means does not determine that the pixel is a pixel of the object image other than the pixel corresponding to the irradiation range, maintaining the setting of the parallax search range corresponding to the reference distance as the parallax range in the parallax calculation; When the determination means determines that the pixels of the object image are pixels other than the pixels corresponding to the irradiation range, the distance to the object acquired by the acquisition means and the distance to the object acquired by the acquisition means and the The width of the parallax search range is set so that the pixels at the positions corresponding to the parallax of the captured image calculated based on the distance between the optical axes of the plurality of imaging units are included in the parallax search range. setting means for setting a start point and an end point of the parallax search range as fixed values shorter than the distance between the objects .

According to each embodiment of the present invention, it is possible to avoid extraction of erroneous parallax candidates and improve the accuracy of parallax calculation.

1 is a diagram showing the overall configuration of a parallax calculation system according to an embodiment; FIG. FIG. 10 is a diagram showing an arrangement example of a laser radar ranging unit and an imaging unit when the parallax calculation system is applied to a vehicle; FIG. 3 is a diagram showing a hardware configuration of an information processing device that configures the parallax calculation system; FIG. 3 is a diagram showing a functional configuration of a parallax image generation unit; FIG. 10 is a diagram showing parallax of predetermined pixels in an object image included in a captured image obtained by capturing an object existing at a position away from the imaging unit; FIG. 10 is a diagram showing parallax of predetermined pixels in an object image included in a captured image obtained by capturing an object existing at a position close to an imaging unit; FIG. 10 is a diagram showing a setting example of a parallax search range; FIG. 5 is a diagram showing the relationship between the distance from the imaging unit to the object and the width of the parallax search range; 4 is a flowchart showing the flow of parallax calculation processing in the parallax calculation system; 4 is a flow chart showing the flow of stereo image processing by the information processing apparatus according to the first embodiment; FIG. 10 is a diagram showing the relationship between the degree of similarity and the parallax search range at each shift position; 9 is a flow chart showing the flow of stereo image processing by the information processing apparatus according to the second embodiment; FIG. 10 is a diagram showing another setting example of the parallax search range; It is a figure which shows the example of arrangement|positioning of the laser radar range-finding part at the time of applying a parallax calculation system for FA, and an imaging part. FIG. 4 is a diagram showing parallax of predetermined pixels of an object image included in a captured image; FIG. 10 is a diagram showing a setting example of a parallax search range; FIG. 10 is a diagram showing the relationship between the degree of similarity and the parallax search range at each shift position;

Embodiments of the present invention will be described below with reference to the accompanying drawings. In the present specification and drawings, constituent elements having substantially the same functional configuration are denoted by the same reference numerals, thereby omitting redundant description.

[First Embodiment]
<1. Overall Configuration of Parallax Calculation System>
First, the overall configuration of the parallax calculation system according to this embodiment will be described. FIG. 1 is a diagram showing the overall configuration of a parallax calculation system 100 according to this embodiment. As shown in FIG. 1 , the parallax calculation system 100 includes a laser radar ranging unit 110 , a stereo imaging unit 120 and an information processing device 130 .

The laser radar ranging unit 110 detects an object based on a laser light reception signal generated by projecting and receiving a laser beam, and calculates the distance to the object irradiated with the laser beam, thereby obtaining distance information. It transmits to the information processing device 130 .

The stereo imaging unit 120 includes a monocular imaging unit 121 and a monocular imaging unit 122 . The image capturing units 121 and 122 transmit captured images to the information processing device 130 by capturing images at predetermined frame intervals.

The information processing device 130 performs parallax calculation on the captured images transmitted from the stereo imaging unit 120 to generate parallax images. Note that the information processing device 130 performs parallax calculation according to distance information transmitted from the laser radar distance measuring unit 110 when performing the parallax calculation (details will be described later).

<2. Arrangement of Laser Radar Ranging Unit and Imaging Unit Constituting Parallax Calculation System>
Next, an arrangement example of the laser radar ranging unit 110 and the imaging units 121 and 122 when the parallax calculation system 100 is applied to a vehicle will be described. FIG. 2 is a diagram showing an arrangement example of the laser radar ranging unit 110 and imaging units 121 and 122 when the parallax calculation system 100 is applied to a vehicle.

As shown in FIG. 2 , the laser radar distance measuring unit 110 and the imaging units 121 and 122 are mounted near the front windshield of the ceiling portion of the vehicle 200 and at the central position of the vehicle 200 in the width direction.

At this mounting position, laser radar ranging unit 110 emits a laser beam forward in the traveling direction of vehicle 200 . In addition, the imaging units 121 and 122 capture images of the forward direction of the vehicle 200 . In the parallax calculation system 100, the laser beam emission direction of the laser radar ranging unit 110, the optical axis of the imaging unit 121, and the optical axis of the imaging unit 122 are adjusted to be parallel to each other.

<3. Configuration of Information Processing Device>
Next, the hardware configuration of the information processing device 130 will be described. As described above, the information processing device 130 performs parallax calculation on the captured images transmitted from the stereo imaging unit 120 to generate parallax images. The parallax image generated by the information processing device 130 is used, for example, to calculate the distance to each object drawn as an object image in the captured image.

Therefore, hereinafter, in describing the hardware configuration of the information processing apparatus 130, first, a distance measurement technique (stereo distance measurement technique) for calculating the distance to each object using the parallax image generated by the information processing apparatus 130 is described. ) is briefly explained.

<3.1 Overview of Stereo Ranging Technology>
Generally, in stereo ranging technology, the correlation (similarity) is obtained for a pair of captured images transmitted from two monocular imaging units arranged on the left and right to extract the same points, and the extracted same points are extracted. By calculating the parallax for one point, the distance to the same point is calculated in the manner of triangulation.

Specifically, portions in which the same object is drawn are extracted from a pair of photographed images transmitted from two monocular imaging units. When two monocular imaging units are installed on the left and right, the position where the same object is drawn shifts left and right between a pair of captured images. Therefore, one captured image is shifted in the left-right direction with respect to the other captured image by one pixel with a predetermined range (parallax search range) as a shift range. By obtaining, the same point in the same object is extracted between a pair of captured images. Then, the shift amount (parallax) between the extracted same points is calculated, and the distance to the same point is calculated using the distance between the optical axes of the two monocular imaging units.

At this time, if the number of pixels shifted at this time is n, the focal length of the camera lens of the monocular imaging unit is f, the base line length (the distance between the optical axes of the monocular imaging unit) is B, and the pixel pitch is d, the same The distance Z to the same point in the object can be calculated by the following formula.

なお、式１の分母（ｎ×ｄ）が視差である。

Note that the denominator (n×d) of Equation 1 is parallax.

<3.2 Configuration of Information Processing Device Performing Parallax Calculation for Realizing Stereo Ranging Technology>
Next, the configuration of an information processing apparatus that performs parallax calculation for realizing the above-described stereo ranging technique will be described. FIG. 3 is a diagram showing the configuration of the information processing device 130 that performs parallax calculation for realizing the stereo ranging technique.

As shown in FIG. 3 , the information processing device 130 includes a CPU (Central Processing Unit) 301 , a RAM (Random Access Memory) 302 , a storage device 303 and an input/output unit 304 . It is assumed that each unit of the information processing device 130 is connected to each other via the bus 305 .

The CPU 301 is a computer that executes a program stored in the storage device 303 (a program for functioning as the parallax image generation unit 310). By the CPU 301 executing the program, the information processing device 130 performs parallax calculation on the photographed images transmitted from the stereo imaging unit 120 to generate parallax images.

A RAM 302 is a main storage device such as a DRAM (Dynamic Random Access Memory) or an SRAM (Static Random Access Memory). The RAM 302 functions as a work area that is expanded when the programs stored in the storage device 303 are executed by the CPU 301 .

The storage device 303 is a memory such as EPROM or EEPROM, and stores a program for causing the CPU 301 to function as the parallax image generation unit 310 .

The input/output unit 304 is an interface unit for communicating with the laser radar ranging unit 110 and the stereo imaging unit 120 (imaging units 121 and 122).

<4. Functional Configuration of Parallax Image Generation Unit>
Next, the functional configuration of the parallax image generator 310 will be described. FIG. 4 is a diagram showing the functional configuration of the parallax image generator 310. As shown in FIG.

As shown in FIG. 4 , the parallax image generation unit 310 includes a distance information acquisition unit 401 , a parallax search range selection unit 402 , a captured image acquisition unit 411 , a captured image correction unit 412 and a parallax calculation unit 420 .

A distance information acquiring unit 401 obtains a distance to an object detected by the laser radar distance measuring unit 110 by projecting and receiving laser light while the image capturing units 121 and 122 capture one frame of captured images. Information is acquired from the laser radar ranging unit 110 .

A parallax search range selection unit 402 selects a parallax search range from among a plurality of parallax search ranges having different widths, which are set when performing parallax calculation on images captured by the imaging units 121 and 122. A parallax search range is selected according to the received distance information. Also, the selected parallax search range is set in the parallax calculation unit 420 .

The captured image acquisition unit 411 acquires captured images (stereo images) captured by the imaging units 121 and 122 at predetermined frame intervals. The captured image correction unit 412 performs various correction processes such as gamma correction and distortion correction on the captured image acquired by the captured image acquisition unit 411 .

The parallax calculation unit 420 performs parallax calculation on the captured image on which various types of correction processing have been performed by the captured image correction unit 412, and generates a parallax image. It should be noted that, upon parallax calculation, parallax search is performed for the parallax search range set by the parallax search range selection unit 402 .

<5. Relationship between distance to object and parallax>
Next, the relationship between the distance to an object drawn as an object image in a photographed image and the parallax between the photographed images of predetermined pixels constituting the object image will be described.

FIG. 5 is a diagram showing parallax of predetermined pixels of an object image included in a photographed image obtained by photographing an object existing at a position away from the imaging units 121 and 122. As shown in FIG. In FIG. 5 , a captured image 510 is a captured image (reference image) captured by the imaging unit 121 , and a captured image 520 is a captured image (reference image) captured by the imaging unit 122 .

As shown in FIG. 2, the imaging unit 121 and the imaging unit 122 are installed on the left and right sides of each other. Therefore, as shown in FIG. Pixels of the same point (the license plate portion of the forward vehicle image 530) are shifted by d1 in the horizontal direction. That is, it has parallax d1.

On the other hand, FIG. 6 is a diagram showing parallax of predetermined pixels of an object image included in a captured image obtained by capturing an object existing at a position close to the imaging units 121 and 122 . In FIG. 6 , a captured image 610 is a captured image (reference image) captured by the imaging unit 121 , and a captured image 620 is a captured image (reference image) captured by the imaging unit 122 .

Similarly to FIG. 5, in FIG. 6 as well, the pixels of the same point (the license plate portion of the forward vehicle image 630) that constitute the same object image (the forward vehicle image 630 in the example of FIG. 6) are shifted in the horizontal direction by d2. . That is, it has parallax d2.

Here, the forward vehicle depicted as the forward vehicle image 630 in the captured images 610 and 620 is closer to the imaging units 121 and 122 than the forward vehicle depicted as the forward vehicle image 530 in the captured images 510 and 520. exists in Therefore, the relationship of parallax d1<parallax d2 is established.

In this way, an object drawn as an object image in a captured image has greater parallax as it is closer to the imaging units 121 and 122, and has greater parallax as it is further away from the imaging units 121 and 122. become smaller.

<6. Example of Parallax Search Range>
In this way, the parallax of each pixel forming the object image included in the captured image changes according to the distance from the imaging units 121 and 122 to the object drawn in the captured image as the object image.

Therefore, in the parallax calculation unit 420, when performing a parallax search for predetermined pixels forming the forward vehicle images 530 and 630, respectively, for the captured images 510 and 520 and the captured images 610 and 620, parallax search ranges having different widths are required. should be set. By setting a parallax search range having an appropriate width, it is possible to avoid a situation in which an erroneous parallax candidate is extracted as the same point when extracting the same point between captured images in the parallax search. It is from.

FIG. 7 shows a parallax search range set when performing a parallax search on the captured images 510 and 520 shown in FIG. 5 and a parallax search range set when performing a parallax search on the captured images 610 and 620 shown in FIG. and is a diagram showing.

In FIG. 7, the horizontal axis indicates the amount of shift between captured images for extracting the same point. Here, the shift position specified by the shift amount=d1 is such that when the photographed image 510 is shifted, the same point (the license plate portion of the front vehicle image 530) of the photographed image 510 and the photographed image 520 overlaps at that position. It is shown that. Further, the shift position specified by the shift amount=d2 is such that when the photographed image 610 is shifted, the same point of the photographed image 610 and the photographed image 620 (the license plate portion of the front vehicle image 630) overlaps at that position. is shown.

Therefore, in order to extract the same point (the license plate portion of the forward vehicle image 530) between the captured images 510 and 520, for example, a parallax search with a width of W1, where the shift start point is Da and the shift end point is Db1. It is desirable to set the range in the parallax calculator 420 . This is because the parallax d1 can be reliably extracted.

On the other hand, in order to extract the same point (the license plate portion of the forward vehicle image 630) between the captured images 610 and 620, for example, a width W2 (where W1<W2 ) is preferably set in the parallax calculation unit 420 . This is because the parallax d2 can be reliably extracted.

<7. Functions of Parallax Search Range Selection Section>
Next, the function of the parallax search range selection unit 402 that selects the parallax search range set in the parallax calculation unit 420 will be described.

FIG. 8 is a diagram for explaining the function of the parallax search range selection unit 402 that selects the parallax search range. It shows the relationship with the width of the range.

As shown in Equation 1 above, the distance Z to the object and the parallax (n×d) are in an inversely proportional relationship. It is inversely proportional to the distance to

According to FIG. 8, when the distance information acquired by the distance information acquisition unit 401 is L1, the width of the parallax search range selected by the parallax search range selection unit 402 is W1. Also, when the distance information acquired by the distance information acquisition unit 401 is L2, the width of the parallax search range selected by the parallax search range selection unit 402 is W2. Furthermore, when the distance information acquired by the distance information acquisition unit 401 is L0, the width of the parallax search range selected by the parallax search range selection unit 402 is W0.

Note that the parallax search range selection unit 402 in this embodiment selects the parallax search range at the timing when the distance information acquired by the distance information acquisition unit 401 satisfies a predetermined condition.

Specifically, a reference distance is defined in advance as a distance to an object irradiated with laser light, and a reference parallax search range is defined in advance as a parallax search range corresponding to the reference distance. Then, the parallax search range selection unit 402 determines whether or not the distance information acquired by the distance information acquisition unit 401 is different from the reference distance. Select the parallax search range of .

In the example of FIG. 8, the reference distance is L0, and while the distance information acquired by the distance information acquisition unit 401 is approximately equal to the reference distance L0, the parallax calculation unit 420 displays the width A parallax search range of W0 is set.

On the other hand, when the distance information acquired by the distance information acquisition unit 401 is shorter than the reference distance L0, the parallax search range selection unit 402 selects a width Select the parallax search range of W2.

Further, when the distance information acquired by the distance information acquisition unit 401 is longer than the reference distance L0, the parallax search range selection unit 402 selects a parallax search range narrower than the reference parallax search range (width=W0), Select a parallax search range of width=W1.

Thus, the parallax search range selection unit 402 in this embodiment is configured to operate based on comparison with the reference distance L0 and select a parallax search range having a width corresponding to the comparison result.

<8. Flow of Parallax Calculation Processing in Parallax Calculation System>
Next, the flow of parallax calculation processing in the parallax calculation system 100 will be described. FIG. 9 is a flow chart showing the flow of parallax calculation processing in the parallax calculation system 100 .

When the parallax calculation process by the parallax calculation system 100 is started, in step S901, based on an instruction from the information processing device 130, the image pickup units 121 and 122 perform photographing processing in synchronization with each other. Furthermore, in step S902, the laser radar ranging unit 110 executes laser ranging processing.

Note that the imaging process in step S901 and the laser ranging process in step S902 are executed in parallel. In other words, while the imaging units 121 and 122 are capturing the captured image for one frame, the laser radar distance measuring unit 110 projects and receives pulsed laser light.

In step S903, the information processing apparatus 130 performs parallax calculation based on the captured image for one frame, and executes stereo image processing to generate a parallax image. At this time, the information processing device 130 selects the parallax search range based on the distance information calculated in the laser ranging process executed in parallel with the shooting process of the frame.

In step S904, it is determined whether or not an instruction to end the parallax calculation process has been input, and if it is determined that the instruction has not been input, the process returns to steps S901 and S902. Then, the imaging process is executed for the next frame, and the laser ranging process is executed by projecting and receiving the next pulsed laser beam.

After that, until an instruction to end the parallax calculation process is input, the imaging process is executed in units of frames, and the stereo image processing is executed in units of frames while executing the laser ranging process for each frame.

On the other hand, if it is determined in step S904 that an instruction to end the parallax calculation process has been input, the process ends.

<9. Stereo image processing by information processing device>
Next, a detailed flow of stereo image processing (step S903) by the information processing device 130 will be described with reference to FIGS. 10 and 11. FIG. FIG. 10 is a flowchart showing a detailed flow of stereo image processing (step S903) by the information processing device 130. As shown in FIG. FIG. 11 is a diagram showing the relationship between the degree of similarity at each shift position calculated for extracting the same point for each pixel between the captured images and the parallax search range. It should be noted that the vertical axis shown in FIG. 11 is normalized so that the higher the degree of similarity, the smaller the value, and the lower the degree of similarity, the larger the value.

In FIG. 10, in step S1001, the captured image acquisition unit 411 acquires captured images for one frame from the imaging units 121 and 122, respectively.

In step S1002, the captured image correction unit 412 performs correction processing such as gamma correction and distortion correction on the captured image for one frame (a pair of captured images (reference image and standard image)) acquired in step S1001. .

In step S1003, the parallax calculation unit 420 sets a pixel of interest Pn to be subjected to parallax calculation among the pixels included in the reference image.

In step S1004, a parallax calculation is performed to determine whether or not the target pixel Pn set in step S1003 is a pixel in the vicinity of a pixel in the reference image (referred to as an irradiation range pixel) corresponding to the irradiation range irradiated with the laser beam. Unit 420 determines.

Here, the pixels near the irradiation range pixels refer to pixels other than the irradiation range pixels, among the pixels forming the object image in the reference image in which the object irradiated with the laser light is drawn.

Note that the coordinates of the irradiation range pixels in the reference image are set in advance, and in step S1004, the target pixel Pn set in step S1003 is a pixel within a predetermined range near the set pixel. Whether or not there is is determined based on the coordinates.

If the pixel of interest Pn is determined to be a pixel in the vicinity of the irradiation range pixel, the process advances to step S1005.

In step S1005, the distance information acquisition unit 401 acquires distance information. In step S1006, the parallax search range selection unit 402 determines whether the distance information acquired in step S1006 is substantially equal to the reference distance L0. If it is determined in step S1006 that the distance is substantially equal to the reference distance L0, the setting of the reference parallax search range (width W0) in the parallax calculator 420 is maintained.

On the other hand, if it is determined in step S1006 that the distance is different from the reference distance L0, the process proceeds to step S1007. In step S1007, the parallax search range selection unit 402 determines whether the distance information acquired in step S1005 is shorter than the reference distance L0.

If it is determined in step S1007 that the distance is shorter than the reference distance L0, the process advances to step S1008 to select a parallax search range with a width W2 and set it in the parallax calculation unit 420 as the parallax search range.

On the other hand, if it is determined in step S1007 that the distance is longer than the reference distance L0, the process advances to step S1009 to select a parallax search range with a width W1 as the parallax search range and set it in the parallax calculation unit 420. FIG.

Here, the effect of changing the parallax search range in step S1008 or S1009 will be described with reference to FIG. Conventionally, the parallax search range is fixed regardless of the distance to the object (see parallax search range 1110 in FIG. 11B). Therefore, as shown in FIG. 11(b), when a plurality of local minimum points exist (a plurality of parallax candidates exist), the parallax candidate to be extracted (the parallax candidate in FIG. 11(b) There is a possibility that a parallax candidate (parallax candidate 2 in FIG. 11B) different from 1) is extracted.

In contrast, in this embodiment, the parallax search range selection unit 402 sets a parallax search range having a width corresponding to the comparison result based on the distance information (see parallax search range 1100 in FIG. 11A). Therefore, it is possible to avoid a situation in which a parallax candidate (parallax candidate 2 in FIG. 11A) different from the parallax candidate to be extracted (parallax candidate 1 in FIG. 11A) is extracted.

Return to FIG. If it is determined in step S1004 that the target pixel Pn is not a pixel near the irradiation range pixel, the process proceeds to step S1010.

In step S1010, the setting of the reference parallax search range (width W0) in parallax calculation section 420 is maintained.

In step S1011, parallax calculation unit 420 performs parallax search using the parallax search range set in steps S1008 to S1010, and calculates the parallax of target pixel Pn based on the extracted parallax candidate.

In step S1012, the parallax calculation unit 420 determines whether or not parallax calculation has been performed for all the pixels in the captured image for one frame acquired in step S1001. If it is determined in step S1009 that there is a pixel for which parallax calculation has not been performed, the process proceeds to step S1013. After the counter n is incremented in step S1013, the process returns to step S1003 to set the next pixel as the pixel of interest Pn, and the processes from step S1004 to step S1011 are executed.

On the other hand, if it is determined in step S1012 that parallax calculation has been performed for all pixels, the process advances to step S1014 to generate a parallax image for the captured image.

<10. Summary>
As is clear from the above description, in the parallax calculation system 100 according to this embodiment,
A configuration in which the laser radar distance measurement unit 110 emits laser light forward in the traveling direction of the vehicle, which is the shooting direction of the stereo imaging unit, and calculates distance information indicating the distance to the object irradiated with the laser light. and
When the distance information calculated by projecting and receiving the laser light is different from the reference distance, a parallax search range having a width different from the preset reference parallax search range is selected.

As a result, among the objects drawn as object images in the captured image, the parallax search range is set separately for objects existing at a position close to the imaging unit and objects existing at a distant position. becomes possible. As a result, it is possible to avoid a situation in which an erroneous parallax candidate is extracted, and it is possible to improve the accuracy of parallax calculation.

[Second embodiment]
In the first embodiment, the reference distance and the reference parallax search range are defined, and the parallax search range having a width corresponding to the comparison result between the acquired distance information and the reference distance is selected. The invention is not so limited.

For example, the graph shown in FIG. 8 may be made into a table or function, and the width of the parallax search range corresponding to the acquired distance information may be derived by referring to the table or function. According to this configuration, the parallax search range selection unit 402 operates each time distance information is acquired, and it is possible to set a parallax search range having a width corresponding to the acquired distance information.

The flow of stereo image processing by the information processing apparatus 130 according to this embodiment will be described below.

FIG. 12 is a flow chart showing the flow of stereo image processing by the information processing apparatus 130 according to this embodiment. Note that the processing of steps S1001 to S1005 and S1010 to S1014 is the same as the processing of steps S1001 to S1005 and S1010 to S1014 of the stereo image processing described with reference to FIG. 10, so description thereof is omitted here. do.

In step S<b>1201 , the parallax search range selection unit 402 derives a parallax search range having a width corresponding to the distance information acquired in step S<b>1005 and sets it in the parallax calculation unit 420 .

Accordingly, in step S1011, parallax search can be performed using a parallax search range having a width corresponding to the distance information.

[Third embodiment]
In the above first and second embodiments, when changing the width of the parallax search range, the starting point of the parallax search is fixed and the end point of the parallax search range is changed, but the present invention is limited to this. not.

For example, as shown in FIG. 13A, the configuration may be such that the end point (Db) of the parallax search range is fixed and the start point (Da) of the parallax search range is changed.

Alternatively, as shown in FIG. 13B, the width of the parallax search range, the start point (Da), and the end point (Db) of the parallax search range may all be changed.

[Fourth embodiment]
In the first to third embodiments described above, the case where the parallax calculation system 100 is applied to a vehicle has been described, but the present invention is not limited to this. For example, it may be applied to FA (Factory Automation) and used to calculate a distance to a picking target when picking a product or the like to be transported in a production factory. Details of the present embodiment will be described below.

<1. Arrangement of Laser Radar Ranging Unit and Imaging Unit Constituting Parallax Calculation System>
First, an arrangement example of the laser radar ranging unit 110 and the imaging units 121 and 122 when the parallax calculation system 100 according to the present embodiment is applied to FA will be described. FIG. 14 is a diagram showing an arrangement example of the laser radar ranging unit 110 and imaging units 121 and 122 when the parallax calculation system 100 is applied for FA.

As shown in FIG. 14, the laser radar ranging unit 110 is arranged between the imaging units 121 and 122, and is arranged above a belt conveyor 1410 on which a product (for example, a metal pipe) 1400 is conveyed. . It is assumed that the photographing ranges of the imaging units 121 and 122 are adjusted so that the product 1400 conveyed by the belt conveyor 1410 is photographed. It is also assumed that the laser radar ranging unit 110 is adjusted so that the product 1400 is irradiated with laser light.

<2. Relationship Between Object Arrangement and Parallax, and Method for Setting Parallax Search Range>
Next, the relationship between the placement of the object (product 1400) rendered as the object image in the captured image and the parallax, and the method of setting the parallax search range will be described. FIG. 15 shows a captured image (reference image) 1510 captured by the imaging unit 121 and a captured image (reference image) 1520 captured by the imaging unit 122 .

In FIG. 15, it is assumed that the object image (product image 1411) drawn in the captured image 1510 corresponds to the object image (product image 1421) drawn in the captured image 1520. FIG. It is also assumed that the object image (product image 1412) drawn in the captured image 1510 corresponds to the object image (product image 1422) drawn in the captured image 1520. FIG.

That is, in the example of FIG. 15, the point 1511 in the product image 1411 and the point 1521 in the product image 1421 are the same point. Also, the point 1512 in the product image 1412 and the point 1522 in the product image 1422 are the same point. In other words, pixels at the same points in the captured image 1510 and the captured image 1520 are shifted by d2 in the transport direction of the product 1400 (has parallax d2).

Here, the products 1400 on the belt conveyor 1410 have the same material and the same shape, and are continuously conveyed. Therefore, as shown in FIG. 15, for example, a product image 1421 of the same material and shape is drawn on the left side of the product image 1422 drawn in the photographed image 1520 .

Therefore, when the parallax d2 is large, for example, when searching for a point 1522 that is the same point as the point 1512, if the starting point of the parallax search range is set to a shift position with a shift amount of zero, the point 1521 is erroneously identified as the same point. will be extracted.

In this embodiment, in order to avoid such a situation, the process of setting the parallax search range based on the distance information indicating the distance to the product 1400 acquired by the distance information acquisition unit 401 is performed according to the following procedure. bottom.
1) Using the distance information Z acquired by the distance information acquisition unit 401, the base line length (the distance between the optical axes of the imaging units 121 and 122) B, and the focal length f of the camera lens of the imaging units 121 and 122, A parallax (a shift amount of the same point) is calculated by Equation (1).
2) Set the width of the parallax search range to a fixed value smaller than the distance between adjacent products.
3) Determine the start point and end point of the parallax search range so as to have the width set in 2) above and include the pixel at the shift position specified by the shift amount calculated in 1) above.

FIG. 16(a) is a diagram showing a parallax search range 1600 set according to the above procedure. As shown in FIG. 16A, d2 is calculated as the parallax of the point 1522 based on the distance information, and W is set as the width of the parallax search range, thereby determining the start point Da2 and the end point Db2. . This allows the point 1521 of the adjacent product image 1421 to be out of the parallax search range.

FIG. 17A is a diagram showing how parallax search is performed using the parallax search range 1600 shown in FIG. ing.

In the example of FIG. 17A, since the shift positions corresponding to points 1521 and 1522 are local minimum points, there are two parallax candidates (parallax candidate 1 and parallax candidate 2), but a parallax search range 1600 is set. By doing so, only parallax candidate 2 can be extracted.

On the other hand, in FIG. 16B, for comparison with the above processing, the starting point of the parallax search range is fixed, and a parallax search range having a width corresponding to the distance information acquired by the distance information acquisition unit 401 is selected. FIG. 10 is a diagram showing a parallax search range in this case;

As shown in FIG. 16B, if the width of the parallax search range is set to W2 while the start point Da1 of the parallax search range is fixed, the end point of the parallax search range becomes Db1, and a point Db1 is set within the parallax search range. Both 1521 and point 1522 will be included.

FIG. 17B is a diagram showing how parallax search is performed using the parallax search range 1610 shown in FIG. ing.

As shown in FIG. 17B, when a parallax search range having a width corresponding to the distance information is selected, two parallax candidates (parallax candidate 1 and parallax candidate 2) exist in the parallax search range 1610. It can be seen that candidate 1 (point 1521) may be erroneously extracted.

<3. Summary>
As is clear from the above description, in the parallax calculation system 100 according to this embodiment,
A shift position for parallax search is calculated by back-calculating the parallax between the captured images based on the distance information acquired by projecting and receiving the laser light.
The start point and end point of the parallax search range are determined so that the pixels at the calculated shift positions are included in the parallax search range whose width is fixed.

This makes it possible to avoid extracting an erroneous parallax candidate even when parallax calculation is performed on captured images in which objects having the same material and the same shape are arranged adjacent to each other. It becomes possible. As a result, it is possible to improve the accuracy of parallax calculation.

[Fifth embodiment]
In each of the above-described embodiments, no particular reference was made to the detailed method of parallax calculation, but any method can be used as the method of parallax calculation.

For example, a method such as EBM (Edge-Based Matching) that calculates parallax for portions with strong textures (portions with strong edges) may be used. Alternatively, a method of calculating the parallax of each pixel in the entire photographed image by using a sub-pixel estimation method (fitting by equiangular straight lines, parabolas, high-order polynomials, etc.) may be used. In addition, the sub-pixel estimation method propagates the matching index value from around each pixel by a recurrence formula like SGM (Semi Global Matching), and from the integer disparity that gives the minimum index value and the disparity index value adjacent to it This is a method of calculating decimal parallax.

Further, in each of the above-described embodiments, the distance information is calculated based on the reflected light when the laser beam is emitted from the laser radar distance measuring unit 110, but the present invention is not limited to this. The distance information may be calculated based on the reflected wave when an electromagnetic wave is emitted from an electromagnetic wave ranging unit other than the laser radar ranging unit.

It should be noted that the present invention is not limited to the configurations shown here, such as combinations with other elements, to the configurations listed in the above embodiments. These points can be changed without departing from the gist of the present invention, and can be determined appropriately according to the application form.

100: parallax calculation system 110: laser radar ranging unit 120: stereo imaging units 121 and 122: imaging unit 130: information processing device 310: parallax image generation unit 401: distance information acquisition unit 402: parallax search range selection unit 403: photography Image Acquisition Unit 404: Captured Image Correction Unit 405: Parallax Calculation Unit

Japanese Patent Application Laid-Open No. 2003-346130 International Publication No. 2011-096251 JP 2013-114477 A JP 2013-174494 A

Claims (4)

An information processing device that performs parallax calculation based on captured images captured by a plurality of imaging units,
acquisition means for acquiring distance information indicating a distance to an object in the imaging direction calculated based on reflected waves when electromagnetic waves are emitted toward the imaging direction of the plurality of imaging units;
Pixels of interest for which the parallax calculation is performed are pixels of the object image other than pixels corresponding to the irradiation range, among the pixels of the object image in the captured image in which the object irradiated with the electromagnetic wave is drawn. Determination means for determining whether or not
if the determining means does not determine that the pixel is a pixel of the object image other than the pixel corresponding to the irradiation range, maintaining the setting of the parallax search range corresponding to the reference distance as the parallax range in the parallax calculation; When the determination means determines that the pixels of the object image are pixels other than the pixels corresponding to the irradiation range, the distance to the object acquired by the acquisition means and the distance to the object acquired by the acquisition means and the The width of the parallax search range is set so that the pixels at the positions corresponding to the parallax of the captured image calculated based on the distance between the optical axes of the plurality of imaging units are included in the parallax search range. and setting means for setting a start point and an end point of the parallax search range as fixed values shorter than a distance between objects . A parallax calculation system having a plurality of imaging units, an electromagnetic wave ranging unit, and an information processing device that performs parallax calculation based on images captured by the plurality of imaging units,
acquisition means for acquiring distance information indicating a distance to an object in the imaging direction calculated based on reflected waves when electromagnetic waves are emitted toward the imaging direction of the plurality of imaging units;
Pixels of interest for which the parallax calculation is performed are pixels of the object image other than pixels corresponding to the irradiation range, among the pixels of the object image in the captured image in which the object irradiated with the electromagnetic wave is drawn. Determination means for determining whether or not
if the determining means does not determine that the pixel is a pixel of the object image other than the pixel corresponding to the irradiation range, maintaining the setting of the parallax search range corresponding to the reference distance as the parallax range in the parallax calculation; When the determination means determines that the pixels of the object image are pixels other than the pixels corresponding to the irradiation range, the distance to the object acquired by the acquisition means and the distance to the object acquired by the acquisition means and the The width of the parallax search range is set so that the pixels at the positions corresponding to the parallax of the captured image calculated based on the distance between the optical axes of the plurality of imaging units are included in the parallax search range. and setting means for setting a start point and an end point of the parallax search range as fixed values shorter than a distance between objects . An information processing method in an information processing device that performs parallax calculation based on captured images captured by a plurality of imaging units,
an obtaining step of obtaining distance information indicating a distance to an object in the imaging direction calculated based on reflected waves when electromagnetic waves are emitted toward the imaging direction of the plurality of imaging units;
Pixels of interest for which the parallax calculation is performed are pixels of the object image other than pixels corresponding to the irradiation range, among the pixels of the object image in the captured image in which the object irradiated with the electromagnetic wave is drawn. A determination step of determining whether or not
if it is not determined in the determination step that the pixels of the object image are pixels other than the pixels corresponding to the irradiation range, maintaining the setting of the parallax search range corresponding to the reference distance as the parallax range in the parallax calculation; When it is determined in the determination step that the pixels of the object image are other than the pixels corresponding to the irradiation range, the parallax range in the parallax calculation is the distance to the object acquired in the acquisition step, and the The width of the parallax search range is set so that the pixels at the positions corresponding to the parallax of the captured image calculated based on the distance between the optical axes of the plurality of imaging units are included in the parallax search range. and a setting step of setting a start point and an end point of the parallax search range as fixed values shorter than a distance between objects . In the computer of the information processing device that performs parallax calculation based on the captured images captured by a plurality of imaging units,
an obtaining step of obtaining distance information indicating a distance to an object in the imaging direction calculated based on reflected waves when electromagnetic waves are emitted toward the imaging direction of the plurality of imaging units;
Pixels of interest for which the parallax calculation is performed are pixels of the object image other than pixels corresponding to the irradiation range, among the pixels of the object image in the captured image in which the object irradiated with the electromagnetic wave is drawn. A determination step of determining whether or not
if it is not determined in the determination step that the pixels of the object image are pixels other than the pixels corresponding to the irradiation range, maintaining the setting of the parallax search range corresponding to the reference distance as the parallax range in the parallax calculation; When it is determined in the determination step that the pixels of the object image are other than the pixels corresponding to the irradiation range, the parallax range in the parallax calculation is the distance to the object acquired in the acquisition step, and the The width of the parallax search range is set so that the pixels at the positions corresponding to the parallax of the captured image calculated based on the distance between the optical axes of the plurality of imaging units are included in the parallax search range. and a setting step of setting a start point and an end point of the parallax search range as fixed values shorter than the distance between the objects .

Images