WO2023017635A1

WO2023017635A1 - Computing device, monitoring system and parallax calculation method

Info

Publication number: WO2023017635A1
Application number: PCT/JP2022/011116
Authority: WO
Inventors: 圭介稲田; 雅士高田; 進一野中
Original assignee: 日立Astemo株式会社
Priority date: 2021-08-11
Filing date: 2022-03-11
Publication date: 2023-02-16
Also published as: DE112022001667T5; JP2023025976A

Abstract

This computing device is connected to a first imaging unit and a second imaging unit, and generates parallax information between a first image captured by the first imaging unit and a second image captured by the second imaging unit by searching in prescribed matching block units, said computing device being equipped with: a similarity generation unit for generating a first degree of similarity of a matching block within a search range of a second image relative to a matching block in the first image in matching block units; a similarity correction unit for generating a second degree of similarity by correcting the first degree of similarity by using first correction information which is generated on the basis of the first degree of similarity in a desired adjacent matching block within the search range of the second image, and/or second correction information generated on the basis of parallax information; and a similarity determination unit for generating parallax information on the basis of the second degree of similarity.

Description

Arithmetic device, monitoring system, parallax calculation method

The present invention relates to an arithmetic device, a monitoring system, and a parallax calculation method.

A known method is to use a stereo camera to calculate the distance between the camera and the object to be observed, and perform recognition processing of the object to be observed. In Patent Literature 1, among left and right captured images captured by a stereo camera that captures images of the same space from left and right viewpoints, each of a plurality of regions obtained by dividing one image is used as a reference block, and each reference block is: Similarity data for generating similarity data by setting a search range in the other image and calculating the similarity of the image with the reference block with respect to a position within the search range by associating the position within the search range with the position within the search range. a generation unit, a similarity correction unit that smoothes the similarity data in the spatial direction based on the similarity data generated for a predetermined number of reference blocks surrounding the corresponding reference block, and each smoothed similarity A result evaluation unit that detects a position where the degree of similarity is maximum in the degree data, and a parallax for each reference block is obtained using the detection result of the result evaluation unit, and based on this, the position of the subject in the depth direction is calculated and an image is obtained. a depth image generation unit that generates a depth image by associating it with a plane; an output information generation unit that performs predetermined information processing based on the position of a subject in a three-dimensional space using the depth image and outputs the result; An information processing apparatus characterized by comprising:

Japanese Patent Application Laid-Open No. 2016-039618

In the invention described in Patent Document 1, there is room for improvement in calculating parallax.

An arithmetic device according to a first aspect of the present invention is connected to a first imaging unit and a second imaging unit, and is configured to provide a first image captured by the first imaging unit and a second image captured by the second imaging unit. An arithmetic unit for generating parallax information with an image by searching for each matching block unit, wherein matching blocks within a search range of the second image for matching blocks within the first image are generated for each matching block unit. a similarity generating unit that generates a first similarity with the first correction information generated based on the first similarity of any adjacent matching block within the search range of the second image, and the parallax a similarity correction unit that corrects the first similarity using at least one of second correction information generated based on information to generate a second similarity; and disparity information based on the second similarity. and a similarity determination unit that generates the similarity determination unit.
A monitoring system according to a second aspect of the present invention includes the arithmetic device described above, the first imaging section, and the second imaging section.
A parallax calculation method according to a third aspect of the present invention is connected to a first imaging unit and a second imaging unit, and provides a first image captured by the first imaging unit and a second image captured by the second imaging unit. A parallax calculation method executed by an arithmetic device that generates by searching for parallax information between two images in units of predetermined matching blocks, wherein the second image with respect to matching blocks in the first image is calculated in units of matching blocks. and a first correction information generated based on the first similarity of any adjacent matching block within the search range of the second image. , and second correction information generated based on the parallax information, generating a second similarity by correcting the first similarity using at least one of the parallax information based on the second similarity and generating a

According to the present invention, matching errors can be reduced and the accuracy of parallax calculation can be improved.

Configuration diagram of a monitoring system including an arithmetic unit Configuration diagram of the similarity correction unit Diagram for explaining first similarity information and first correction information A diagram showing a first example of parallax information used to generate second correction information. A diagram showing a second example of parallax information used to generate the second correction information. Diagram explaining the correction factor A diagram for explaining a specific example of a correction coefficient generation method. Configuration diagram of a correction execution unit in the second embodiment Configuration diagram of a similarity correction unit in the third embodiment

-First Embodiment-
A first embodiment of the monitoring system will be described below with reference to FIGS. 1 to 7. FIG.

FIG. 1 is a configuration diagram of a monitoring system S including an arithmetic device 1. FIG. A vehicle surroundings monitoring system S includes an arithmetic device 1 , a first imaging unit 2 , a second imaging unit 3 , a recognition processing unit 4 , and a vehicle control unit 5 . A vehicle surroundings monitoring system S is mounted on a vehicle C. As shown in FIG. Arithmetic device 1 is connected to first imaging unit 2 , second imaging unit 3 , and recognition processing unit 4 . The recognition processing section 4 is connected to the arithmetic device 1 and the vehicle control section 5 .

Each of the arithmetic device 1, the recognition processing unit 4, and the vehicle control unit 5 is an ECU (Electronic Control Unit) mounted on the vehicle C. However, the arithmetic unit 1, the recognition processing unit 4, and the vehicle control unit 5 may not be individual ECUs, and two or more may be realized by the same ECU.

Both the first imaging unit 2 and the second imaging unit 3 are cameras, and output captured images, which are images obtained by capturing, to the arithmetic device 1 . The captured image output by the first imaging unit 2 is called a first captured image 100 , and the captured image output by the second imaging unit 3 is called a second captured image 101 . The first imaging unit 2 and the second imaging unit 3 face the vehicle C in the same direction and are arranged laterally apart from each other by a predetermined baseline length. Note that the horizontal direction in which the first imaging unit 2 and the second imaging unit 3 are arranged is hereinafter also referred to as the “base line direction”. Based on the first captured image 100 and the second captured image 101 , the computing device 1 generates the parallax information 106 by a method described later and outputs the parallax information 106 to the recognition processing section 4 .

The computing device 1 includes an input unit 10 , a similarity generation unit 11 , a similarity correction unit 12 and a similarity determination unit 13 . The input unit 10 is implemented by a hardware interface with the first imaging unit 2 and the second imaging unit 3, and an arithmetic processing unit. The similarity generation unit 11, the similarity correction unit 12, and the similarity determination unit 13 are realized by an arithmetic processing device. The hardware interface is, for example, RJ45 or camera link.

An arithmetic processing unit includes, for example, a CPU that is a central processing unit, a ROM that is a read-only storage device, and a RAM that is a readable/writable storage device. By doing so, the calculation described later is performed. The arithmetic processing unit may be realized by FPGA (Field Programmable Gate Array), which is a rewritable logic circuit, or ASIC (Application Specific Integrated Circuit), which is an application specific integrated circuit, instead of the combination of CPU, ROM, and RAM. good. Also, the arithmetic processing unit may be realized by a combination of different configurations, for example, a combination of CPU, ROM, RAM and FPGA, instead of the combination of CPU, ROM, and RAM.

The input unit 10 supplies a first image 102 obtained by performing image processing on the first captured image 100 and a second image 103 obtained by performing image processing on the second captured image 101 to the similarity generating unit 11. . This image processing is performed using, for example, the intrinsic and extrinsic parameters of the camera to eliminate adverse effects on the image processing. Examples of image processing include correction of lens distortion and correction of mounting position and mounting orientation errors. However, since the processing in the input unit 10 is not essential in the present embodiment, the first captured image 100 and the first image 102 may be regarded as the same, or the second captured image 1021 and the second image 103 may be regarded as the same.

The similarity generation unit 11 receives the first image 102 and the second image 103 supplied from the input unit 10 , generates first similarity information 104 , and supplies the first similarity information 104 to the similarity correction unit 12 . The similarity correction unit 12 is information representing the similarity between matching blocks in the first image 102 and matching blocks in the second image 103 . The first similarity information 104 is, for example, the amount of luminance variation between each pixel in the matching block of the first image 102 and the corresponding pixel in the matching block of the second image 103 . Luminance variation is calculated by SAD (Sum of Absolute Difference), ZSAD (Zero means Sum of Absolute Difference), SSD (Sum of Squared Difference), NCC (Normalized Cross Correlation), ZNCC (Zero means Normalized Cross Correlation), etc. method can be used.

The first similarity information 104 may be the luminance fluctuation amount, luminance fluctuation direction, or luminance fluctuation pattern between each pixel in the matching block of the first image 102 and the second image 103 and its surrounding pixels. As an example of the luminance variation pattern, there is an increase/decrease pattern from the left for three horizontal pixels including pixels adjacent to the left and right of the target pixel. For example, when the left adjacent pixel value is 10, the target pixel value is 25, and the right adjacent pixel value is 80, the increase/decrease pattern is "increase→increase".

The similarity correction unit 12 generates second similarity information 105 by correcting the first similarity information 104 based on the first similarity information 104 or the parallax information 106 . Based on the second similarity information 105, the similarity determination unit 13 detects the position with the highest similarity within the search range, and generates parallax information 106 based on this position information. The similarity determination unit 13 outputs the generated parallax information 106 to the recognition processing unit 4 and the similarity correction unit 12 .

The recognition processing unit 4 receives the parallax information 106 and performs various recognition processes. An example of recognition processing in the recognition processing unit 4 is three-dimensional object detection using the parallax information 106 . Examples of objects to be recognized by the recognition processing unit 4 include subject position information, type information, motion information, and risk information. An example of position information is the direction and distance from the own vehicle. Examples of type information include pedestrians, adults, children, elderly people, animals, falling rocks, bicycles, surrounding vehicles, surrounding structures, and curbs. Examples of motion information include swaying, jumping, crossing, moving direction, moving speed, and moving trajectory of a pedestrian or a bicycle. Examples of danger information include pedestrians running out, falling rocks, and abnormal behavior of surrounding vehicles such as sudden stops, sudden deceleration, and sudden steering. The recognition information 107 generated by the recognition processing section 4 is supplied to the vehicle control section 5 .

The vehicle control unit 5 performs various vehicle controls for the vehicle C based on the recognition information 107. Examples of vehicle control performed by the vehicle control unit 5 include brake control, steering wheel control, accelerator control, in-vehicle lamp control, warning sound generation, in-vehicle camera control, peripheral vehicles connected via a network, and remote center equipment. output of information related to objects to be observed around the imaging device. A specific example is speed and brake control according to the parallax information 106 of obstacles present in front of the vehicle.

Note that distance information may be generated based on the parallax information and used instead of the parallax information 106. Further, although not described in this embodiment, the vehicle control unit 5 may perform subject detection processing based on image processing results using the first image 102 and the second image 103 . In addition, the vehicle control unit 5 displays an image obtained through the first imaging unit 2 or the second imaging unit 3 and a display for the viewer to recognize on a display device connected to the vehicle control unit 5. you can go Further, the vehicle control unit 5 may supply the information of the observation object detected based on the image processing result to an information device that processes traffic information such as map information and traffic information.

FIG. 2 is a configuration diagram of the similarity correction unit 12. As shown in FIG. The similarity correction unit 12 includes a first correction information generation unit 20 , a second correction information generation unit 21 and a correction execution unit 22 . The first correction information generator 20 generates first correction information 200 based on the first similarity information 104 . The second correction information generator 21 generates second correction information 201 based on the parallax information 106 . The correction execution unit 22 corrects the first similarity information 104 based on the first correction information 200 and the second correction information 201 to generate the second similarity information 105 .

The configuration of the first correction information generation unit 20 will be explained. The first correction information generation section 20 includes a correction determination section 40 and a first generation section 41 . Based on the first similarity information 104 , the correction determination unit 40 generates correction effective information 400 indicating whether or not to generate the first correction information 200 . For example, the correction determination unit 40 determines that generation of the first correction information 200 is necessary when the following first determination condition is satisfied, and generation of the first correction information 200 is unnecessary when the first determination condition is not satisfied. I judge.

(Similarity S(i-1)-Similarity S(i))>TH0 and (Similarity S(i+1)-Similarity S(i))>TH1

Note that the similarity A(i) is the similarity at the search position A(i) to be determined within the search range, and the similarity A(i-1) is the search position A adjacent to the left of the search position A(i). The similarity at (i−1), the similarity A(i+1), is the similarity at the search position A(i+1) adjacent to the right of the search position A(i). TH0 and TH1 are predetermined threshold values. Furthermore, the size determination in the first determination condition may be changed so that the condition is satisfied when the sizes are equal.

The reason for referring to the left and right instead of the top and bottom in the first determination condition, that is, the reason for scanning in the horizontal direction is that the baseline direction is horizontal. Further, the first determination condition may be determined using not only 3 similarities but also 5 similarities as follows. However, TH2 to TH5, which are predetermined threshold values, are used here.

(Similarity S(i-2)-Similarity S(i-1)) > TH2 AND (Similarity S(i-1)-Similarity S(i)) > TH3 AND (Similarity S(i+1)- Similarity S(i))>TH4 and (Similarity S(i+2)-Similarity S(i+1))>TH5

The first generation unit 41 receives the first similarity information 104 and the correction effectiveness information 400 and generates the first correction information 200 based on the correction effectiveness information 400 . The first correction information 200 is, for example, a combination of a search position to be corrected and corrected similarity information, or a combination of a newly generated search position and similarity information. The correction execution unit 22 refers to the first correction information 200 and corrects the similarity. The operation of the first correction information generator 20 will be described with reference to FIG.

FIG. 3 is a diagram for explaining the first similarity information 104 and the first correction information 200. FIG. The upper portion of FIG. 3 shows the entire similarity curve 1010 and the lower portion of FIG. 3 shows details of the location portion of the similarity curve 1010 . Specifically, a similarity curve 1012 shown in the lower part of FIG. 3 indicates the range of reference numeral 1011 in the upper part of FIG. In the similarity curve 1010 and the similarity curve 1012, the horizontal axis indicates the search position in the search range in the horizontal direction, the vertical axis indicates the similarity, and the lower the vertical axis, the higher the similarity.

The black circles described in the similarity curve 1012 shown at the bottom of FIG. 3 indicate the similarity at each search position. Since the similarity generation unit 11 performs a search in units of predetermined pixels, the similarity in the similarity curve 1012 is indicated by the interval of the search in units of pixels. The pixel unit for search can be set arbitrarily, but the smallest unit is one pixel. The search positions in the central part of the drawing are 1003, 1001, 1000, 1002, and 1004 from the left.

The similarity S0 at the search position 1000 on the similarity curve 1012 is higher than the similarity at the left and right search positions. Therefore, the search position 1000 satisfies the first determination condition described above. As an example of a method of correcting the similarity curve 1012, there is a method of replacing the similarity S0 at the search position 1000 with the similarity S1. As an example of another method, there is a method of generating a new similarity S1 at search positions around the search position 1000, for example, search position 1003. FIG.

The search position 1003 having the similarity S1 within the similarity curve 1012 is an example of the first correction information 200 . As an example of a method of generating the similarity S1, there is a method of calculating by polynomial interpolation such as linear approximation or SSD parabolic fitting based on a plurality of neighboring similarities. As an example of multiple degrees of similarity existing in the vicinity, there are three degrees of similarity of

search positions

1000, 1001, and 1002. FIG. A search position 1003 is a decimal precision search position located between the search position 1000 and the search position 1002 .

In this way, the first correction information generation unit 20 scans in the baseline direction to search for an inflection point where the difference in similarity switches between positive and negative, calculates a new similarity using the similarity around the inflection point, A combination of the position of the inflection point and the new degree of similarity is calculated as the first correction information 200 .

When the search position 1003 and the similarity S1 are included in the first correction information 200, there are at least the following three methods for the correction execution unit 22 to correct the similarity using the first correction information 200. A first method is a method of rewriting the information on the search position 1000 and the similarity S0 with the information on the search position 1003 and the similarity S1. A second method is to leave the information of the search position 1000 and the similarity S0 as they are, and add the information of the search position 1003 and the similarity S1. A third method is a method of rewriting the value of the degree of similarity S0 to the value of S1 without changing the position of the search position 1000 . When adopting the third method, the information of the search position 1003 may be calculated by 3-point approximation or the like.

Returning to FIG. 2, the configuration of the second correction information generation unit 21 will be described. The second correction information generation unit 21 includes a peripheral parallax continuity determination unit 50 and a second generation unit 51 , and generates second correction information 301 based on the parallax information 106 or the parallax boundary information 300 .

4 and 5 are diagrams showing examples of the parallax information 106 used for generating the second correction information 201 in the second correction information generation unit 21. FIG. Below, the parallax information 106 is set corresponding to each position, for example, for each block of pixels. The region is called a “reference peripheral parallax region”.

FIG. 4 is a diagram showing a first example of the parallax information 106 used for generating the second correction information 201. FIG. In other words, FIG. 4 is a diagram showing a first example of determining a reference peripheral parallax region. FIG. 4 shows the parallax calculation process of the parallax generation target block 801 in the current frame 800, which is the latest captured image. A grid in FIG. 4 is a unit block for calculating parallax. The block size may be 1 pixel, 4 pixels (2 pixels wide by 2 pixels high), 16 pixels (4 pixels wide by 4 pixels high), or a square with 5 pixels or more on each side. , may be a rectangle composed of any number of pixels.

In the example shown in FIG. 4, the parallax of the upper left block is calculated first, and then the processing target is changed to the right direction as indicated by the arrow. , and similarly, the processing targets are changed in order from left to right. Therefore, when calculating the parallax of the block indicated by reference numeral 801, the calculation of all the parallaxes in the first to third stages in the drawing has been completed, and the calculation of the parallax on the left side of the parallax generation target block 801 in the fourth stage has also been completed. are doing. The shaded area in FIG. 4 is a five-pixel area centered on the parallax generation target block 801 , in which the parallax has already been calculated when the parallax of the parallax generation target block 801 is calculated. The parallax information 106 in this area is called "reference peripheral parallax 802". In the example of FIG. 4 , this reference peripheral parallax 802 is used for generating the second correction information 201 . That is, in the first example shown in FIG. 4, the reference peripheral parallax area is the reference peripheral parallax 802 .

FIG. 5 is a diagram showing a second example of the parallax information 106 used for generating the second correction information 201. FIG. In other words, FIG. 5 is a diagram showing a second example of determining the reference peripheral parallax region. FIG. 5 shows three images captured at different times, the current frame 800 being the latest frame, the first past frame 901 being the frame acquired in the immediately preceding processing cycle, and the second past frame 902 being the processing two frames before. It is a frame acquired periodically. A block 801 of the current frame 800 is a parallax generation target block, and corresponding blocks in the past frame are

blocks

903 and 904 .

The parallax in the shaded area in the first past frame 901 is called "first past reference peripheral parallax 905", and the parallax in the shaded area in the second past frame 902 is called "second past reference peripheral parallax 906". The first past reference peripheral parallax 905 and the second past reference peripheral parallax 906 are a second example of the parallax information 106 used to generate the second correction information 201 . Since these pieces of parallax information are included in past frames, they have already been calculated at the time of generating the parallax generation target block 801 , and parallax information around the

blocks

903 and 904 of the past frames corresponding to the parallax generation target block 801 is. That is, in the second example shown in FIG. 5 , the reference peripheral parallax area is only the first past reference peripheral parallax 905 or both the first past reference peripheral parallax 905 and the second past reference peripheral parallax 906 . Although two past frames are referred to in the example of FIG. 5, only one past frame or three or more past frames may be referred to.

Based on the parallax information 106, the peripheral parallax continuity determination unit 50 generates second correction necessity information 500 indicating whether or not the generation of the second correction information 301 is necessary. Examples of the second correction necessity information 500 include peripheral parallax information and correction necessity information.

An example of peripheral parallax information is the average parallax value in the reference peripheral parallax area. As an example of the calculation method of this parallax value, the average parallax value of the reference peripheral parallax 802 shown in FIG. 4, the average parallax value of the first past reference peripheral parallax 905 shown in FIG. and a second past-reference peripheral disparity 906 average disparity value. Also, the parallax value in only the horizontal direction may be referred to, or the parallax value in only the vertical direction may be referred to. Also, isolated parallaxes within the reference peripheral parallax area may be excluded from the reference target parallax. As an example of a method for detecting isolated parallax, there is a method of determining isolated parallax when the difference between the value of parallax to be detected and the value of parallax around the parallax to be detected is greater than a predetermined value. .

The second correction necessity information 500 is information indicating whether or not the second correction information 301 is generated in the second generation unit 51 . When the second correction necessity information 500 is “Generate: Necessary”, the correction execution unit 22 corrects the similarity using the second correction information 301 . When the second correction information 301 is “generate: unnecessary”, the correction execution unit 22 does not correct the similarity using the second correction information 301 .

For example, the peripheral parallax continuity determination unit 50 calculates the degree of variation in parallax within the reference peripheral parallax area, and if the degree of variation is large, sets the second correction necessity information 500 to "Generation: Not required". Examples of the degree of variation include the sum of differences between the average parallax value in the reference peripheral parallax area and each parallax value, and the variance value in the reference peripheral parallax area. To generate the degree of variation, the parallax value only in the horizontal direction may be referred to, or the parallax value only in the vertical direction may be referred to. As another example of the correction necessity information generation method, the degree of continuity of parallax in the reference peripheral parallax area may be calculated, and if the degree of continuity is small, the correction necessity information may be set as no correction. An example of the degree of continuity is the average or variance of the difference values between the parallax values in the reference peripheral parallax area. To generate the degree of continuity, the parallax value only in the horizontal direction may be referred to, or the parallax value only in the vertical direction may be referred to.

The second generation unit 51 generates second correction information 301 based on the second correction necessity information 500 . An example of the second correction information 301 is a correction coefficient K corresponding to the search position. The role of this correction coefficient K will be described with reference to FIG. A method of calculating the correction coefficient K will be described later.

FIG. 6 is a diagram for explaining the correction coefficient K. A graph 1110 shown in the upper part of FIG. 6 is an example of the second correction information 301, and is a set of correction coefficients K for each search position. A graph 1111 shown in the lower part of FIG. 6 indicates the degree of similarity for each search position, and indicates the same kind of information as in FIG. However, in FIG. 6, the solid line indicates the degree of similarity before correction, and the dashed line indicates the correction coefficient after correction. The upper part and the lower part of FIG. 6 will be described in order.

In the upper part of FIG. 6, the horizontal axis of the graph 1110 indicates the search position within the search range, and the vertical axis indicates the correction coefficient K for the first similarity information 104. The correction coefficient K0 is a value when the first similarity information 104 is not corrected. When the correction coefficient is smaller than the correction coefficient K0, the first similarity information 104 is corrected so that the similarity is higher. Also, when the correction coefficient is larger than the correction coefficient K0, the first similarity information 104 is corrected so that the similarity becomes smaller.

When the correction coefficient K is used by being multiplied by the first similarity information 104 before correction, for example, the correction coefficient K0 is "1", the correction coefficient K1 is a real number smaller than "1" and larger than "0", The correction coefficient K2 is a real number greater than "1". When the correction coefficient K is used by being multiplied by the first similarity information 104 before correction, for example, the correction coefficient K0 is "1", the correction coefficient K1 is a real number smaller than "1" and larger than "0", The correction coefficient K2 is a real number greater than "1". When the correction coefficient K is used in addition to the first similarity information 104 before correction, for example, the correction coefficient K0 is "0", the correction coefficient K1 is a positive real number, and the correction coefficient K2 is a negative real number. .

　In the lower part of Fig. 6, the horizontal axis of the graph 1111 indicates the search position within the search range, and the vertical axis indicates the degree of similarity. A first similarity curve 1104 indicated by a solid line is a similarity curve created using the first similarity information 104 as it is. A second similarity curve 1105 indicated by a dashed line is a similarity curve created using the second similarity information 105 generated based on the second correction information 1100 and the first similarity information 104 .

As shown in the graph 1110, the correction coefficient K at the search position A1 is "K0" as indicated by the element 1101 of the second correction information 301. Therefore, as indicated by element 1106 on second similarity curve 1105, the value of first similarity curve 1104 and the value of second similarity curve 1105 are the same at search position A1.

The correction coefficient K at the search position A2 is "K1", which is a value smaller than K0 as indicated by the element 1102 of the second correction information 301. FIG. Therefore, as indicated by element 1107 on second similarity curve 1105, the second similarity is higher than the first similarity at search position A2. The correction coefficient at search position A3 is “K2”, which is a value greater than K0 as shown in element 1103 of second correction information 301 . Therefore, as indicated by the element 1108 on the second similarity curve 1105, the second similarity is lower than the first similarity at the search position A3.
Correction of the first similarity information 104 using the correction coefficient K is, for example, as follows.

　　Sa (Ai) = Sb (Ai) x K (i)

Note that Sb(i) is the similarity at the search position (i) on the first similarity curve 1104, Sa(i) is the similarity at the search position (i) on the second similarity curve 1105, and K(i) is the correction coefficient K on the second correction information 1100 at the search position (i). In this case, the similarity at search position A2 in FIG. 6 has the following relationship.

　　　S1 = S0 x K1

The correction execution unit 22 reads the correction coefficient K for each search position with reference to the second correction information 301, and corrects the first similarity by multiplying it by the similarity corresponding to the position as in the above equation. do. Note that the correction execution unit 22 performs similarity correction using the second correction information 301 after similarity correction using the first correction information 200 is performed.

A specific example of a method of generating the second correction information 301 by the second generation unit 51, specifically a method of generating the correction coefficient K corresponding to the search position will be described with reference to FIG. FIG. 7A is a diagram showing a first example of calculating the correction coefficient K. FIG. The first row shows the parallax position, the second row shows the parallax value, the third row shows the continuity, the fourth row shows the function, and the fifth row shows the correction coefficient K. Here, if the difference from the parallax value adjacent to the left is less than or equal to a predetermined threshold value of "2", it is determined to be continuous and the degree of continuity is increased by "1". is decreased by "1", and the upper limit of continuity is set to "3".

Also, for the function, "0" means no correction, "1" means weak correction, and "2" means strong correction. For continuity '0' and '1' the function is set to '0', for continuity '2' the function is set to '1' and for continuity '3' the function is set to '1'. is set to "2". Unlike FIG. 6, the correction coefficient K is indicated by a numerical value for the convenience of drawing. "5" corresponds to "K0" in FIG. 4" corresponds to between "K1" and "K0" in FIG. The correction coefficient is set according to the value of the function. When the function is "0", the correction coefficient K is set to "5", and when the function is "1", the correction coefficient K is set to "4". , the correction coefficient K is set to "3" when the function is "2".

In the example of FIG. 7(a), parallax positions A to G have a continuous parallax value of "8". Therefore, the degree of continuity increases by 1 from the initial value of "0" and continues to the maximum value of "3". Since the parallax value is switched to "2" after the parallax position H, the difference from "8" is larger than the threshold value "2". , the continuity is "0". Since the parallax value "2" continues after that, the degree of continuity turns to increase, meaning the continuation of "2", and the maximum value "3" continues. Since the function is linked to the degree of continuity and the correction coefficient K is linked to the function, the value of the function varies according to the degree of continuity in FIG. is fluctuating.

FIG. 7(b) is a diagram showing a second example of calculating the correction coefficient K. FIG. 1st row is parallax position, 2nd row is parallax value, 3rd row is continuity A, 4th row is continuity B, 5th row is maximum continuity, 6th row is function, 7th row is correction coefficient indicates K. Continuity A and continuity B count continuous states of different parallax values. The maximum continuity is the larger of continuity A and continuity B. The function is set in conjunction with maximum continuity in this example.

In the example of FIG. 7B, the parallax positions A to G have a continuous parallax value of "8", the parallax positions H to M have a continuous parallax value of "2", and the parallax positions N to Q have a parallax value of "5". are consecutive. Therefore, at the parallax positions A to G, the degree of continuity A increases in the same way as the degree of continuity in FIG. 7A and maintains the maximum value of "3". At this time, the degree of continuity B is described with an asterisk indicating no value because only one type of parallax value still exists and is not used.

When the parallax value switches to "2" at the parallax position H, the difference from the previous parallax value "8" is greater than the threshold value "2", so the degree of continuity A decreases and becomes the lower limit "0" in this example. . The degree of continuity B has an initial value of “0” at the parallax position H, and thereafter increases to a maximum value of “3” at the parallax position K because the parallax value continues to be “2”. After that, when the parallax value changes to "5" at the parallax position N, the degree of continuity B decreases, and the degree of continuity A increases to replace this. Since the maximum continuity is the maximum value of the continuity A and the continuity B as described above, the maximum continuity is the same as the continuity A up to the parallax position G where the value of the continuity B does not exist, and from the parallax position J to N Up to , the value of the continuity B, which is larger, becomes the maximum continuity. The value of the function is determined according to the maximum continuity, and the value of the correction factor K is determined according to the value of the function.

In the example shown in FIG. 7, the correction coefficient K takes only values corresponding to K0 to K1. can be changed to take the value of For example, when the function is "0", the correction coefficient K is set to "7", namely K2; when the function is "1", the correction coefficient K is set to "5", namely K0; , the correction coefficient K is set to "3", that is, K1.

According to the first embodiment described above, the following effects are obtained.
(1) The computing device 1 is connected to the first imaging unit 2 and the second imaging unit 3, and the first image 100 captured by the first imaging unit 2 and the second image captured by the second imaging unit 3 101 is generated by searching in predetermined matching block units. The computing device 1 includes a similarity generating unit 11 that generates a first similarity between a matching block in the first image 100 and a matching block in the search range of the second image 101 for each matching block, Using at least one of the first correction information 200 generated based on the first similarity of any adjacent matching block within the search range of and the second correction information 301 generated based on the parallax information 106 A similarity correction unit 12 that corrects first similarity information 104 to generate second similarity information 105 and a similarity determination unit 13 that generates disparity information 106 based on the second similarity information 105 are provided. Therefore, by correcting the degree of similarity using the first correction information 200 and the second correction information 301, erroneous matching can be prevented and the accuracy of disparity calculation can be improved.

(2) The first image capturing unit 2 and the second image capturing unit 3 generate the first image 100 and the second image 101 for each predetermined processing cycle. As shown in FIGS. 4 and 5, the arithmetic device 1 calculates the latest first image 100 and the latest second image 101, or the first image 100 before the predetermined period and the second image 101 before the predetermined period, A second correction information generation unit 21 is provided for generating second correction information, which is a correction coefficient K by which the degree of similarity is multiplied, based on the parallax information 106 in the peripheral region of the matching block. Therefore, when using the latest captured image, the correction coefficient K can be calculated based on the parallax information 106 with no positional deviation, and when using the past captured image, the correction coefficient can be calculated using the parallax information of the entire frame.

(3) The second correction information 301 is the first similarity degree corresponding to the parallax information in the peripheral area when the continuity of the parallax information in the peripheral area is equal to or greater than the threshold, as in the search position A2 in FIG. contains a correction factor that corrects to increase Therefore, by increasing the similarity of a matching block that is expected to have a similar parallax to its surroundings, erroneous matching can be prevented and the accuracy of parallax calculation can be improved.

(4) The second correction information 301 is the first similarity degree corresponding to the parallax information in the peripheral area when the continuity of the parallax information in the peripheral area is equal to or less than the threshold, as in the search position A3 in FIG. contains a correction factor that corrects to reduce Therefore, it is possible to prevent erroneous matching and improve the accuracy of parallax calculation by making it difficult to match by reducing the similarity of a matching block whose parallax is not similar to its surroundings.

(5) After correcting the first similarity information 104 based on the first correction information 200, the correction execution unit 22 of the similarity correction unit 12 further corrects the first similarity information 104 based on the second correction information 301. A second degree of similarity is generated by correcting. Therefore, the calculation device 1 can use the result of the first correction for correcting the local similarity for the second correction for correcting the global similarity.

(6) The first imaging unit 2 and the second imaging unit 3 are arranged in the baseline direction. Scan in the baseline direction to search for inflection points where the difference in similarity switches between positive and negative, calculate a new similarity using the similarity around the inflection point, and combine the position of the inflection point and the new similarity A first correction information generator 20 is provided to calculate the first correction information 200 . Therefore, local similarity correction can be performed more finely than the similarity calculation unit. For example, if the similarity is calculated for each pixel, the search position 1003 has a decimal coordinate value, so it can be said that the similarity can be calculated with sub-pixel precision.

(7) The monitoring system S includes an arithmetic device 1 , a first imaging section 2 and a second imaging section 3 . Therefore, the monitoring system S has less false matching and good parallax calculation accuracy.

(Modification 1)
In the first embodiment described above, the similarity correction unit 12 of the arithmetic device 1 includes the first correction information generation unit 20 and the second correction information generation unit 21 . However, the similarity correction section 12 may include only one of the first correction information generation section 20 and the second correction information generation section 21 .

(Modification 2)
In the above-described first embodiment, since the first imaging unit 2 and the second imaging unit 3 are arranged horizontally, the baseline direction is horizontal. was scanned. However, the first imaging section 2 and the second imaging section 3 may be arranged vertically or diagonally. In this case, the first generation unit 41 may scan the similarity along the baseline direction, that is, along the epipolar line.

-Second Embodiment-
A second embodiment of the monitoring system will be described with reference to FIG. In the following description, the same components as those in the first embodiment are assigned the same reference numerals, and differences are mainly described. Points that are not particularly described are the same as those in the first embodiment. This embodiment differs from the first embodiment mainly in that it includes a correction information invalidation unit.

FIG. 8 is a configuration diagram of the correction execution unit 22A in the second embodiment. The correction executing section 22A includes a correction information invalidating section 60 and a correcting section 61 . The corrector 61 includes a first corrector 70 and a second corrector 71 .

First similarity information 104, first correction information 200, and second correction information 301 are input to correction execution unit 22 in the first embodiment. , an invalidation instruction 602 is further input in addition to the above three. The correction execution unit 22A performs correction processing of the first similarity information 104 based on the invalidation instruction 602, the first correction information 200 and the second correction information 301, and generates the second similarity information 105. FIG.

The correction information invalidation unit 60 can invalidate at least one of the first correction information 200 and the second correction information 301 based on the invalidation instruction 602 . For example, the correction information invalidation unit 60 may invalidate both the first correction information 200 and the second correction information 301, may invalidate only one of them, or may invalidate none of them. . Examples of the invalidation instruction 602 include information indicating validity or invalidity of the first correction information 200 and information indicating validity or invalidity of the second correction information 301 . That is, the correction information invalidation unit 60 regards the first correction information 200 output by the first correction information generation unit 20 and the second correction information 301 output by the second correction information generation unit 31 as also the second effective correction information 601. accept.

When the correction information invalidation unit 60 receives the invalidation instruction 602 indicating that the first correction information 200 is valid and the second correction information 301 is invalid, for example, it performs the following processing. That is, the correction information nullification section 60 outputs the first correction information 200 as the first effective correction information 600 and outputs the second effective correction information 601 as information obtained by nullifying the second correction information 301 .

The correction unit 61 corrects the first similarity information 104 based on the first effective correction information 600 and the second effective correction information 601 to generate the second similarity information 105 . As an example of correction processing for the first effective correction information 600, the first correction information 200 is used to replace the similarity at the search position to be corrected, or insert the similarity at the new search position to obtain the second similarity information. There is a way to generate 105. As an example of correction processing for the second effective correction information 601, there is the method of generating the second similarity curve 1105 described above with reference to FIG. Further, when the first correction information 200 is invalidated in the first effective correction information 600, the correction processing using the first correction information 200 is not performed. When the first correction information 200 is invalidated in the second effective correction information 601, correction processing using the first correction information 200 is not performed.

The first correction unit 70 performs correction processing of the first similarity information 104 based on the first effective correction information 600 and generates intermediate similarity information 700 . The second correction unit 71 performs correction processing of the intermediate similarity information 700 based on the second effective correction information 601 to generate the second similarity information 105 . The correction processing in the first correction unit 70 and the correction processing in the second correction unit 71 are as described with reference to FIGS. 3 and 6, so details thereof will be omitted.

According to the above-described second embodiment, it is possible to selectively use appropriate corrections.

-Third Embodiment-
A third embodiment of the monitoring system will be described with reference to FIG. In the following description, the same components as those in the first embodiment are assigned the same reference numerals, and differences are mainly described. Points that are not particularly described are the same as those in the first embodiment. The present embodiment is different from the first embodiment mainly in that the boundary of parallax is not detected to generate the second correction information.

FIG. 9 is a configuration diagram of the similarity correction unit 12B in the third embodiment. The similarity correction unit 12B includes a second correction information generation unit 21B instead of the second correction information generation unit 21, and a parallax boundary detection unit 30 newly. The parallax boundary detection unit 30 refers to the calculated parallax information 106, and upon detecting a parallax boundary in the matching block, outputs the parallax boundary information 300 to the second correction information generation unit 21B. For example, one of the following three methods can be used to determine the presence or absence of a parallax boundary within a matching block.

The first method is to individually check the parallax of each block in the matching block in the current frame. Specifically, the parallax of each block in the matching block is compared with the parallax of other blocks on the left, right, top and bottom, and if the difference is equal to or greater than a predetermined threshold value, it is determined that there is a boundary of parallax. Any block is judged to have no parallax boundary if the difference in parallax from other blocks on the left, right, top and bottom is less than a predetermined threshold. In this first method, the current frame is used, so the latest information is used, but the difference with blocks for which parallax has not yet been calculated cannot be evaluated.

The second method is to individually confirm the parallax of each block within the matching block in the past frame acquired immediately before. The difference from the first method is that the target frame is the target frame, and all blocks in the past have already calculated the parallax difference. can be calculated. This second method assumes that the position of the subject does not change significantly in the time of one frame, and embodies the idea that the benefit of being able to calculate the disparity difference in all blocks outweighs the loss due to the change in the position of the subject. It is what I did.

The third method is to detect parallax boundaries by hierarchical search. This method may be applied to the current frame, as in the first method, or to past frames, as in the second method. In the hierarchical search, first, in the first layer, the parallax is calculated using a block that is larger than the matching block, for example, a block that is twice as large both vertically and horizontally as the matching block. This block is hereinafter referred to as a large size block.

Next, in the second layer, the parallax of each block within the matching block is calculated based on the parallax of the large size block. Specifically, a total of three parallaxes, that is, the parallax of a large-sized block including a block for which parallax is to be calculated and the parallax of left and right large-sized blocks adjacent to the large-sized block, are set as parallax candidates, and a predetermined and the most probable parallax is set as the parallax to be calculated. Using the parallax calculated by the hierarchical search in this way, it is determined whether or not the difference from the parallax of other blocks on the left, right, top and bottom is equal to or greater than a predetermined threshold value, and whether or not there is a parallax boundary.

Upon receiving c from the parallax boundary detection unit 30, the peripheral parallax continuity determination unit 50 determines whether or not the second correction is necessary to generate the second correction information 301 when the parallax boundary exists in the matching block. Information 500 is created and transmitted to the second generator 51 . Since the second generation unit 51 that has received the second correction necessity information 500 does not generate the second correction information 301, the correction execution unit 22 also corrects the first similarity information 104 based on the second correction information 301. do not have.

According to the third embodiment described above, the following effects are obtained.
(8) The computing device 1 includes a parallax boundary detection unit 30 that generates parallax boundary information 300 based on luminance information or parallax information of the first image 100 or the second image 101 in the peripheral area. The second correction information generator 21 disables generation of the second correction information 201 based on the parallax boundary information 300 . Therefore, when there is a parallax boundary in the surroundings, the similarity degree is not corrected by the second correction information 301, and the adverse effect of the similarity correction by the second correction information 301 can be prevented.

(Modification of the third embodiment)
In the third embodiment described above, the parallax boundary detection unit 30 uses the parallax information 106 to determine whether or not there is a parallax boundary. However, the parallax boundary detection unit 30 may simply determine whether there is a parallax boundary by using the first image 102 or the second image 103 and using the presence or absence of luminance change. The parallax boundary detection unit 30 refers to the first image 102 or the second image 103, compares the luminance of each block in the matching block with the luminance of other blocks on the left, right, top and bottom, and determines if the difference is equal to or greater than a predetermined threshold. It is determined that there is a parallax boundary in a certain case. Any block is judged to have no luminance boundary if the difference in luminance from other blocks on the left, right, top and bottom is less than a predetermined threshold.

In each embodiment and modification described above, the configuration of the functional blocks is merely an example. Some functional configurations shown as separate functional blocks may be configured integrally, or a configuration represented by one functional block diagram may be divided into two or more functions. Further, a configuration may be adopted in which part of the functions of each functional block is provided in another functional block.

In each of the embodiments and modifications described above, the program is stored in a ROM (not shown), but the program may be stored in a non-volatile storage device (not shown) such as a flash memory. Alternatively, the arithmetic device may have an input/output interface (not shown), and the program may be read from another device via a medium that can be used by the input/output interface and the arithmetic device when necessary. Here, the medium refers to, for example, a storage medium that can be attached to and detached from an input/output interface, or a communication medium, that is, a wired, wireless, or optical network, or a carrier wave or digital signal that propagates through the network. Also, part or all of the functions realized by the program may be realized by a hardware circuit or FPGA.

Each of the above-described embodiments and modifications may be combined. Although various embodiments and modifications have been described above, the present invention is not limited to these contents. Other aspects conceivable within the scope of the technical idea of the present invention are also included in the scope of the present invention.

S: Monitoring system 1: Arithmetic device 10: Input unit 11: Similarity generation units 12, 12B: Similarity correction unit 13: Similarity determination unit 20: First correction

information generation units

21, 21B: Second correction

information generation unit

22, 22A... Correction execution unit 30... Parallax boundary detection unit 31... Second correction information generation unit 40... Correction determination unit 41... First generation unit 50... Peripheral parallax continuity determination unit 51... Second generation unit 60... Correction information Invalidation unit 61...Correction unit 70...First correction unit 71...Second correction unit

Claims

connected to a first imaging unit and a second imaging unit, and disparity information between a first image captured by the first imaging unit and a second image captured by the second imaging unit in predetermined matching block units; A computing device generated by searching,
a similarity generating unit that generates a first similarity between a matching block in the first image and a matching block in a search range of the second image for each matching block;
At least one of first correction information generated based on the first similarity of any adjacent matching block within the search range of the second image, and second correction information generated based on the parallax information. A similarity correction unit that corrects the first similarity using to generate a second similarity,
and a similarity determining unit that generates parallax information based on the second degree of similarity.
The computing device according to claim 1,
The first imaging unit and the second imaging unit generate the first image and the second image for each predetermined processing cycle,
Based on the parallax information in the peripheral region of the matching block in the latest first image and the latest second image, or the first image before the predetermined period and the second image before the predetermined period. The computing device further comprising a second correction information generation unit that generates the second correction information that is a correction coefficient to be multiplied by one degree of similarity.
The arithmetic device according to claim 2,
The second correction information includes a correction coefficient for correcting so as to increase the first similarity corresponding to the parallax information in the peripheral area when the continuity of the parallax information in the peripheral area is equal to or greater than a threshold. , arithmetic unit.
The arithmetic device according to claim 2,
The second correction information includes a correction coefficient for correcting so as to reduce the first similarity corresponding to the parallax information in the peripheral area when the continuity of the parallax information in the peripheral area is equal to or less than a threshold. , arithmetic unit.
The arithmetic device according to claim 2,
further comprising a parallax boundary detection unit that generates parallax boundary information based on the luminance information or the parallax information of the first image in the peripheral area;
The arithmetic device, wherein the second correction information generation unit disables generation of the second correction information based on the parallax boundary information.
The computing device according to claim 1,
The similarity correction unit corrects the first similarity based on the first correction information, and then further corrects the first similarity based on the second correction information to generate the second similarity. , arithmetic unit.
The computing device according to claim 1,
The first imaging unit and the second imaging unit are arranged in a baseline direction,
Scanning in the baseline direction to search for an inflection point where the difference in similarity switches between positive and negative, calculate a new similarity using the similarity around the inflection point, and calculate the position of the inflection point and the new A computing device, further comprising a first correction information generator that calculates a combination of similarities as first correction information.
A computing device according to claim 1;
the first imaging unit;
A monitoring system comprising: the second imaging unit;
connected to a first imaging unit and a second imaging unit, and disparity information between a first image captured by the first imaging unit and a second image captured by the second imaging unit in predetermined matching block units; A parallax calculation method executed by an arithmetic device generated by searching,
generating a first similarity between a matching block in the first image and a matching block in a search range of the second image for each matching block;
At least one of first correction information generated based on the first similarity of any adjacent matching block within the search range of the second image, and second correction information generated based on the parallax information. generating a second similarity by correcting the first similarity using
and generating parallax information based on the second similarity.