DE112022001667T5 - DATA PROCESSING DEVICE, MONITORING SYSTEM AND PARALLAX CALCULATION METHOD - Google Patents

Abstract

A data processing device is connected to a first image capture unit and a second image capture unit and generates parallax information of a first image captured by the first image capture unit and a second image captured by the second image capture unit by searching in units of predetermined matching blocks, the data processing device being a similarity generation unit , which produces a first similarity to a matching block in a search area of the second image with respect to a matching block in the first image in units of matching blocks; a similarity correction unit that corrects the first similarity using at least first correction information generated based on a first similarity of any adjacent matching block within a search area of the second image and/or second correction information generated based on the parallax information to create a second similarity; and a similarity determination unit that generates parallax information based on the second similarity.

Description

Technical area

The present invention relates to a data processing device, a monitoring system and a parallax calculation method.

State of the art

A method of calculating a distance between a camera and a viewing object using a stereo camera and executing a process of recognizing the viewing object is known. PTL 1 discloses an information processing apparatus including a similarity data generating unit that captures each of a plurality of areas captured by dividing an image into a left and a right photographed image captured by a stereo camera that captures a same space from a left and a right viewing angle are, as a reference block, sets a search area in the other image for each reference block and generates similarity data which is calculated by linking the similarity of an image to the reference block and a position within the search area with respect to the position within the search area, a similarity correction unit , which smoothes the similarity data in a spatial direction based on similarity data which has been generated for a predetermined number of reference blocks on a circumference of a relevant reference block, a result evaluation unit which detects a position at which the similarity in all smoothed similarity data has a maximum value , a depth image generation unit that obtains a parallax for each of the reference blocks using the detection result of the result evaluation unit and generates a depth image by calculating a position of an object in a depth direction based on the calculation and linking the position with an image plane, and an output information generation unit that previously performs specified information processing based on the position of the object in a three-dimensional space using the depth image and outputs a result thereof.

Sources

Patent literature

PTL 1: Published Japanese Patent Application No. 2016-039618

Summary of the invention

Technical task

The invention described in PTL 1 has a need for improvement with regard to the calculation of the parallax.

the solution of the problem

A data processing device in accordance with a first aspect of the present invention is a data processing device which is connected to a first image capture unit and a second image capture unit and generates parallax information of a first image captured by the first image capture unit and a second image captured by the second image capture unit by searches it in units of predetermined matching blocks, the data processing device comprising a similarity generating unit which generates a first similarity to a matching block in a search area of the second image with respect to a matching block in the first image in units of matching blocks; a similarity correction unit that generates the first similarity using at least first correction information based on a first similarity of a first arbitrary adjacent matching block within a search area of the second image, and/or second correction information based. on which parallax information is generated, corrected to produce a second similarity; and a similarity determination unit that generates parallax information based on the second similarity.

A surveillance system in accordance with a second aspect of the present invention comprises the data processing device described above, the first image capture unit; and the second image capture unit.

A parallax calculation method in accordance with a third aspect of the present invention relates to a parallax calculation method carried out by a data processing device, the data processing device being connected to a first image capture unit and a second image capture unit, and parallax information of a first image captured by the first image capture unit and a second, image captured by the second image capture unit by searching in units of predetermined matching blocks, the method comprising generating a first similarity to a matching block in a search area of the second image with respect to a matching block in the first image in units of matching blocks; correcting the first similarity using at least first correction information generated based on the first similarity of a first arbitrary adjacent matching block within a search area of the second image and/or second correction information generated based on the parallax information by a second similarity to create; and generating parallax information based on the second similarity.

Advantageous effects of the invention

In accordance with the present invention, it is possible to reduce erroneous matches and improve parallax calculation accuracy.

Brief description of the drawings

• [ 1 ] 1 is a configuration diagram of a monitoring system having a data processing device.
• [ 2 ] 2 is a configuration diagram of a similarity correction unit.
• [ 3 ] 3 is a diagram for explaining first similarity information and first correction information.
• [ 4 ] 4 is a diagram illustrating a first example of parallax information used to generate second correction information.
• [ 5 ] 5 is a diagram illustrating a second example of parallax information used to generate second correction information.
• [ 6 ] 6 is a diagram for explaining a correction coefficient.
• [ 7 ] 7 is a diagram for explaining a concrete example of a method of generating a correction coefficient.
• [ 8th ] 8th is a configuration diagram of a correction execution unit in accordance with a second embodiment.
• [ 9 ] 9 is a configuration diagram of a similarity correction unit in accordance with a third embodiment.

Description of embodiments

-First embodiment-

The following is a first embodiment of a monitoring system with reference to 1 until 7 described.

1 is a configuration diagram of a monitoring system S having a data processing device 1. A vehicle environment monitoring system S has a data processing device 1, a first image capture unit 2, a second image capture unit 3, a recognition processing unit 4 and a vehicle control unit 5. The vehicle environment monitoring system S is attached to the vehicle C. The data processing device 1 is connected to the first image capture unit 2, the second image capture unit 3 and the recognition processing unit 4. The recognition processing unit 4 is connected to the data processing device 1 and the vehicle control unit 5.

The data processing device 1, the recognition processing unit 4 and the vehicle control unit 5 are each an electronic control unit (ECU) attached to the vehicle C. However, instead of individual ECUs, two or more of the elements from the group including the data processing device 1, the recognition processing unit 4 and the vehicle control unit 5 may be implemented by the same ECU.

The first image capture unit 2 and the second image capture unit 3 are each a camera, and each outputs a photographed image, which is an image acquired by capture, to the data processing device 1. The photographed image output from the first image capture unit 2 is referred to as a first photographed image 100, and the photographed image output from the second image capture unit 3 is referred to as a second photographed image 101. The first image capture unit 2 and the second image capture unit 3 are separated from each other by a predetermined baseline length in the transverse direction in the same direction with respect to the vehicle C. It should be noted that the transverse direction in which the first image capture unit 2 and the second image capture unit 3 are arranged is also referred to as a “baseline direction” in this description. Based on the first photographed image 100 and the second photographed image 101, the data processing device 1 generates parallax information 106 using a method described later driving, and outputs the parallax information 106 to the recognition processing unit 4.

The data processing device 1 has an input unit 10, a similarity generation unit 11, a similarity correction unit 12 and a similarity determination unit 13. The input unit 10 is implemented by a hardware interface with the first image capture unit 2 and the second image capture unit 3 as well as an arithmetic processing unit. The similarity generation unit 11, the similarity correction unit 12 and the similarity determination unit 13 are implemented by an arithmetic processing unit. The hardware interface is, for example, an RJ45 connection or a camera connection.

The arithmetic processing unit includes, for example, a CPU, which is a central processing unit, a ROM, which is a read-only memory device, and a RAM, which is a readable/writable memory device, and the CPU executes a calculation described later , by building a program stored in ROM in RAM and executing the program. Instead of the combination of the CPU, the ROM and the RAM, the arithmetic processing unit may be implemented by a field programmable gate matrix (FPGA), which is a rewritable logic circuit, or an application specific integrated circuit (ASIC), which is an application specific integrated circuit . Furthermore, instead of the combination of the CPU, the ROM and the RAM, the arithmetic processing unit can also be realized by a combination of different configurations, for example a combination of a CPU, a ROM, a RAM and an FPGA.

The input unit 10 provides the similarity generation unit 11 with a first image 102 obtained by performing image processing on the first photographed image 100 and a second image 103 obtained by performing image processing on the second photographed image 101. This image processing is carried out using, for example, an internal parameter or an external parameter of the camera to prevent an adverse effect on the image processing. Examples of image processing include correction of lens distortion, error correction of mount position and mount orientation, and the like. However, since the process in the input unit 10 is not essential in the present embodiment, the first photographed image 100 and the first image 102 may be considered as the same image, and/or the second photographed image 1021 and the second image 103 may be regarded as the same image to be viewed as.

The similarity generation unit 11 receives the first image 102 provided by the input unit 10 and the second image 103 provided by the input unit 10, generates the first similarity information 104, and provides the first similarity information 104 to the similarity correction unit 12. The similarity correction unit 12 handles information indicating similarities between the matching block in the first image 102 and the matching block in the second image 103. The first similarity information 104 is, for example, a luminance change amount of each of the pixels in the matching block of the first image 102 and a pixel in the matching block of the corresponding second image 103. For example, a calculation method such as sum of absolute difference (SAD) may calculate the average zero absolute sum Difference (ZSAD), the sum of squares of deviation (SSD), the normalized cross-correlation (NCC) or the average normalized zero cross-correlation (ZNCC) can be used as the luminance deviation amount.

The first similarity information 104 may be a luminance deviation amount, a luminance deviation direction, or a luminance deviation pattern for each of the pixels and the surrounding pixels thereof in the matching block of the first image 102 and the second image 103. An example of the luminance variation pattern is an increase/decrease pattern from the left side with respect to three horizontal pixels that include the left and right adjacent pixels of a target pixel. For example, if the value of the left neighboring pixel is 10, the value of the target pixel is 25, and the value of the right neighboring pixel is 80, the increase/decrease pattern is “increase → increase”.

The similarity correction unit 12 generates second similarity information 105 obtained by correcting the first similarity information 104 based on the first similarity information 104 or the parallax information 106. The similarity determination unit 13 detects a position having the highest similarity within the search area based on the second similarity information 105 and generates parallax information 106 based on the position information. The similarity determination unit 13 outputs the generated parallax information 106 to the recognition processing unit 4 and the similarity correction unit 12.

The recognition processing unit 4 carries out various recognition processes with the parallas xeninformation 106 as input. An example of the recognition process in the recognition processing unit 4 is three-dimensional object detection using the parallax information 106. Other examples of the recognition of the target by the recognition processing unit 4 are position information, type information, motion information and danger information of the object. An example of the position information includes a direction and a distance from the host vehicle. Examples of the type information include a pedestrian, an adult, a child, an elderly person, an animal, a fallen rock, a bicycle, a surrounding vehicle, a surrounding structure, and a curb. Examples of the motion information include lurching, jumping out, crossing, moving direction, moving speed and trajectory of a pedestrian or a bicycle. Examples of the danger information include a pedestrian jumping out, a rock falling, and abnormal operations of surrounding vehicles such as sudden stopping, sudden braking, and sudden steering. The recognition information 107 generated by the recognition processing unit 4 is provided to the vehicle control unit 5.

The vehicle control unit 5 performs various control operations of the vehicle C based on the recognition information 107. Examples of vehicle control operations carried out by the vehicle control unit 5 are brake control, steering wheel control, accelerator control, control of lamps in the vehicle, warning sound generation, control of the camera in the vehicle, and outputting information related to a viewing object around the image capture device to a surrounding vehicle or a vehicle stationary central device, which is connected via a network. Specific examples include speed and braking control in accordance with the parallax information 106 of an obstacle present in front of the vehicle.

Note that the distance information can be generated based on the parallax information and used in place of the parallax information 106. Although not described in the present embodiment, the vehicle control unit 5 may execute an object detection process based on the image processing result using the first image 102 and the second image 103. Furthermore, the vehicle control unit 5 can display an image or a display acquired via the first image capture unit 2 or the second image capture unit 3, which allows the viewer to recognize an obstacle on the display device which is connected to the vehicle control unit 5. Further, the vehicle control unit 5 may provide information about the detected observation object based on the image processing result to an information device that processes traffic information such as road map information and traffic congestion information.

2 is a configuration diagram of the similarity correction unit 12. The similarity correction unit 12 includes a first correction information generation unit 20, a second correction information generation unit 21, and a correction execution unit 22. The first correction information generation unit 20 generates first correction information 200 based on the first similarity information 104. The second correction information generation unit 21 generates second correction information 201 based on the parallax information 106. The correction execution unit 22 corrects the first similarity information 104 based on the first correction information 200 and the second correction information 201 to generate the second similarity information 105.

und $$\left(\ddot{\text{A}}\text{hnlichkeit S}\left(\text{i}+1\right)-\ddot{\text{A}}\text{hnlichkeit S}\left(\text{i}\right)\right)>\text{TH}1$$

A configuration of the first correction information generation unit 20 will be described. The first correction information generation unit 20 has a correction determination unit 40 and a first generation unit 41. The correction determination unit 40 generates first information regarding the validity of a correction 400, which indicates the need for generating the first correction information 200 based on the first similarity information 104. For example, the correction determination unit 40 determines that the generation of the first correction information 200 is necessary when the following first determination condition is satisfied, and determines that the generation of the first correction information 200 is not necessary when the first determination condition is not satisfied.
$$\left(\ddot{\text{A}}\text{Similarity S}\left(\text{i}-1\right)-\ddot{\text{A}}\text{Similarity S}\left(\text{i}\right)\right)>\text{TH}\mathrm{0,}$$

and $$\left(\ddot{\text{A}}\text{Similarity S}\left(\text{i}+1\right)-\ddot{\text{A}}\text{Similarity S}\left(\text{i}\right)\right)>\text{<figure-callout id="1" label="TH" filenames="DE112022001667T5\_0009.png,DE112022001667T5\_0011.png" state="{{state}}">TH</figure-callout>}1$$

Note that the similarity A(i) is a similarity at the search position A(i) to be set within the search area, the similarity A(i - 1) is a similarity at the search position A(i - 1) adjacent to the left side of the search position A(i), and the similarity A(i + 1) is a similarity at the search position A(i + 1) adjacent to the right side of the search position A(i). TH0 and TH1 are predetermined threshold values. Further, the data size determination in the first determination condition may be modified such that the condition is satisfied in a case where the sizes are equal.

und $$\left(\ddot{\text{A}}\text{hnlichkeit S}\left(\text{i}-1\right)-\ddot{\text{A}}\text{hnlichkeit S}\left(\text{i}\right)\right)>\text{TH}\mathrm{3,}\text{ und}$$

$$\left(\ddot{\text{A}}\text{hnlichkeit S}\left(\text{i}+1\right)-\ddot{\text{A}}\text{hnlichkeit S}\left(\text{i}\right)\right)>\text{TH}\mathrm{4,}\text{ und}$$

$$\left(\ddot{\text{A}}\text{hnlichkeit S}\left(\text{i}+2\right)-\ddot{\text{A}}\text{hnlichkeit S}\left(\text{i}+\text{1}\right)\right)>\text{TH}5$$

The reason for referring to the right and left sides instead of up and down in the first detection condition, that is, the reason for scanning in the lateral direction is that the baseline direction is transverse. The first determination condition can be determined by using not only three points but also five similarity points, as described below. TH2 to TH5 are used here, which are predetermined threshold values.
$$\left(\ddot{\text{A}}\text{Similarity S}\left(\text{i}-2\right)-\ddot{\text{A}}\text{Similarity S}\left(\text{i}-\text{1}\right)\right)>\text{TH}\mathrm{2,}$$

and $$\left(\ddot{\text{A}}\text{Similarity S}\left(\text{i}-1\right)-\ddot{\text{A}}\text{Similarity S}\left(\text{i}\right)\right)>\text{<figure-callout id="3" label="TH" filenames="DE112022001667T5\_0009.png,DE112022001667T5\_0011.png" state="{{state}}">TH</figure-callout>}\mathrm{3,}\text{and}$$

$$\left(\ddot{\text{A}}\text{Similarity S}\left(\text{i}+1\right)-\ddot{\text{A}}\text{Similarity S}\left(\text{i}\right)\right)>\text{<figure-callout id="4" label="TH" filenames="DE112022001667T5\_0009.png,DE112022001667T5\_0015.png" state="{{state}}">TH</figure-callout>}\mathrm{4,}\text{and}$$

$$\left(\ddot{\text{A}}\text{Similarity S}\left(\text{i}+2\right)-\ddot{\text{A}}\text{Similarity S}\left(\text{i}+\text{1}\right)\right)>\text{<figure-callout id="5" label="TH" filenames="DE112022001667T5\_0009.png,DE112022001667T5\_0015.png" state="{{state}}">TH</figure-callout>}5$$

The first generation unit 41 receives an input of the first similarity information 104 and the correction validity information 400, and generates the first correction information 200 based on the correction validity information 400. The first correction information 200 is, for example, Combination of a search position to be corrected and similarity information after correction, or a combination of a newly generated search position and similarity information. The correction execution unit 22 corrects the similarity with reference to the first correction information 200. An operation of the first correction information generation unit 20 is explained with reference to 3 described.

3 is a diagram for explaining the first similarity information 104 and the first correction information 200. The upper part of 3 represents the entire similarity curve 1010, and the lower part of 3 illustrates details of the position portion of the similarity curve 1010. In particular, the similarity curve 1012 shows which is in the lower part of 3 is shown, an area of a reference numeral 1011 in the upper part of 3 . In the similarity curve 1010 and the similarity curve 1012, the horizontal axis represents the search position in the transverse direction within the search area, and the vertical axis represents the similarity, with the similarity being greater towards the lower part of the vertical axis.

A black circle that appears in the similarity curve 1012, which is in the lower part of 3 shown shows the similarity at each of the search positions. Since the similarity generation unit 11 performs a search in a predetermined pixel unit, the similarity is displayed in the similarity curve 1012 at the distances of the search pixel unit. The search pixel unit can be set arbitrarily, but cannot be less than one pixel. The search positions in the middle of the drawing are labeled 1003, 1001, 1000, 1002 and 1004 from the left.

The similarity S0 at the search position 1000 of the similarity curve 1012 is greater than the similarity of the left and right search positions. Therefore, the search position 1000 satisfies the first determination condition described above. As an example of a method of correcting the similarity curve 1012, there is a method of replacing the similarity SO at the search position 1000 with the similarity S1. As another example of the method, there is a method of generating a new similarity S1 at a search position around the search position 1000, for example at the search position 1003.

That the search position 1003 in the similarity curve 1012 has the similarity S1 is an example of the first correction information 200. As an example of a method for generating the similarity S1, there is a method of calculating the similarity S1 by polynomial interpolation, such as linear approximation or SSD parabola matching based on a plurality of nearby similarity levels. As an example of the plurality of similarities arranged nearby, there are three similarities of the search positions 1000, 1001 and 1002. The search position 1003 is a search position with a fractional precision located between the search position 1000 and the search position 1002.

As described above, the first correction information generation unit 20 scans in the baseline direction to search for a turning point at which the sign of the difference of similarities around the turning point turns around, calculates a new similarity using the similarity around the turning point, and calculates a combination of the position of the turning point and the new similarity as first correction information 200.

There are at least the following three methods as the method for corrective execution unit 22 for correcting the similarity using the first correction information 200 when the first correction information 200 has the search position 1003 and the similarity S1. The first method is a method of overwriting the information of the search position 1000 and the similarity S0 with the information of the search position 1003 and the similarity S1. The second method is a method of adding the information of the search position 1003 and the similarity S1 to the present information of the search position 1000 and the similarity S0. The third method is a method of overwriting the similarity value S0 with the value of S1 without changing the position of the search position 1000. When the third method is adopted, the information of the search position 1003 can be calculated by three-point approximation or the like.

Referring again to 2 The configuration of the second correction information generating unit 21 will be described below. The second correction information generation unit 21 has a peripheral parallax continuity determination unit 50 and a second generation unit 51, and generates the second correction information 301 based on the parallax information 106 or the parallax boundary information 300.

The 4 and 5 are diagrams illustrating examples of the parallax information 106 used by the second correction information generating unit 21 to generate the second correction information 201. Hereinafter, since the parallax information 106 is set in accordance with each of the positions, for example, each block of pixels, the area corresponding to the parallax information 106, which the second correction information generating unit 21 uses to generate the second correction information 201, referred to as a “peripheral parallax reference area”.

4 is a view illustrating a first example of the parallax information 106 used to generate the second correction information 201. In other words is 4 a diagram illustrating a first example of determining the peripheral parallax reference range. 4 represents a process of calculating the parallax of the parallax generation target block 801 in the current frame 800, which is the last photographed image. The grid in 4 is a block in units for calculating parallax. The size of the block can be 1 pixel, 4 pixels with a width of 2 pixels and a height of 2 pixels, 16 pixels with a width of 4 pixels and a height of 4 pixels, a square with 5 or more pixels on one side or a Be a rectangle that has any number of pixels.

In the example, which in 4 As shown, the parallax of the upper left block in the drawing is first calculated, the processing target is sequentially moved to the right as indicated by an arrow, then the processing target is moved to the second stage when all the parallaxes in the top stage are calculated and here too the processing target is shifted sequentially from left to right. When the parallax of the block indicated by reference numeral 801 has been calculated, the calculation of all parallaxes of the illustrated first to third stages is thus completed, and the fourth stage calculation is also completed on the left side of the parallax generation target block 801. The shaded area in 4 is a region which has five pixels on each side and which is centered on the parallax generation target block 801 in which the parallax has already been calculated at the time of calculating the parallax of the parallax generation target block 801. The parallax information 106 in this area is referred to as “peripheral reference parallax 802”. In the example of 4 the peripheral reference parallax 802 is used to generate the second correction information 201. This means, in the first example, which in 4 As shown, the peripheral parallax reference range is the peripheral reference parallax 802.

5 is a view illustrating a second example of the parallax information 106 used to generate the second correction information 201. In other words is 5 a diagram illustrating a second example of determining the peripheral parallax reference range. 5 represents three images captured at different times, where the current frame 800 is the last frame, a first previous frame 901 is a frame captured in the processing cycle immediately before, and a second previous frame 902 is a frame , which was recorded two cycles before in the processing cycle. Block 801 of the current frame 800 is the parallax generation target block, and the blocks in the previous frame corresponding to it are block 903 and block 904.

The parallax in the hatched area in the first previous frame 901 is referred to as the "first previous peripheral reference parallax 905", and the parallax in the hatched area in the second previous frame 902 is referred to as the "second previous peripheral reference parallax 906”. The first prior peripheral reference parallax 905 and the second prior peripheral reference parallax 906 are second examples of the parallax information 106 used to generate the second correction information 201. Since the previous frame has the parallax information, the parallax information has already been calculated at the time of generating the parallax generation target block 801, and is the parallax information of the perimeter of the blocks 903 and 904 in the previous block corresponding to the parallax generation target block 801. This means, in the second example, which in 5 As shown, the peripheral parallax reference range is only the first previous peripheral reference parallax 905, or both the first previous peripheral reference parallax 905 and the second previous peripheral reference parallax 906. In the example of 5 refers to two previous frames, but may also refer to just one previous frame or to three or more previous frames.

Based on the parallax information 106, the peripheral parallax continuity determination unit 50 generates second correction necessity information 500 indicating the need to generate the second correction information 301. Examples of the second correction necessity information 500 include peripheral parallax information and correction necessity information.

An example of the peripheral parallax information is an average of the parallax values in the peripheral parallax reference area. Examples of the method for calculating the parallax values include an average parallax value of the peripheral reference parallax 802, which is in 4 is shown, an average parallax value of the first previous peripheral reference parallax 905, which in 5 is shown, and an average parallax value of the first previous peripheral reference parallax 905 and the second previous peripheral reference parallax 906, which in 5 are shown on. Further, only the parallax value in the horizontal direction may be referred to, or only the parallax value in the vertical direction may be referred to. The parallax isolated in the peripheral parallax reference area can be excluded from the reference target parallax. As an example of a method of detecting an isolated parallax, there is a method of detecting an isolated parallax when a difference between a value of the detection target parallax and a value of the parallax of the extent of the detection target parallax exceeds a predetermined value.

The second correction necessity information 500 is information indicating the presence or absence of generation of the second correction information 301 in the second generation unit 51. When the second correction necessity information 500 indicates "Generation: necessary", the correction execution unit 22 corrects the similarity using the second correction information 301. When the second correction necessity information 301 indicates "Generation: not necessary", the correction execution unit 22 does not correct the similarity using the second correction information 301.

For example, the peripheral parallax continuity determination unit 50 calculates the degree of deviation in parallax in the peripheral parallax reference range and sets the second correction necessity information 500 to “Generation; not necessary” if the degree of deviation is high. Examples of the deviation degree are a difference sum of an average of parallax values in the peripheral parallax reference area and each of the parallax values, and a deviation value in the peripheral parallax reference area. To generate the degree of deviation, either only the parallax value in the horizontal direction can be referenced or only the parallax value in the vertical direction can be referred to. As another example of the method for generating the correction necessity information, the continuity of the. Parallax is calculated in the peripheral parallax reference area, and the correction need information cannot be corrected when the continuity is low. An example of continuity includes an average of the difference values between the respective parallaxes and a deviation value in the peripheral parallax reference region. To create continuity, either only the parallax value in the horizontal direction can be referenced or only the parallax value in the vertical direction can be referred to.

The second generation unit 51 generates the second correction information 301 based on the second correction necessity information 500. An example of the second correction information 301 has a correction coefficient K corresponding to the search position. The role of the correction coefficient K is discussed with reference to 6 described. A method for calculating the correction coefficient K will be described later.

6 is a diagram for explaining the correction coefficient K. A graph 1110, which in the upper part of 6 is an example of the second correction information 301, and is a set of correction coefficients K for each of the search positions. A graphic 1111, which is in the lower part of 6 is shown shows the similarity for each of the search positions and shows the same type of information as that in 3 . In 6 However, the similarity before correction is represented by a solid line, and the correction coefficient after correction is represented by a dashed line. The upper part and the lower part of 6 are described in turn.

In the upper part of 6 the horizontal axis of the graph 1110 represents the search position within the search area, and the vertical axis represents the correction coefficient K for the first similarity information 104. The correction coefficient K0 is a value in a case where the first similarity information 104 is not corrected. When the correction coefficient is smaller than the correction coefficient K0, the first similarity information 104 is corrected so that the similarity becomes larger. On the other hand, if the correction coefficient is larger than the correction coefficient K0, the first similarity information 104 is corrected such that the similarity becomes smaller.

In a case where the correction coefficient K is multiplied and used by the first similarity information 104 before correction, for example, when the correction coefficient K0 is "1", the correction coefficient K1 is a real number smaller than "1" and larger than “0”, and the correction coefficient K2 is a real number greater than “1”. In a case where the correction coefficient K is multiplied and used by the first similarity information 104 before correction, for example, when the correction coefficient K0 is "1", the correction coefficient K1 is a real number smaller than "1" and larger than “0”, and the correction coefficient K2 is a real number greater than “1”. If the correction coefficient K is added to the first similarity information 104 before correction and used, for example if the correction coefficient K0 is equal to “0”, then the Correction coefficient K1 is a positive real number, and the correction coefficient K2 is a negative real number.

In the lower part of 6 the horizontal axis of the graph 1111 represents the search position within the search area, and the vertical axis represents the similarity. A first similarity curve 1104, which is indicated by a solid line, is a similarity curve which is generated using the given similarity information 104. A second similarity curve 1105, which is indicated by a broken line, is a similarity curve created using the second correction information 1100 and the second similarity information 105 generated based on the first similarity information 104.

As shown in the graph 1110, the correction coefficient K at the search position A1 is “K0” as indicated by an element 1101 of the second correction information 301. Thus, as indicated by an element 1106 on the second similarity curve 1105, the value of the first similarity curve 1104 and the value of the second similarity curve 1105 at the search position A1 are the same.

The correction coefficient K at the search position A2 is equal to “K1”, which corresponds to a value smaller than K0, as indicated by an element 1102 of the second correction information 301. Therefore, as indicated by an element 1107 on the second similarity curve 1105, at the search position A2, the second similarity is greater than the first similarity. The correction coefficient at the search position A3 is equal to “K2”, which corresponds to a value larger than K0, as indicated by an item 1103 of the second correction information 301. Therefore, as indicated by an element 1108 on the second similarity curve 1105, at the search position A3, the second similarity is less than the first similarity.

For example, using the correction coefficient K, the correction of the first similarity information 104 is as follows.
$$\text{Sat}\left(\text{Ai}\right)=\text{Sb}\left(\text{Ai}\right)\times \text{K}\left(\text{i}\right)$$

Sb (i) is the similarity at the search position (i) on the first similarity curve 1104, Sa(i) is the similarity at the search position (i) on the second similarity curve 1105, and K(i) is the correction coefficient K on the second Correction information 1100 at the search position (i). In this case, the similarity at the search position A2 points in 6 the following relationship.
$$\text{<figure-callout id="1" label="S" filenames="DE112022001667T5\_0009.png,DE112022001667T5\_0011.png" state="{{state}}">S</figure-callout>}1=\text{<figure-callout id="0" label="S" filenames="DE112022001667T5\_0013.png,DE112022001667T5\_0015.png" state="{{state}}">S</figure-callout>}0\times \text{<figure-callout id="1" label="K" filenames="DE112022001667T5\_0009.png,DE112022001667T5\_0011.png" state="{{state}}">K</figure-callout>}1$$

The correction execution unit 22 reads the correction coefficient K for each of the search positions with reference to the second correction information 301 and corrects the first. Similarity, as in the above equation by multiplying the correction coefficient K by the similarity corresponding to the position. The correction execution unit 22 corrects the similarity using the second correction information 301 after correcting the similarity using the first correction information 200.

A specific example of a method of generating the second correction information 301 by the second generating unit 51, particularly a method of generating the correction coefficient K corresponding to the search position, will be described with reference to 7 described. 7(a) is a diagram showing a first example of calculating the correction coefficient K. The first stage indicates the parallax position, the second stage indicates the parallax value, the third stage indicates the continuity, the fourth stage indicates the function, and the fifth stage indicates the correction coefficient K. Here, if the difference from the left adjacent parallax value is less than or equal to "2", which is a predetermined threshold value, it is determined that the parallax values are continuous and the continuity increases by "1", and if the difference is greater than or is equal to the threshold, the continuity decreases by "1", with the specified continuity upper limit set to "3".

In addition, for the function, “0” means no correction, “1” means weak correction, and “2” means strong correction. In case of continuity of "0" and "1" the function is set to "0", in case of continuity of "2" the function is set to "1", and in case of continuity of "3" the function is set to “2”. In contrast to 6 For ease of representation, the correction coefficient K is given by a numerical value, where- “5” in 6 “K0” corresponds to “3” in 6 “K1” corresponds, and “4” between “K” and “K0” in 6 corresponds. The correction coefficient is set in accordance with the value of the function, where the correction coefficient K is set to "5" when the function is "0", the correction coefficient K is set to "4" when the function is "1", and the correction coefficient K is set to “3” when the function is “2”.

In the example of 7(a) The parallax values “8” are continuous at the parallax positions A to G. Therefore, the continuity increases from the original value “0” followed by the maximum value “3”. Since the parallax value at and after the parallax position H is switched to “2”, the difference of “8” is greater than the threshold value “2”, and thus the continuity at the parallax positions H, I and J is reduced by “1” and reaches a continuity of “0” at position J. Since the parallax value “2” continues thereafter, the continuity begins to increase and the maximum value “3” follows, assuming that the continuity means continuity at the value “2”. Since the function is related to the continuity and the correction coefficient K is related to the function, the value of the function varies in accordance with the continuity, and the value of the correction coefficient K also varies in accordance with the value of the function in 7 (a) .

7(b) is a diagram showing a second example of calculating the correction coefficient K. The first level indicates the parallax position, the second level indicates the parallax value, the third level indicates the continuity A, the fourth level indicates the continuity B, the fifth level indicates the maximum continuity, the sixth level indicates the function , and the seventh stage gives the correction coefficient K. The continuity A and the continuity B count a continuous state of different parallax values. Maximum continuity is a larger value of continuity A and continuity B. In this example, the function is related to maximum continuity.

In the example of 7(b) The parallax value “8” is continuous at the parallax positions A to G, the parallax value “2” is continuous at the parallax positions H to M, and the parallax value “5” is continuous at the parallax positions N to Q. Therefore, the continuity A increases at the parallax positions A to G similar to the continuity in 7(a) and remains at the maximum value of “3”. At this point, continuity B has only a parallax value and is not used, with an asterisk indicating that no value is written.

When the parallax value at the parallax position H is switched to "2", the difference from the previous parallax value "8" is greater than the threshold value "2", and thus the continuity A in this example decreases to the lower limit "0". The continuity B becomes an initial value “0” at the parallax position H, and thereafter increases as the parallax value “2” continues, and reaches the maximum value “3” at the parallax position K. As a result, when the parallax value at the parallax position N changes to “5”, the continuity B decreases and the continuity A increases to replace the continuity B. Since the maximum continuity is the maximum value of the continuity A and the continuity B, as described above, the maximum continuity is the same as the continuity A up to the parallax position G, where the value of the continuity B does not exist, and the larger value, or the Continuity B, becomes maximum continuity at parallax positions J to N. The value of the function is determined according to the maximum continuity, and the value of the correction coefficient K is determined according to the value of the function.

It should be noted that in the example given in 7 is shown, the correction coefficient K only assumes values which correspond to K0 to K1, but the correction coefficient K has values from K0 to K2 by changing the relationship between the function and the correction coefficient K as follows. For example, the correction coefficient K is set to "7", that is, to K2, when the function is "0", the correction coefficient K is set to "5", that is, to K0, when the function is "1", and the Correction coefficient K is set to “3”, that is, to K1 if the function is “2”.

In operation, the first embodiment described above has the following effects.
(1) The data processing device 1 is connected to the first image capture unit 2 and the second image capture unit 3 and produces predetermined matching units by searching. Blocks parallax information of the first image 100 captured by the first image capture unit 2 and the second image 101 captured by the second image capture unit 3. The data processing device 1 has a similarity generation unit 11, which generates a first similarity between a matching block in the first image 100 and a matching block in a search area of the second image 101 in units of matching blocks, a similarity correction unit 12, which the first similarity information 104 using at least either first correction information 200 generated based on the first similarity of any adjacent matching block within the search area of the second image 101, and/or second correction information 301 generated based on parallax information 106 to generate the second similarity information 105, and a similarity determination unit 13, which generates the parallax information 106 based on the second similarity information 105. Thus, it is possible to avoid erroneous association and improve the calculation accuracy of parallax by correcting the similarity using the first correction information 200 and the second correction information 301.

(2) The first image capture unit 2 and the second image capture unit 3 generate a first image 100 and a second image 101, respectively, in predetermined processing cycles. Like in the 4 and 5 As shown, the data processing device 1 has a second correction information generation unit 21 which generates the second correction information, the second correction information being the correction coefficient K which corresponds to the similarity based on the parallax information 106 in the peripheral area of the matching block in the last first image 100 and in the last second image 101, or in the first image 100 of a predetermined previous cycle and in the second image 101 of a predetermined previous cycle is to be multiplied. Thus, when the last photographed image is used, the correction coefficient K can be calculated based on the parallax information 106 which has no position shift, and when the previously photographed image is used, the correction coefficient can be calculated using the parallax information of the entire frame.

(3) As in the search position A2 in 6 The second correction information 301 has a correction coefficient for making a correction to increase the first similarity corresponding to the parallax information in the peripheral area when the continuity of the parallax information in the peripheral area is greater than or equal to the threshold value. Thus, it is possible to avoid erroneous matching and improve the calculation accuracy of the parallax by increasing the similarity of the matching block, which has a parallax similar to its surroundings and is expected to be similar to its surroundings.

(4) As in the search position A3 in 6 the second correction information 301 has a correction coefficient for making a correction to reduce the first similarity corresponding to the parallax information in the peripheral area when the continuity of the parallax information in the peripheral area is lower than or equal to the threshold value. Thus, it is possible to avoid erroneous matching and improve the calculation accuracy of the parallax by reducing the similarity of the matching block, which has a parallax not similar to its surroundings and makes matching difficult.

(5) The correction execution unit 22 of the similarity correction unit 12 corrects first similarity information 104 based on the first correction information 200, and then additionally corrects the first similarity information 104 based on the second correction information 301 to generate the second similarity. Thus, the data processing device 1 can use the result of the first correction for correcting the local similarity for the second correction for correcting the similarity in a large area.

(6) The first image capture unit 2 and the second image capture unit 3 are arranged in the baseline direction. The first correction information generating unit 20 is provided, which scans in the baseline direction to search for a turning point at which the time is chen of the difference in similarities, calculates a new similarity using the similarity around the turning point, and calculates a combination of the position of the turning point and the new similarity as the first correction information 200. The similarity can therefore be corrected more finely than with the similarity calculation unit. For example, in a case where the similarity is calculated for each pixel, the search position 1003 has a fractional coordinate value, whereby it can be said that the similarity can be calculated with the precision of sub-pixels.

(7) The monitoring system S has the data processing device 1, the first image capture unit 2 and the second image capture unit 3. Thus, in the monitoring system S, there is little misallocation, and the calculation accuracy of parallax is high.

(First modified example)

In the first embodiment described above, the similarity correction unit 12 of the data processing device 1 includes the first correction information generation unit 20 and the second correction information generation unit 21. However, the similarity correction unit 12 can also have only one of the units from the group comprising the first correction information generation unit 20 and the second correction information generation unit 21.

(Second modified example)

In the first embodiment described above, since the first image capture unit 2 and the second image capture unit 3 are arranged side by side, the baseline direction is horizontal, and the first generation unit 41 scans the similarity in the transverse direction, as shown in FIG 3 shown. However, the first image capture unit 2 and the second image capture unit 3 can also be arranged one above the other or offset diagonally from one another. In this case, the first generation unit 41 may sample the similarity in the baseline direction, that is, along the epipolar line.

-Second embodiment-

A second embodiment of a monitoring system is described with reference to 8th described. In the following description, the same reference numerals denote the same components as in the first embodiment, and the differences will be primarily described. The points that are not described in more detail correspond to the first embodiment. The present embodiment differs from the first embodiment mainly in that a correction information cancellation unit is provided.

8th is a configuration diagram of a correction execution unit 22A in the second embodiment. The correction execution unit 22A includes a correction information cancellation unit 60 and a correction unit 61. The correction unit 61 has a first correction unit 70 and a second correction unit 71.

In the first embodiment, the first similarity information 104, the first correction information 200 and the second correction information 301 are input to the correction execution unit 22, while in the present embodiment, the correction execution unit 22A is further inputted with an annulment instruction 602 in addition to the above three items of information. The correction execution unit 22A performs the correction process of the first similarity information 104 based on the cancellation instruction 602, the first. Correction information 200 and the second correction information 301 to generate the second similarity information 105.

The correction information cancellation unit 60 may cancel at least one of the first correction information 200 and the second correction information 301 based on the cancellation instruction 602. For example, the correction information invalidation unit 60 may invalidate both the first correction information 200 and the second correction information 301, may invalidate only one or the others, or may invalidate none of them. Examples of the cancellation instruction 602 include information indicating the validity or invalidity of the first correction information 200 and information indicating the validity or invalidity of the second correction information 301. That is, the correction information cancellation unit 60 receives the output of the first correction information 200 from the first correction information generation unit 20 and the output of the second correction information 301 from the second correction information generation unit 31 as the second valid correction information 601.

When it receives the cancellation instruction 602 indicating that the first correction information 200 is valid and the second correction information 301 is invalid, the correction information cancellation unit 60 performs the following process, for example. Namely, the correction information cancellation unit 60 gives the first ones outputs correction information 200 as the first valid correction information 600, and outputs the second valid correction information 601 as the information obtained by canceling the second correction information 301.

The correction unit 61 performs the correction process of the first similarity information 104 based on the first valid correction information 600 and the second valid correction information 601 to generate the second similarity information 105. As an example of the correction process on the first valid correction information 600, there is a method of generating the second similarity information 105 by replacing the similarity at the search position to be corrected or by inserting the similarity at the new search position using the first correction information 200. As an example for the correction process on the second valid correction information 601, there is a method of generating the above with reference to 6 described second similarity curve 1105. Further, the correction process using the first correction information 200 is not performed if the first correction information 200 in the first valid correction information 600 has been canceled. Further, the correction process using the first correction information 200 is not performed if the first correction information 200 in the second valid correction information 601 has been canceled.

The first correction unit 70 performs the correction process of the first similarity information 104 based on the first valid correction information 600 to generate transition similarity information 700. The second correction unit 71 performs the correction process of the transition similarity information 700 based on the second valid correction information 601 to generate the second similarity information 105. Since the correction process in the first correction unit 70 and the correction process in the second correction unit 71 are described above with reference to FIG 3 and 6 correspond, the details of the same have been omitted.

In accordance with the second embodiment described above, a corresponding correction can be properly used.

-Third Embodiment-

A third embodiment of a monitoring system is described with reference to 9 described. In the following description, the same reference numerals denote the same components as in the first embodiment, and the differences will be primarily described. The points that are not described in more detail correspond to the first embodiment. The present embodiment differs from the first embodiment mainly in that the limit of parallax is detected and the second correction information is not generated.

9 is a configuration diagram of a similarity correction unit 12B in accordance with the third embodiment. The similarity correction unit 12B includes a second correction information generation unit 21B instead of the second correction information generation unit 21, and also includes a parallax boundary detection unit 30. The parallax boundary detection unit 30 refers to the calculated parallax information 106, and outputs the parallax boundary information 300 to the second correction information generation unit 21B when the parallax boundary is detected in the matching block. For example, to detect the presence or absence of the parallax boundary in the matching block, one of the following three methods can be used.

The first method is a method of individually checking the parallax of each of the blocks in the matching block in the current frame. Specifically, the parallax of each of the blocks in the matching block is compared with the parallax of the other blocks on the right, left, top and bottom sides, and if the difference is greater than or equal to a predetermined threshold, the determination is made that a parallax limit is arranged here. In any of the blocks, the determination is made that there is no parallax limit there if the difference to the parallax of the other blocks on the right, left, upper and lower sides is less than a predetermined threshold value. Since the current individual image is used, the most recent information is used in this first method, although no difference can be evaluated compared to a block for which the parallax has not yet been calculated.

The second method is a method of individually checking the parallax of each of the blocks in the matching block in the previous frame acquired immediately before. The difference from the first method is that the frame is a target and the parallax has been calculated for all of the blocks in the previous frame, which means that with the second method the parallax difference can be calculated even for the block in which the parallax lax difference cannot be calculated using the first method. The second method assumes that there is no large change in the position of the object in the time period of a frame, and embodies the idea that the advantage of being able to calculate the parallax difference in all of the blocks is greater than the loss due to the change in position of the object.

A third method is a method of detecting the parallax boundary by hierarchical search. This method can be applied to the current frame, as in the first method, or can be applied to the previous frame, as in the second method. In hierarchical search, parallax is first calculated with a block that has a larger size than that of the matching block in a hierarchy, for example, twice the size of that of the matching block in both the vertical and horizontal directions. In the following description, this block is referred to as a large-sized block.

Next, in the second hierarchy, the parallax of each of the blocks in the matching block is calculated based on the parallax of the large-sized block. In particular, a sum of three parallaxes is to be specified as parallax candidates, which has the parallax of the large-format block which has the target block for which the parallax is to be calculated, and the parallaxes of the one on the right adjacent to the relevant large-format block and of the one on the left of the relevant large-format one Block adjacent large-sized block, and the most probable parallax within a predetermined range is set among the parallax candidates as the parallax to be calculated. Using the parallax thus calculated by the hierarchical search, it is determined whether the difference from the parallax of the other blocks on the right, left, top and bottom sides is greater than or equal to a predetermined threshold, and will be Presence or absence of the parallax limit is determined.

When the parallax boundary is located in the matching block, the peripheral parallax continuity detection unit 50, which has received c from the parallax boundary detection unit 30, generates the second correction necessity information 500 indicating that the generation of the second correction information 301 is not necessary, and transmits the second correction necessity information 500 to the parallax boundary detection unit 30 second generation unit 51. Since the second generation unit 51, which has received the second correction necessity information 500, does not generate the second correction information 301, the correction execution unit 22 does not correct the first similarity information 104 based on the second correction information 301.

In operation, the third embodiment described above has the following effect.
(8) The data processing device 1 includes the parallax boundary detection unit 30, which generates the parallax boundary information 300 based on the luminance information or the parallax information of the first image 100 or the second image 101 in the peripheral area. The second correction information generation unit 21 cancels the generation of the second correction information 201 based on the parallax boundary information 300. Thus, when a parallax boundary is arranged in the environment, the correction of similarity by the second correction information 301 is not performed, and the adverse effect of the correction of similarity by the second correction information 301 can be avoided in advance.

(Modified Example of Third Embodiment)

In the third embodiment described above, the parallax limit detection unit 30 determines the presence or absence of the parallax limit using the parallax information 106. However, the parallax limit detection unit 30 can easily and easily determine the presence or absence of the parallax limit by using the presence or absence of a change in luminance the first image 102 or the second image 103 is used. The parallax boundary detection unit 30 compares the luminance of each of the blocks in the matching block with the luminance of other blocks on the left, right, upper and lower sides with reference to the first image 102 or the second image 103 and determines that a parallax boundary is located , if the difference is greater than or equal to a predetermined threshold. In any one of the blocks, the determination is made that there is no luminance limit there if a difference to the luminance of the other blocks on the right, left, upper and lower sides is less than a predetermined threshold value.

In the embodiment described above and the modified examples, the design of the functional block is just an example. Some functional configurations, which are shown as separate functional blocks, may be designed in one piece, or a configuration which shown in a functional block diagram may be divided into two or more functions. Additionally, another functional block may include some of the functions of each of the functional blocks.

In each of the embodiments described above and each of the modified examples, the program is stored in ROM (not shown), but the program may also be stored in a non-volatile storage device (not shown), such as flash memory. In addition, the data processing device may have an input/output interface (not shown), and the program may, if necessary, receive data from another device via a medium that can be used by the input/output interface and the data processing device read in. Here, the medium refers, for example, to a storage medium that can be removed from the input/output interface, or a communication medium, which means a wired, wireless or optical network or a carrier wave or a digital signal which is transmitted through the network. Some or all of the functions implemented by the program may be implemented by hardware circuitry or a field programmable gate array (FPGA).

The embodiments and modified examples described above can be combined with each other. Although various embodiments and modified examples have been described above, the present invention is not limited to their contents. Other aspects which may be considered within the scope of the technical idea of the present invention are also included within the scope of the present invention.

Reference number list

S

Surveillance system

1

Data processing device

10

Input unit

11

Similarity generation unit

12, 12B

Similarity correction unit

13

Similarity determination unit

20

first correction information generation unit

21, 21B

second correction information generation unit

22, 22A

Correction execution unit

30

Parallax limit detection unit

31

second correction information generation unit

40

Correction determination unit

41

first generation unit

50

peripheral parallax continuity determination unit

51

second generation unit

60

Correction information cancellation unit

61

Correction unit

70

first correction unit

71

second correction unit

QUOTES INCLUDED IN THE DESCRIPTION

This list of documents listed by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• JP 2016039618 [0003]

Claims (9)

Data processing device, which is connected to a first image capture unit and a second image capture unit and by searching in units of predetermined matching blocks parallax information of a first image captured by the first image capture unit and a second image captured by the second image capture unit, generated, the data processing device having: a similarity generating unit that generates a first similarity to a matching block in a search area of the second image with respect to a matching block in the first image in matching block units; a similarity correction unit that calculates the first similarity using at least either first correction information generated based on the first similarity of any adjacent matching block within a search area of the second image and/or second correction information generated based on the parallax information, corrected to create a second similarity; and a similarity determination unit that generates parallax information based on the second similarity. Data processing device according to Claim 1 , wherein the first image capture unit and the second image capture unit generate the first image and the second image, respectively, for each predetermined processing cycle, and the data processing device further comprises a second correction information generation unit which generates the second correction information, the second correction information being a correction coefficient which corresponds to the first similarity is to be multiplied based on the parallax information in a peripheral area of the matching block in the last first image and the last second image, or in the first image of a predetermined previous cycle and in the second image of a predetermined previous cycle. Data processing device according to Claim 2 , wherein the second correction information includes a correction coefficient for making a correction to increase the first similarity corresponding to the parallax information in the circumferential region when the continuity of the parallax information in the circumferential region is greater than or equal to a threshold value. Data processing device according to Claim 2 , wherein the second correction information includes a correction coefficient for making a correction to reduce the first similarity corresponding to the parallax information in the circumferential region when the continuity of the parallax information in the circumferential region is less than or equal to a threshold value. Data processing device according to Claim 2 , further comprising a parallax boundary detection unit that generates parallax boundary information based on luminance information of the first image in the peripheral area or the parallax information, wherein the second correction information generation unit cancels the generation of the second correction information based on the parallax boundary information. Data processing device according to Claim 1 , wherein the similarity correction unit corrects the first similarity based on the first correction information, and further corrects the first similarity based on the second correction information to generate the second similarity. Data processing device according to Claim 1 , wherein the first image capture unit and the second image capture unit are arranged in a baseline direction, and the data processing device further comprises a first correction information generation unit which scans in the baseline direction to search for a turning point at which the sign of a difference in similarities turns around, a new one Similarity is calculated by using the similarity around the turning point, and calculating a combination of a position of the turning point and the new similarity as the first correction information. Monitoring system, comprising: the data processing device according to Claim 1 ; the first image capture unit; and the second image capture unit. Parallax calculation method, which is carried out by a data processing device connected to a first image capture unit and a second image capture unit, and by searching in units of predetermined matching blocks parallax information of a first image captured by the first image capture unit and a second image, which has been captured by the second image capture unit, the method comprising: generating a first similarity with an over matching block in a search area of the second image with respect to a matching block in the first image in matching block units; Correcting the first similarity using at least one of first correction information generated based on the first similarity of any adjacent matching block within a search area of the second image and second correction information generated based on the parallax information by a second to create similarity; and generating parallax information based on the second similarity.

Images