JP2019128350A - Image processing method, image processing device, on-vehicle device, moving body and system - Google Patents

Abstract

To provide a range-finding method that can integrate range-finding methods, making use of respective advantages of the range-finding methods more than a conventional way.SOLUTION: An image processing method, which generates a distance image for measuring a distance to an object, has the steps of: detecting, by a range-finding unit 110, distance information to an object; and conducting, by an image processing unit 120, integration processing of the distance information with respect to the matching evaluation value of the pixel corresponding to a location in which the object having the distance information detected by the range-finding unit exists of a pixel including a matching evaluation value of a stereo image, and generating the distance image, in which the image processing unit is configured to conduct the integration processing before generating the distance information from the matching evaluation value.SELECTED DRAWING: Figure 4

Description

  The present invention relates to a distance measuring method, a distance measuring apparatus, an in-vehicle apparatus, a moving body, and a system.

  Conventionally, as a parallax calculation algorithm for calculating a parallax value for each pixel in a stereo camera, a block matching method of feature points, a SGM (Semi-Global-Matching) propagation method, and the like are known. These methods are methods for calculating the cost for each parallax when the feature points of the left and right images are searched, obtain the parallax value giving the minimum cost in the search parallax space as integer parallax, and use a predetermined calculation method. The parallax d including the estimated sub-pixel parallax is calculated, and the distance corresponding to each pixel is calculated by the correspondence equation (Z = BF / d) of the parallax d and the distance Z. In other words, it can be said that the cost is cast in the parallax space (B is the distance between cameras, and F is the focal length).

  In these conventional parallax space cost voting methods, it is known that it is difficult to ensure distance resolution in a far region where integer parallax is small (that is, the distance Z is large). Therefore, in the far area, the variation (dispersion) of the parallax calculation result tends to be large, and the variation in the distance measurement value also tends to be large.

  However, in consideration of the fact that a distance measuring system is mounted on a car, for example, there is an increasing tendency that distance measuring performance at a distance is required as represented by automatic driving. Therefore, LIDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging) with low spatial resolution but high distance resolution is used to integrate the measurement results of the stereo camera (high spatial resolution but low far-distance resolution) and LIDAR ( Attempts to do this (sometimes called fusion) are known. Integration allows the development of sensors that exceed the limits of the stereo camera and LIDAR measurement principles. That is, there is a possibility that a distance image with small distance dispersion and high resolution can be output even at long distances. For example, distance resolution higher than before, dispersion of object distance measurement low, separation of discontinuous surface And, it is expected that improvement of environmental robustness etc. becomes possible.

  There is a method of associating LIDAR distance information with a distance image of a stereo camera as integration of measurement results of LIDAR and stereo camera (see, for example, Patent Document 1). Patent Document 1 discloses a fusion method of increasing the density of a parallax image of an image region with few textures using distance information of LIDAR.

  However, the technology disclosed in Patent Document 1 has a problem that it may not be possible to say that integration is performed by taking advantage of the respective ranging methods. Conventionally, after the stereo camera outputs a distance image by block matching or the like, an integration method is generally used in which the distance information measured by LIDAR is added later. In this integrated method, since many errors are originally included in the distance image of the stereo camera which imaged distant, even if it added the distance information of LIDAR, there was a limit in accuracy improvement.

  That is, LIDAR with a low spatial resolution but a high distance resolution and a stereo camera with a high spatial resolution but a low far-distance resolution are not sufficiently integrated, and it may not be possible to take advantage of each other's advantages.

  An object of the present invention is to provide a distance measuring method which can be integrated by utilizing the advantages of the respective distance measuring methods in comparison with the prior art.

  In view of the above problems, the present invention is an image processing method for generating a distance image for measuring a distance to an object, the distance measuring unit detecting distance information to the object, and a stereo image Among the pixels including the matching evaluation value, the image processing unit detects the matching evaluation value of the pixel corresponding to the position at which the object for which the distance information is detected by the distance measuring unit, and the distance measuring unit detects And a step of generating the distance image by integrating the distance information, wherein the image processing unit performs the integration process before generating the distance image from the matching evaluation value. .

  A distance measuring method with improved distance accuracy in at least a far region can be provided.

It is an example of the figure explaining the ranging performance of a LIDAR and a stereo camera. It is an example of the figure explaining distance resolution and spatial resolution. It is an example of the figure explaining the distance Z obtained when a general parallax space is used. It is an example of the figure explaining the conventional integration method of the distance information of LIDAR and the distance image of a stereo camera. It is an example of the figure explaining the integration method of the distance information of LIDAR and the distance image of a stereo camera in this embodiment. It is an example of the figure which shows the ranging system mounted in the motor vehicle which is an example of a mobile body. It is a figure which shows an example of the ranging system with which the laser radar ranging part and the stereo image calculating part were comprised separately. It is an example of the figure explaining the irradiation range of the laser beam by the laser radar ranging part. It is a figure for demonstrating the imaging range of the stereo image by a stereo image calculating part. It is an example of the figure explaining the method to produce | generate a stereo image from a monocular camera. It is an example of a figure explaining the relationship between the irradiation position of the laser beam by a laser radar ranging part, and the pixel position of the stereo image (reference image) imaged by the stereo image calculating part. It is an example of the functional block diagram of a laser radar ranging part. It is an example of the functional block diagram which shows the function of a stereo image calculating part in block form. It is the figure which showed the example which calculates SAD as a cost of the pixel p = (Px3, Py5) which is focused in the reference | standard image imaged with the right camera, and the comparison image imaged with the left camera. It is a figure which shows an example of cost C (p, d) for every parallax of a certain attention pixel. It is the figure which showed typically the process which calculates propagation cost Lr using a SGM propagation system. It is an example of the flowchart figure explaining the operation | movement procedure of a ranging system. It is a figure which shows an example of stereo matching cost _CST (p, Z). It is a figure which shows an example of LIDAR cost _CLI (p, Z). It is an example of the figure for supplementing and explaining LIDAR cost _CLI (p, Z). LIDAR cost _C LI (, p Z) to the stereo matching cost _C ST (p, Z) is an example of a diagram illustrating the vote. It is an example of the figure explaining the number of pixels of an object surface. It is an example of the figure explaining calculation of the pixel number xL in the distance Z by the resolution of a laser, yL. It is an example of a figure explaining the distance in which an object side includes an irradiation side. It is a figure explaining the relationship between an irradiation surface and an object surface in each of the case of a long distance and the case of a short distance. It is an example of the figure explaining the calculation method of energy cost S (p, Z). It is a figure which shows typically the calculation method of energy cost S (p, Z). It is an example of the figure explaining the calculation method of the distance where energy cost S (p, Z) becomes the minimum. It is an example of the figure explaining the process of multipulse. It is a figure which shows the model of a transmission wave, a reception wave, and a reflected wave. It is a figure which shows the frequency of a transmitting wave and a receiving wave typically. It is an example of the figure which shows the waveform of the wave of a FMCW system transmission wave and a reception wave, and a beat signal. It is a figure explaining the direction of an object. It is a figure which shows the reference | standard image for demonstrating a condition. It is a figure which shows the comparative example of a bird's-eye view map. It is an example of the figure explaining the creation method of a bird's-eye view map. It is a figure which shows the reference | standard image for demonstrating a condition. It is a figure which shows the comparative example of a bird's-eye view map. It is a figure which shows the reference | standard image at night when the headlight was imaged. It is a figure which shows the comparative example of a bird's-eye view map. It is a figure which shows the reference | standard image for demonstrating a condition. It is a figure explaining the suppression result of the expansion which arises with a SGM propagation system. LIDAR cost _C LI (p, Z) distance component cost _C LD (p, Z) of the which is an example of a diagram schematically showing. It is an example of the figure explaining how the LIDAR cost _CLI (p, Z) of the spatial component and the distance component cost _CLD (p, Z) are integrated into the stereo matching cost _CST (p, Z). . It is an example of the flowchart figure which shows the example of control of ECU when there exists abnormality in a laser radar ranging part or a stereo image calculating part. It is a figure which shows the example of a display when abnormality arises in a laser radar ranging part or a stereo image calculating part. It is a figure which shows typically the irradiation position of image data and a laser beam. It is an example of the figure explaining the example of integration of the light reception level matched with distance information, and the stereo matching cost _CST (p, Z). It is an example of the schematic block diagram of a distance image provision system. It is an example of a functional block diagram of a distance image providing system.

  Hereinafter, a distance measuring system and an image processing method performed by the distance measuring system will be described as an example for carrying out the present invention.

<LIDAR and distance measurement performance of stereo camera>
First, the distance measurement performance of the LIDAR and the stereo camera will be described with reference to FIG. FIG. 1A shows the relationship between the distance Z and the distance resolution by comparing the LIDAR and the stereo camera. Note that the smaller the distance resolution, the higher the performance. As shown, LIDAR exhibits a substantially constant distance resolution even if the distance Z is large, but the stereo camera rapidly increases the distance resolution as the distance Z increases. Therefore, the distance measurement accuracy of the stereo camera is greatly reduced in the far area.

  FIG. 1 (b) shows the relationship between the distance Z and the spatial resolution in LIDAR. The smaller the spatial resolution, the higher the performance. In FIG. 1 (b), the spatial resolution is shown at every irradiation interval (irradiation resolution) of 0.1 degree, 0.2 degree, 0.3 degree, 0.4 degree, 0.5 degree and 0.6 degree. In either case, the distance Z increases as the distance Z increases.

  On the other hand, FIG. 1 (c) shows the relationship between the distance Z and the spatial resolution in a stereo camera. In the stereo camera, the space can be decomposed for each pixel, so the spatial resolution is high, and the distance Z does not increase so much.

From the above, the following improvement points are extracted.
・ Shooting accuracy of stereo camera: Since the distance resolution becomes coarser as the distance increases, it is difficult to perform distance measurement and object detection.
Stereo camera object resistance: Repeated patterns or low textures cause incorrect matching, and many distance values with large dispersion (variation) occur.
・ Environmental resistance of stereo cameras: Distance calculation becomes difficult at night.
・ LIDAR: Low spatial resolution.

  The ranging system 100 of the present embodiment improves these improvements. In addition, as will be described later, when a distance image is obtained for an image data of a stereo camera by an algorithm called an SGM (Semi-Global-Matching) propagation method, there is a problem that an object boundary is lost or an object region is expanded. is there. In the present embodiment, it is also possible to suppress the disappearance of the boundary of the object and the expansion of the boundary that are likely to occur in this SGM propagation method.

<Terminology>
Finding corresponding points is called matching, and the degree of matching is called cost. The cost is an “evaluation value” of the degree of matching, and is also expressed as “dissimilarity” or “similarity”. If the dissimilarity is low, the index indicates that they are more consistent, and if the similarity is high, the index indicates that they are more consistent. “Dissimilarity” and “similarity” may be collectively expressed as “matching”.

  The resolution will be described with reference to FIG. FIG. 2 is a diagram for explaining the distance resolution and the spatial resolution. Distance resolution is the ability to identify differences in distance to an object. When the distance resolution in the Z direction is, for example, 10 cm, an object 12 cm from the origin and an object 21 cm (only 9 cm is different) cannot be identified, and are determined to be the same object (same distance).

  Spatial resolution is the ability to identify differences in distance in any direction in two dimensions. If, for example, the spatial resolution is 5 cm in the X and Y planes, two objects separated by only 4 cm can not be identified and it is determined to be the same object. Spatial resolution may be referred to as angular resolution.

  The ranging direction is the direction in which the distance to the object is measured, and is the direction of the object. The pixels specified by the ranging direction include surrounding pixels in addition to the pixels.

The distance evaluation value regarding distance information is an evaluation value set according to the uncertainty of the distance detected by electromagnetic waves etc. about the pixel specified by the irradiation direction of electromagnetic waves, and a surrounding pixel. In the present embodiment, the term LIDAR cost C _LI (p, Z) is used. The matching evaluation value is the degree of matching in block matching. In the present embodiment, the term stereo matching cost C _ST (p, Z) is used.

<Outline of Ranging System of This Embodiment>
FIG. 3 is an example of a diagram for explaining the distance Z obtained when using a general parallax space. FIG. 3A shows the cost C (p, d) and the propagation cost Lr (p, Z) with the shift amount as the horizontal axis, obtained by the block matching and SGM propagation method. In FIG. 3A, the search range is 64 pixels. p is the target pixel, and d is the shift amount (search parallax) between the reference image and the comparison image. The smallest cost C (p, d) or the propagation cost Lr (p, Z) in the search range of 64 pixels is adopted as the parallax (integer parallax) of the pixel p of interest.

FIG. 3 (b) shows the cost C (p, d) or the propagation cost Lr (p, Z) in the Z space. The distance Z is obtained from the parallax d in FIG.
Z = BF / d (1)
B is the distance between the optical axes of the left and right cameras in the stereo camera, and F is the focal length of the left and right cameras. As shown in FIG. 3 (b), in the Z space, a density occurs in the distance Z at which the cost C (p, d) or the propagation cost Lr (p, Z) is obtained. This is because d is included in the denominator of the equation (1) for calculating the distance Z, the distance Z is inversely proportional to d, and when d is close to 0, the distance Z changes significantly.

  Therefore, general block matching is equivalent to coarse cost propagation on the long distance side, and it becomes difficult to obtain high accuracy at long distance.

  FIG. 4 is an example of a diagram for explaining a conventional integration method of the distance information of LIDAR 9 and the distance image of the stereo camera 8. Conventionally, it has been common to use an integrated method in which the distance information measured by the LIDAR 9 is added later after the stereo camera 8 outputs a distance image by block matching or the like. In this integration method, since many errors are included in the distance image of the stereo camera 8 in the first place as described in FIG. 3, there is a limit to improvement in accuracy even if LIDAR distance information is added.

  Therefore, in the present embodiment, as shown in FIG. 5, the distance information measured by the LIDAR 9 is integrated before the stereo camera 8 outputs the distance image by block matching or the like. FIG. 5A is an example for explaining a method for integrating the distance information of the LIDAR 9 and the distance image of the stereo camera 8 in the present embodiment. The stereo camera 8 integrates the distance information output by the LIDAR 9 into the cost C (p, d) before outputting the distance image.

  Then, as shown in FIG. 5B, in this integration, the stereo camera 8 calculates the cost C (p, Z) in the Z space. FIG. 5B shows an example of the cost C (p, Z) and the propagation cost Lr (p, Z) in the Z space. A Z space with low density is prepared in advance, and the cost of LIDAR is voted for the distance Z among the costs C (p, Z) calculated by the stereo camera 8 in this Z space, and the cost propagation by the SGM propagation method or the like is also possible. By performing in the Z space, the distance Z with the lowest cost in the Z space can be specified, and a distance image with high distance resolution can be obtained. In addition, since the distance image originally has a high spatial resolution, a high-density and high-resolution distance image can be obtained.

  As described above, the distance measuring system according to the present embodiment integrates the distance information measured by the LIDAR in the Z space before the stereo camera outputs the distance image by block matching or the like, thereby obtaining a high-density and high-resolution distance image. Can be realized.

<Example of application of ranging system>
An application example of the distance measurement system 100 will be described with reference to FIG. FIG. 6 is a view showing a distance measuring system 100 mounted on a mobile unit 200 which is an example of a mobile unit. In FIG. 6, the ranging system 100 is set at the center position inside the front window of the moving body 200. The ranging system 100 includes a laser radar ranging unit 110 and a stereo image calculation unit 120. The laser radar distance measurement unit 110 and the stereo image calculation unit 120 are both installed so that the front is a distance measurement range. The laser radar ranging unit 110 is disposed between (preferably, at the center of) the stereo cameras (two imaging units or imaging units) included in the stereo image calculation unit 120.

  The laser radar may be called LIDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging). In the present embodiment, both are not distinguished unless otherwise specified. Laser radar or LIDAR emits a pulse of light outside the visible spectrum using a laser diode, and calculates the distance by measuring the time when the pulse returns. The direction and distance in which a pulse at a certain moment is reflected is recorded as a point in the 3D map with the laser radar distance measuring unit 110 as the center.

  Further, FIG. 6 shows the distance measurement system 100 in which the laser radar distance measurement unit 110 and the stereo image calculation unit 120 are integrally configured, but the laser radar distance measurement unit 110 and the stereo image calculation unit 120 are separately provided. It may be configured.

  FIG. 7 shows an example of a ranging system 100 in which the laser radar ranging unit 110 and the stereo image calculation unit 120 are configured separately. In FIG. 7, the laser radar distance measuring unit 110 is mounted, for example, inside the front grille, and the stereo image calculation unit 120 is disposed in front of the vehicle interior side (for example, near the back side of the room mirror). In the present embodiment, the distance information output from the laser radar distance measuring unit 110 may be integrated with the distance information of the stereo image calculating unit 120, and the present invention is not limited to the shapes and configurations shown in FIGS.

<Laser irradiation range of laser radar range finder>
FIG. 8 is an example of a diagram for explaining the laser light irradiation range by the laser radar distance measuring unit 110. FIG. 8A is a top view of the moving body 200 as viewed from above, and FIG. 8B is a side view of the moving body 200 as viewed from the side.

  As shown in FIG. 8A, the laser radar distance measuring unit 110 irradiates a laser beam while scanning a predetermined range in front of the traveling direction of the moving body 200 in the horizontal direction. The laser light may be regarded as light or as electromagnetic waves.

  Further, as shown in FIG. 8B, the laser radar distance measuring unit 110 irradiates laser light toward a predetermined range ahead of the moving direction of the moving body 200. How far the laser reaches can be measured in a range of several hundred meters depending on the output of the laser radar ranging unit 110. Although the detection range on the near side can be detected from less than 1 meter, since the necessity for distance measurement is usually low in such a short distance region, a range for detecting the distance may be set as appropriate.

  The laser radar distance measurement unit 110 is configured to be able to scan in the horizontal direction while rotating the irradiation direction of the laser light in the elevation direction. Accordingly, it is possible to irradiate the irradiation range from the vicinity to the far side with the installation position of the laser radar distance measuring unit 110 as a reference.

<Imaging range of stereo image computing unit>
FIG. 9 is a diagram for explaining a stereo image capturing range by the stereo image calculation unit 120. A stereo image refers to both a reference image and a comparative image. FIG. 9A is a top view of the moving body 200. In the stereo image computing unit 120, two imaging devices (cameras) are installed with the optical axis facing forward in the traveling direction of the moving body 200, and pick up an image in a predetermined range in the traveling direction. The irradiation range of the laser light and the imaging range of the stereo camera at least partially overlap.

  FIG. 9 (b-1) and FIG. 9 (b-2) show a reference image (image data of the right camera) and a comparison image (image data of the left camera) captured by the stereo camera. The right camera 11 and the left camera 12 are horizontally installed at predetermined intervals. For this reason, the reference image and the comparison image have overlapping portions, but the position of the object in the captured image is shifted in the left-right direction.

  The stereo image calculation unit 120 generates and outputs a distance image by calculating the amount of displacement of each object of the reference image and the comparison image (this becomes parallax). The stereo image computing unit 120 also associates distance information with the pixels of the stereo image.

  Alternatively, either the right camera 11 or the left camera 12 may be omitted, and a stereo image may be acquired with a monocular camera. A method for generating a stereo image from a monocular camera will be described with reference to FIG.

  As shown in FIG. 10A, the stereo image calculation unit 120 includes a right camera 11 and a left camera 12. First, a large number of reference images and comparison images are prepared, and the comparison images are learned from the reference images by deep learning (deep learning), or the reference images are learned from the comparison images. Below, it demonstrates as what learns a comparison image from a reference | standard image.

  Each pixel value of the reference image is input to an input layer of a DNN (Deep Neural Network) 121. The middle layer is constructed by combining one or more convolutional layers, one or more pooling layers, a neural network, and an encoder decoder network. The intermediate layer is represented as a coefficient of a two-dimensional filter. The output layer outputs each pixel value of the estimated comparison image. The difference between the pixel value output from the output layer and the pixel value of the actual comparison image is reflected (learned) on the coefficients of the two-dimensional filter by the error back propagation method. Adjusting the coefficients of the two-dimensional filter with a sufficient number of reference images and comparison images corresponds to learning of the DNN 121. The initial value of the two-dimensional filter may be learned by an auto encoder.

  FIG. 10B shows an example of the monocular camera calculation unit 122 and the DNN 121 that are mounted on the vehicle after learning. At the stage of being mounted on the vehicle, the monocular camera operation unit 122 outputs only the reference image. The DNN 121 is an output device that outputs a comparison image. The reference image is input to the DNN 121, and the estimated comparison image is output. Although the comparison image obtained in this way is different from the comparison image captured by the left camera 12, it is actually confirmed that the quality is such that a distance image can be generated. The monocular camera operation unit 122 performs block matching with the comparison image estimated as the reference image.

  Therefore, in the present embodiment, any one of the right camera 11 and the left camera 12 may be provided, and it is not essential to mount a stereo camera. In other words, a stereo image is an essential requirement, and means for generating a stereo image is not limited.

<Relationship between laser beam irradiation position and pixel position of stereo image>
Next, the relationship between the irradiation position of the laser light by the laser radar distance measurement unit 110 and the pixel position of the stereo image (reference image) captured by the stereo image calculation unit 120 will be described using FIG. FIG. 11A is an example for explaining the relationship between the irradiation position of the laser light and the pixel position of the stereo image (reference image).

  The laser irradiation direction by the laser radar ranging unit 110 and the pixel position of the reference image can be associated in advance. FIG. 11A shows two objects O1 and O2 viewed from the side, and FIG. 11B is an example of a reference image in which the objects O1 and O2 are captured. Since the objects O2 and O1 are on a straight line with respect to the optical axes of the laser radar distance measuring unit 110 and the right camera 11, they are imaged in an overlapping manner.

  Assume that the height h2 of the object O2 is exactly twice the height h1 of the object O1, and the distance L2 of the object O2 from the moving body 200 is exactly twice the distance L1 of the object O1 from the moving body 200 (FIG. Then, it is almost doubled due to drawing). Since the sizes and distances of the objects O1 and O2 are proportional to each other, the objects O1 and O2 appear in the same size in the image data, and overlap due to the positional relationship between the objects O1 and O2 and the moving body 200. Therefore, when the laser light passes through the upper ends of the objects O1 and O2, the laser light should appear on the upper ends of the objects O1 and O2 of the reference image captured by the stereo image calculation unit 120 (actually because it is not visible light). Not reflected). As described above, regardless of the distance to the object, the irradiation direction of the laser light and the pixel position of the reference image are in one-to-one correspondence, so both can be associated in advance.

  FIG. 11C shows distance information corresponding to the pixels P1 to P4 of the reference image. For example, the pixel of P1 (x1, y1) corresponds to the irradiation direction of the horizontal direction θ1 and the elevation angle φ1, and the pixel of P2 (x2, y2) corresponds to the irradiation direction of the horizontal direction θ2 and the elevation angle φ2, and P3 (x3, The pixel y3) corresponds to the irradiation direction of the horizontal direction θ3 and the elevation angle φ3, and the pixel P4 (x4, y4) corresponds to the irradiation direction of the horizontal direction θ4 and the elevation angle φ4.

  For this reason, when the irradiation direction and the distance information are output from the laser radar distance measuring unit 110, the stereo image calculation unit 120 can associate the measured distance information with the pixel.

<Functional Configuration of Laser Radar Rangefinder>
Next, FIG. 12 is an example of a functional configuration diagram of the laser radar distance measuring unit 110. The laser radar ranging unit 110 includes a signal processing unit 601, an elevation direction scan drive unit 602, a motor 603, an elevation direction scan mirror 604, a laser light receiving unit 605, a signal amplifier 606, a time interval counter 607, a laser output unit 608, and a laser driver. It has 609.

  Based on an instruction from the signal processing unit 601, the elevation direction scan drive unit 602 drives a motor 603 for rotating the elevation direction scan mirror 604 in the elevation direction. Thereby, the elevation direction scan mirror 604 rotates in the elevation direction.

  Further, based on an instruction from the signal processing unit 601, the laser driver 609 is driven and a laser beam is output from the laser output unit 608. At this time, the output timing of the laser light is temporarily held by the time interval counter 607. The laser light output from the laser output unit 608 is output to the outside through the elevation direction scan mirror 604, and thus irradiates a predetermined irradiation range.

  The laser light output to the outside is reflected by an object (target object) in the irradiation direction, and the reflected light is received by the laser light receiving unit 605 via the elevation direction scan mirror 604. The laser light receiving unit 605 has a plurality of photo detectors (PD: Photo Detector) arranged in the vertical direction, and the laser light is received by one of the photo detectors and converted into an electric signal.

  The converted electric signal is amplified by the signal amplifier 606 and input to the time interval counter 607. The time interval counter 607 calculates the time interval based on the output timing of the laser light output from the laser output unit 608 and the light reception timing of the reflected light received by the laser light receiving unit 605.

  The time interval calculated by the time interval counter 607 is converted into distance information in the signal processing unit 601, and is output to the stereo image computing unit 120 together with the information indicating the irradiation direction.

  Further, the signal processing unit 601 has an abnormality monitoring unit 601 a. The abnormality monitoring unit 601a monitors an abnormality occurring in the laser radar distance measuring unit 110. For example, it is determined that there is an abnormality when the time interval detected by the time interval counter 607 or the time when the distance information detected by the signal processing unit 601 does not change continues for a predetermined time or more. In addition, an abnormality may be detected when the distance information takes a value out of the specification for a certain period of time or when the signal processing unit 601 reaches a temperature equal to or higher than a specified value. The laser radar distance measurement unit 110 notifies the stereo image calculation unit 120 that there is an abnormality.

  The stereo image calculation unit 120 monitors the abnormality of the laser radar ranging unit 110 itself. For example, it is detected that the laser radar distance measuring unit 110 is not responsive, that communication can not be performed, that a predetermined voltage does not come from the laser radar distance measuring unit 110 (power off), and the like.

<Functional Configuration of Stereo Image Arithmetic Unit>
FIG. 13 shows the configuration of the distance measurement system 100. As shown in FIG. Further, FIG. 13 illustrates the function of the stereo image computing unit 120 in a block form. Since the ranging system 100 is a device for ranging, it can be called a ranging device. Other than this, it may be called a distance measuring device, a distance measuring unit or the like.

  As shown in FIG. 13, the stereo image calculation unit 120 includes a right camera 11, a left camera 12, a distortion correction unit 13, and a distance calculation unit 14. A stereo camera is formed by the right camera 11 and the left camera 12. In the present embodiment, a captured image captured by the right camera 11 is used as a reference image, and a captured image captured by the left camera 12 is used as a comparative image.

  The distortion correction unit 13 and the distance calculation unit 14 may be realized by using a dedicated electronic circuit, or may be realized by executing a program for realizing each unit by a CPU (computer). Therefore, the stereo image calculation unit 120 has a function of an information processing device. It is also an image processing apparatus from the viewpoint of processing an image.

  The distortion correction unit 13 performs general distortion correction on the reference image and the comparison image. By this image correction, the reference image and the comparison image are corrected so that no difference other than parallax occurs. Image correction is possible by prior calibration. When the left camera 12 and the right camera 11 are installed, for example, a subject for calibration (for example, a checkered chart) is imaged. A LUT (Look for geometric transformation) that transforms image data so as to minimize hardware internal error factors such as camera lens distortion, optical axis deviation, focal length deviation, and image sensor distortion by comparing two images. Up Table) has been generated. The distortion correction unit 13 performs image correction with reference to such an LUT.

The distance calculation unit 14 calculates parallax by applying an algorithm such as block matching or an SGM propagation method to the reference image and the comparison image. Although the details will be described later, the distance calculation unit 14 sets the distance Z of the stereo matching cost C _ST (p, Z) specified by the distance information output by the laser radar ranging unit 110 before outputting the distance image. Integrate the LIDAR cost C _LI (p, Z). The process concerning integration performed by the distance calculation unit 14 is referred to as integration process. The stereo matching cost C _ST (p, Z) is an example of a matching evaluation value, and the LIDAR cost C _LI (p, Z) is an example of a distance evaluation value.

  The distance calculation unit 14 includes an abnormality monitoring unit 14a. The abnormality monitoring unit 14a monitors an abnormality occurring in the stereo image calculation unit 120. For example, it is determined that there is an abnormality when the pixel value of the reference image or the comparison image does not change for a certain time or longer. In addition, an abnormality may be detected when the pixel value takes a value outside the specification for a certain period of time or when the distance calculation unit 14 reaches a temperature equal to or higher than a specified value. The stereo image calculation unit 120 notifies the ECU 20 that there is an abnormality. The stereo image calculation unit 120 also notifies the ECU 20 when it receives an error from the laser radar distance measurement unit 110 or when it detects it.

  The ECU 20 monitors the abnormality of the stereo image calculation unit 120 itself. For example, it is detected that the stereo image calculation unit 120 is not responding, that communication can not be performed, that a predetermined voltage is not input from the stereo image calculation unit 120 (power off), and the like.

  In FIG. 13, as an example, the distance image and the reference image are sent to the ECU 20 (Electronic Control Unit). The ECU 20 is an electronic control unit for a moving body. The in-vehicle ranging system 100 is referred to as an in-vehicle device. The ECU 20 performs various driving assistances using the distance image and the reference image output from the distance measuring system 100. For the reference image, various pattern matching is performed to recognize the state of the preceding vehicle, pedestrian, white line, traffic light, and the like.

  Driving assistance varies depending on the moving object, but for example, when the lateral position of the object overlaps with the vehicle width of the moving object that is the user, an alarm or an alarm is given according to TTC (Time To Collision) calculated from distance and relative speed. Do braking etc. If it is difficult to stop until the collision, the steering is steered in a direction to avoid the collision.

  Further, the ECU 20 performs inter-vehicle speed inter-vehicle distance control that travels following the preceding vehicle with an inter-vehicle distance corresponding to the vehicle speed. When the preceding vehicle stops, the moving body also stops, and when the preceding vehicle starts, the moving body also starts. In addition, when the ECU 20 performs white line recognition or the like, lane keeping control for steering to travel in the center of the traveling lane, departure prevention control for changing the traveling direction toward the traveling lane when there is a risk of departure from the traveling lane, etc. It can be carried out.

  Moreover, when there is an obstacle in the traveling direction when the vehicle is stopped, sudden start can be suppressed. For example, when there is an obstacle in the traveling direction determined by the operation position of the shift lever and the amount of operation of the accelerator pedal is large, damage is reduced by limiting the engine output or warning.

  A display device 21 is connected to the ECU 20. The display device 21 is a flat panel display (such as liquid crystal or organic EL) disposed on a center console or a dashboard, or a flat panel display incorporated in a meter panel. Alternatively, HUD (Head Up Display) may be used. The ECU 20 displays on the display device 21 that an abnormality has occurred in the laser radar ranging unit 110 or the stereo image calculation unit 120. A display example will be described in Example 3.

  The configuration of FIG. 13 is merely an example, and the laser radar distance measuring unit 110 and the stereo image calculating unit 120 may be integrally configured. Further, the ECU 20 may have the function of the stereo image calculation unit 120.

<Calculation of integer disparity by block matching>
Calculation of integer disparity by block matching will be described with reference to FIG. FIG. 14 shows the SAD (Sum of Absolute Difference) as the cost of the pixel of interest p = (Px3, Py5) in the reference image 420 captured by the right camera 11 and the comparison image 410 captured by the left camera 12. ) Is a diagram illustrating an example of calculating. A formula for calculating SAD is shown in (Equation 1).

  Since the reference image 420 and the comparison image 410 have different imaging positions, the same object is not pointed to even in the pixel p = (Px3, Py5) of interest at the same position on the captured image. It will point to the shifted position. Therefore, the SAD when the block size is 1 × 1 pixel, the luminance value of the focused pixel p = (Px3, Py5) on the reference image 420 and the focused pixel on the comparison image 410 The difference value from the luminance value of p = (Px3, Py5) is a large value.

  Here, the pixel p of interest on the comparison image 410 is shifted rightward by one pixel. That is, the SAD is calculated when the parallax d = 1. Specifically, the difference between the luminance value of the pixel of interest p = (Px3 + 1, Py5) on the comparison image 410 and the luminance value of the pixel of interest p = (Px3, Py5) on the reference image 420. Calculate the value. In the example of FIG. 14, the SAD also becomes a large value when d = 1.

  Hereinafter, similarly, d = 2, 3,... Are changed, and SAD is calculated in each. In the example of FIG. 14, when d = 3, the object indicated by the pixel p = (Px3, Py5) in the reference image 420 and the pixel p = (Px3 + 3, Py5) in the comparison image 410 are The object pointed to is the same. For this reason, the SAD when d = 3 is smaller than the SAD other than when d = 3.

  FIG. 15A shows an example of SAD for each parallax of a pixel p of interest. SAD is an example of cost C (p, d). Since the SAD is minimum at the pixel p of interest at d = 3, the parallax d is calculated to be 3.

  FIG. 15B shows an example of SAD for each parallax of another pixel p of interest. In FIG. 15B, since the change in SAD with respect to the change in parallax d is small, the distance calculation unit 14 cannot extract the parallax. As described above, pixels that cannot specify the parallax can be generated only by block matching. Therefore, the distance calculation unit 14 performs an energy calculation process (SGM propagation method) to make the parallax manifest.

  In addition, if SAD (cost C (p, d)) for every parallax is calculated like FIG. 15, it is suitable to obtain | require decimal parallax. There are calculation methods using high-order polynomial estimation (sixth order), high-order polynomial estimation (fourth order), parabola fitting, or the like as a method of obtaining the decimal parallax.

As described above, SAD is an operation that calculates the absolute value of the difference in luminance value and uses the sum as the correlation value. The smaller the pixel block is, the smaller the value is.

  In block matching, an arithmetic method such as SSD (Sum of Squared Difference), NCC (Normalized Cross Correlation), or ZNCC (Zero means Normalized Cross Correlation) may be employed in addition to SAD.

The SSD is an operation for calculating the square of the difference between luminance values and using the sum as a correlation value. The smaller the pixel block is, the smaller the value is.

NCC is a numerator and calculates the inner product of luminance values. If the luminance value is similar, the inner product value increases. The denominator is a numerical value for normalizing the numerical value of the numerator, and the value increases as the pixel blocks are similar. NCC takes a maximum value of 1 and a minimum value of 0. NCC is also referred to as normalized cross correlation.

ZNCC is the normalized cross-correlation after subtracting the average value. This is equivalent to removing the DC component in terms of frequency, and is effective for comparing images with different brightness. M is the number of pixels in the horizontal direction, and N is the number of pixels in the vertical direction.

  Besides this, it is also possible to use ZSAD (Zero means Sun of Absolute Difference) or ZSSD (Zero means Sum of Squared Difference). ZSAD is the SAD after subtracting the average value, and ZSSD is the SSD after subtracting the average value.

<SGM propagation method>
The distance calculation unit 14 calculates the propagation cost Lr using an algorithm called SGM propagation method, and calculates the energy cost S (p, d) of the pixel p of interest using the propagation cost Lr. Perform energy calculation processing. The SGM propagation method is a form of dense algorithm.

  First, the process of calculating the propagation cost Lr using the SGM propagation method will be described. FIG. 16 is a diagram schematically showing processing for calculating the propagation cost Lr using the SGM propagation method.

  The example of FIG. 16 shows a case where the propagation cost Lr in four directions is obtained for the pixel 1100 that is the pixel p of interest. Specifically, the case where the propagation cost L1 in the direction of the arrow 1111, the propagation cost L2 in the direction of the arrow 1112, the propagation cost L3 in the direction of the arrow 1113, and the propagation cost L4 in the direction of the arrow 1114 are obtained for the pixel 1100. It shows. The direction (r) of the propagation cost to be obtained for the pixel 1100 is not limited to four directions, and may be eight directions or two directions, for example.

  As shown in FIG. 16, the propagation cost L1 in the direction of arrow 1111 can be obtained by the following equation.

... (2)
Here, p represents the coordinates of the pixel 1100, and d represents the parallax. As described above, the propagation cost L1 (p, d) is the propagation cost at the parallax (d−1 to d + 1) of the cost C (p, d) of the pixel 1100 and the pixel located at the left one pixel of the pixel 1100. And can be calculated. That is, the propagation cost in the direction of the arrow 1111 is calculated sequentially from the left direction to the right direction. The propagation interval when propagating the propagation cost from the left direction to the right direction is not limited to one pixel. That is, the propagation cost L1 (p, d) may be calculated using the propagation cost at each parallax of the pixel located on the left side a pixel of the pixel 1100.

  Similarly, the propagation cost L2 in the direction of the arrow 1112 is sequentially calculated from the upper direction to the lower direction. Further, the propagation cost L3 in the direction of the arrow 1113 is sequentially calculated from the right direction to the left direction, and the propagation cost L4 in the direction of the arrow 1114 is sequentially calculated from the lower direction to the upper direction.

  Subsequently, in the energy calculation process, a process of calculating the energy cost S (p, d) of the pixel p of interest using the propagation cost Lr will be described.

  As described above, the energy cost S (p, d) of each pixel is calculated by the following equation based on the propagation cost from each direction calculated for each pixel.

... (3)
Therefore, in the example of FIG. 16, it can be calculated by S (p, d) = L1 (p, d) + L2 (p, d) + L3 (p, d) + L4 (p, d).

<Method of calculating distance in the present embodiment>
Subsequently, the procedure of calculating the distance value for each pixel by the distance measuring system 100 according to the present embodiment will be described with reference to FIG. FIG. 17 is an example of a flowchart for explaining the operation procedure of the distance measuring system 100.

(Steps S1 to S4)
As shown in step S1, the laser radar distance measuring unit 110 acquires distance information. In parallel with this, the right camera 11 of the stereo image calculation unit 120 captures a reference image, and the left camera 12 captures a comparison image (S2). The distortion correction unit 13 performs distortion correction for each so that there is no difference other than parallax (S3). Next, the stereo image calculation unit 120 calculates a stereo matching cost C _ST (p, Z) (S4).

  Note that steps S1 to S4 may be performed synchronously, but the stereo image calculation unit 120 may be performed asynchronously by using the latest distance information of the laser radar ranging unit 110.

FIG. 18 is a diagram illustrating an example of the stereo matching cost C _ST (p, Z). That is, the cost of the parallax space obtained by stereo matching is converted into the stereo matching cost C _ST (p, Z) of Z space. For this reason, the stereo matching cost C _ST (p, Z) cannot be obtained at equal intervals in the Z space. Therefore, the distance calculation unit 14 interpolates the cost obtained by stereo matching. In FIG. 18, the circle is the cost obtained by stereo matching, and the square is the cost obtained by interpolation. The interpolation method may be any method suitable for curve approximation, and for example, parabolic fitting, high-order polynomial, spline curve, or the like can be applied. In FIG. 18, the stereo matching cost C _ST (p, Z) is calculated, for example, every 0.5 meters by interpolation. Note that offset in FIG. 18 is a fraction for obtaining the distance Z every 0.5 meters.

(Step S5)
Next, the distance calculation unit 14 integrates the stereo matching cost C _ST (p, Z) and the LIDAR cost C _LI (p, Z) to calculate the cost C (p, Z). In the formula, it can be expressed as follows.
C (p, Z) = EST x C _ST (p, Z) + ELI x C _LI (p, Z) (4)
EST: coefficient (weight) of stereo matching cost
ELI: LIDAR cost factor (weight)
First, the LIDAR cost C _LI (p, Z) will be described with reference to FIGS. FIG. 19 is a diagram illustrating an example of the LIDAR cost C _LI (p, Z), and FIG. 20 is a diagram for supplementarily explaining the LIDAR cost C _LI (p, Z). Equation (5) shows an example of the LIDAR cost C _LI (p, Z).

... (5)
r _0: pixels corresponding to the irradiation direction of the laser p: surrounding pixels of _{r 0} containing _{r 0} sigma: LIDAR cost _C LI (p, Z) parameter defines the spread of A: LIDAR cost _C LI (p, Z) Parameter to determine the depth of the (0-1)
The inventor of the present application newly sets a LIDAR cost C _LI (p, Z) in order to integrate the distance information of the laser radar ranging unit 110 into the stereo matching cost C _ST (p, Z) before outputting the distance image. did. As shown in FIG. 19, the LIDAR cost C _LI (p, Z) is the smallest at the pixel at the center of the light ray of p = r ₀ , and becomes larger according to σ as it gets away from r ₀ . As is clear from the equation (5), when A is 1, the LIDAR cost C _LI (p, Z) of the pixel r ₀ is 0. On the other hand, LIDAR cost C _LI (p, Z) becomes 1 at peripheral pixels far from r ₀ .

On the other hand, one rectangle in FIG. 20 indicates a pixel, and the horizontal resolution and the vertical resolution of the central pixel r ₀ and one beam of laser light are shown in pixel units. That is, the range in which one light beam of laser light spreads is indicated by the number of pixels. In the range in which one light beam of laser light spreads, the horizontal direction is called horizontal resolution and the vertical direction is called vertical resolution. In FIG. 20, the horizontal resolution is 5 pixels and the vertical resolution is 9 pixels. One laser beam spreads farther away. However, the area (number of pixels) captured by the stereo camera is almost constant. This is because even if the laser beam spreads as it goes further, the far laser beam appears smaller. Therefore, the LIDAR cost C _LI (p, Z) is expressed by the equation (5) for the pixel p having a constant horizontal resolution and vertical resolution regardless of the distance Z measured by the laser radar ranging unit 110 for the center pixel r _0. What is necessary is just to calculate.

When the LIDAR cost C _LI (p, Z) is obtained by the equation (5), the distance calculation unit 14 votes the LIDAR cost C _LI (p, Z) for the stereo matching cost C _ST (p, Z). FIG. 21 is an example of a diagram illustrating voting of LIDAR cost C _LI (p, Z) to stereo matching cost C _ST (p, Z). Since the distance Z of the pixel r ₀ corresponding to the irradiation direction of the laser light is obtained by the laser radar ranging unit 110, the LIDAR cost C _LI (p, Z) is voted for this distance Z of the pixel r ₀ . Voting corresponds to the process of addition. In the case of the pixel r ₀ , the LIDAR cost C _LI (p, Z) is 1−A = 0 (when A is 1).

The distance calculation unit 14, a similar voting is performed for the pixels p included in the horizontal resolution and vertical resolution around the pixel r _0. However, the LIDAR cost C _LI (p, Z) is a value calculated by the equation (5) according to the distance between r ₀ and p.

As described above, in the present embodiment, the stereo matching cost C _ST (p, Z) and the LIDAR cost C _LI (p, Z) can be integrated in the Z space. Becomes smallest cost of pixel r _0, the cost of the surrounding pixels of the pixel r ₀ is higher than the pixel r _0, reduce the cost of the pixels r ₀ corresponding to the irradiation direction of the laser beam in the lateral resolution and vertical resolution, The cost of surrounding pixels can be increased.

The shape of the LIDAR cost C _LI (p, Z) in FIG. 19 is merely an example. For example, a cylindrical shape showing the minimum cost within a certain range from the pixel r ₀ may be used, or it may be an inverted triangle similar to FIG. Further, in the formula (4), the stereo matching cost C _ST (p, Z) and the LIDAR cost C _LI (p, Z) are added, but from the stereo matching cost C _ST (p, Z), the LIDAR cost C _LI ( p, Z) may be subtracted. Alternatively, LIDAR cost C _LI (p, Z) may be a negative value.

  The processing in step S5 will be supplemented with reference to FIGS. In the following, it will be described that the accuracy of the distance information decreases as the distance from the horizontal resolution and the vertical resolution range of the laser increases.

  First, FIG. 22 is an example for explaining the number of pixels on the object plane. FIG. 22A shows the number of pixels of a certain object, and FIG. 22B shows the correspondence between the size of the object and the number of pixels on the object surface. The object plane is a range occupied by an object in the image data.

An object having a horizontal width N [m] and a vertical width M [m] exists at a distance Z [m]. By using the focal length F, the number X of pixels in the horizontal direction and the number Y of pixels in the vertical direction in the image data are calculated. pt is a pixel pitch.
X = F · N / (Z · pt)
Y = F · M / (Z · pt)
FIG. 23 is an example of a diagram for explaining the calculation of the number of pixels xL and yL at the distance Z based on the resolution of the laser. The resolution of the laser is horizontal θx [deg] and vertical θy [deg].
The width A of the irradiation surface of the laser is A = Ztan (θx / 2) × 2
The vertical width B of the laser irradiation surface is A = Ztan (θy / 2) × 2
The number of pixels xL occupied by the width A on the image is as follows.
xL = F · A / Z = Ztan (θx / 2) × 2 = 2 (F / pt) tan (θx / 2) [pixel]
The number of pixels yL occupied by the vertical width B on the image is as follows.
yL = F · B / Z = Ztan (θy / 2) × 2 = 2 (F / pt) tan (θy / 2) [pixel]
As is clear from the calculation results of xL and yL, the irradiation surface spreads according to the distance Z, but the number of pixels is constant regardless of the distance Z on the image. However, this is a case where it is assumed that the emission size of the laser beam is a point light source to facilitate the description. When the emission area of the laser has a certain size, the corresponding pixel size is large on the short distance side, and converges to a constant value as it goes further.

From these, when the laser beam is applied to the object surface, the distance that the object surface includes the irradiation surface can be obtained. FIG. 24 is an example of a diagram illustrating the distance in which the object plane includes the irradiation surface.
Since X> xL and Y> yL,
F · N / (Z · pt)> 2 (F / pt) tan (θx / 2) and F · M / (Z · pt)> 2 (F / pt) tan (θy / 2)
Therefore, N / (2 tan (θ x / 2)> Z and M / (2 tan (θ y / 2)> Z)
Is a conditional expression of the distance in which the object surface includes the irradiation surface.
・ In the case of mobile
N = M = 1.8 m θx = θy = 0.6 [deg] to 171.9> Z
· In the case of objects N = 0.25 M = 1.6 m θ x = 0.1 θ y = 0.6 [deg] to 143.2> Z
Therefore, at a long distance, as shown in FIG. 24, the object surface becomes larger than the irradiation surface, and it is uncertain where the object surface is reflected. This will be described in more detail with reference to FIG.

FIG. 25 is a view for explaining the relationship between the irradiation surface and the object surface in each of the long distance and the short distance. FIG. 25 (a) shows the case of a short distance, and the object plane is larger than the irradiation plane as described in FIG. When the laser irradiation surface is converted into the number of pixels, it is only the size of xL and yL, so it is not known where it is reflected on the object surface XY. This means that the reliability of the distance information measured by the laser radar distance measuring unit 110 decreases as the distance from the irradiation surface increases. For this reason, as described with reference to FIG. 19, it is effective to increase the LIDAR cost C _LI (p, Z) (reduce the reliability) around the irradiation surface of the laser.

  FIG. 25B shows that the irradiation surface is larger than the object surface at a long distance. In this case, the laser may not be reflected on the object plane represented by XY, and may possibly escape backward. In this case, there is a possibility that a plurality of objects can be detected because of so-called multi-pulse. Processing when multipulses occur will be described later.

(Step S6)
Next, the distance calculation unit 14 calculates the propagation cost Lr (p, Z). The formula for calculating the propagation cost Lr (p, Z) is shown below.

... (6)
The first term of the propagation cost Lr (p, Z) is the cost C (p, Z) obtained by integrating the stereo matching cost C _ST (p, Z) and the LIDAR cost C _LI (p, Z) calculated in step S5. ). The second term is the propagation cost by the SGM propagation method performed in the Z space. The propagation cost Lr (p, Z) is calculated from the first and second terms.

  In the present embodiment, the propagation cost according to the SGM propagation method performed in the Z space is not essential, and the propagation cost according to the SGM propagation method may not be calculated.

(Step S7)
The distance calculation unit 14 determines whether the propagation cost Lr (p, Z) has been calculated for all the pixels. Steps S5 and S6 are repeatedly executed until the processing of all the pixels is completed.

(Step S8)
When the propagation cost Lr (p, Z) is calculated for all pixels, the distance calculation unit 14 calculates the energy cost S (p, Z).
S (p, Z) = ΣLr (p, Z) (7)
FIG. 26 is an example of a diagram illustrating a method for calculating the energy cost S (p, Z). Although the propagation cost Lr (p, Z) was calculated for each pixel, it is considered that the surrounding propagation cost Lr (p, Z) affects a certain pixel. Therefore, the more accurate propagation cost Lr (p, Z) of the pixel of interest is calculated by superimposing the propagation costs Lr (p, Z) around the pixel of interest.

In FIG. 26, the propagation costs Lr (p, Z) of the surrounding 8 pixels are superimposed. That is,
S (p, Z) = L ₀ (p, Z) + L ₄₅ (p, Z) + L ₉₀ (p, Z) + L ₁₃₅ (p, Z) + L ₁₈₀ (p, Z) + L ₂₂₅ (p, Z) + L ₂₇₀ (p, Z) + L ₃₁₅ (p, Z), ... (8)
FIG. 27 is a diagram schematically illustrating a method for calculating the energy cost S (p, Z). As shown in Expression (8), the propagation costs Lr (p, Z) of 8 pixels around the pixel of interest are overlapped in the Z space. Thereby, the energy cost S (p, Z) of the pixel of interest is obtained.

  Note that the superposition of the propagation costs Lr (p, Z) of the surrounding 8 pixels is merely an example, and the propagation costs Lr (p, Z) of how many pixels around the pixels, such as 4 pixels, 5 pixels, or 16 pixels, are overlaid. This may be determined in consideration of the calculation load and the accuracy of the distance. Further, it is not necessary to superimpose the propagation cost Lr (p, Z) at all.

(Step S9)
The distance calculation unit 14 determines the distance Z ₀ that minimizes the energy cost S (p, Z). The distance Z ₀ is the distance value of the pixel of interest.

  FIG. 28 is an example of a diagram illustrating a method for calculating the distance Z0 at which the energy cost S (p, Z) is minimized. It is estimated that Z at which the energy cost S (p, Z) is minimum is most likely as the distance value of the pixel of interest.

  Note that the high-order polynomial estimation (sixth order), the high-order polynomial estimation (fourth order), parabola fitting, or the like may be used to calculate the distance Z of a few or less.

(Step S10)
When the calculation is performed for all the pixels, the process of FIG. 17 ends.

<About multi-pulse>
The process in the case where the multi-pulse described in FIG. 25 (b) is generated will be described. FIG. 29 is an example of a diagram for explaining multi-pulse processing. FIG. 29A is a view of the laser irradiation range of the laser radar distance measuring unit 110 as viewed from above, and FIG. 29B is an example of a diagram for explaining the light reception level with respect to the distance Z.

  As shown in FIG. 29A, when the irradiation surface of the laser is larger than the object surface, the laser radar distance measuring unit 110 receives reflected light from a plurality of objects O1 and O2 by multi-pulse. As shown in FIG. 29B, when the light reception level is graphed with respect to the distance Z, the light reception level becomes large at the distances Z1 and Z2 at which the objects O1 and O2 exist. Therefore, it is detected that there are a plurality of objects. Generally, the laser radar distance measuring unit 110 determines that the light receiving level is an object at a distance equal to or greater than the threshold, so in the situation as shown in FIG. 29, one light ray can detect the distance between two objects at different distances.

Thus, when the distance between two objects at different distances is obtained with one ray, the distance calculation unit 14 calculates the stereo matching cost C _ST (p, Z) and the LIDAR cost C _LI (p, Z) for each. Integrate. That is, the LIDAR cost C _LI (p, P, calculated by the equation (5) is added to the distances Z1, Z2 between the stereo matching cost C _ST (p, Z) of the pixel r ₀ corresponding to the irradiation direction of the laser light and the surrounding pixels. Add Z).

Conventionally, it has been difficult to integrate the two distances Z1 and Z2 into one target pixel. However, in this embodiment, in order to integrate in the Z space, when multiple pulses occur, the two distances are appropriately set. It can be handled. In FIG. 29, it has been described that the expression (5) is applied as it is. However, for example, when a plurality of distances are detected, the LIDAR cost C _LI (p, Z) of the expression (5) is more determined in consideration of the ambiguity of the distance. You may multiply by the coefficient which becomes large.

<Another example of laser ranging>
The laser radar distance measuring unit 110 of the present embodiment described with reference to FIG. 12 is a TOF (Time Of Flight) type distance measuring method. On the other hand, there are distance measuring methods called FCM (Fast Chirp Modulation) and FMCW (Frequency Modulation Continuous Wave) methods. FCM and FMCW are systems that convert the frequency of a beat signal of a mixed wave, which is generated when the frequency is slightly different between a transmission wave and a reception wave, into a distance.

  A distance measurement method based on the FCM method will be described with reference to FIGS. FIG. 30 shows models of a transmission wave, a reception wave, and a reflected wave. A reflected wave obtained by reflecting the transmission wave transmitted from the millimeter wave transmitting / receiving apparatus 25 on the object 26 is received as a reception wave. The distance between the millimeter wave transceiver 25 and the object is R.

  FIG. 31 schematically shows the frequencies of the transmission wave and the reception wave. In the signal used in the FCM system, the frequency is linearly increased with the passage of time. FIG. 31 (a) shows the transmission wave and the reception wave whose frequency gradually increases. Such a signal whose frequency changes with time is called chirp. As shown in FIG. 31A, the frequencies of the transmission wave 27 and the reception wave 28 increase with time while the amplitude remains constant. Note that the received wave is measured with a delay of Δt, which is the time to return after being reflected by the object 26. For this reason, the frequencies of the transmission wave 27 and the reception wave 28 are slightly different, and a beat signal is generated.

  FIG. 31 (b) shows changes in frequency components of the transmission wave 27 and the reception wave 28 with respect to time. The reception wave 28 is measured with a delay of Δt after the transmission of the transmission wave 27 is started. Since the transmission wave 27 increases at a constant speed with respect to time, the frequency shift when there is a delay of Δt is constant at Δf. The time during which the frequency is changing is T, and the frequency difference between the minimum frequency and the maximum frequency is F. T and F are known (control values). Therefore, if Δf can be observed, it can be converted to Δt. If Δt is known, it can be converted to the distance to the object.

  Since the frequencies of the transmission wave 27 and the reception wave 28 are shifted by Δf, a signal is generated in which the transmission wave 27 and the reception wave 28 are superimposed. The beat signal is a large vibration that includes fine vibrations. If the frequency Δf is constant, the frequency of the large vibration is also constant. Also, it is known that the frequency of the beat signal is equal to the frequency Δf.

  Since the beat signal is subjected to Fourier transform (preferably fast Fourier transform) to obtain a frequency spectrum in which a power peak appears at the frequency of the signal, the beat signal may be subjected to Fourier transform to detect the frequency Δf. FIG. 31 (c) schematically shows the frequency Δf obtained by Fourier transformation.

Next, a method for converting Δf into distance R will be described. First, the distances R and Δt have the following relationship. However, C is the speed of light in the air.
Δt = 2R / C (9)
Next, Δt and Δf have the following relationship as is apparent from the diagram of FIG. 31B: Δt: Δf = T: F
By transforming this, the following equation is obtained.
Δf = F × Δt / T (10)
Substituting equation (9) into equation (10) yields:
Δf = 2RF / CT (11)
R = CTΔf / 2F (12)
Therefore, the distance R can be converted by substituting Δf obtained by the Fourier transform into the equation (12).

  FMCW is a method of continuously performing FCM, and the principle is similar to that of FCM. FIG. 32 shows the waveform 29 of the frequency of the transmit wave 27 and the receive wave 28 with respect to time in the FMCW system. In the FMCW system, the frequency transitions repeatedly relatively slowly. The FCM is considered to be higher in the relative speed recognition ability and the multi-target recognition ability.

<Direction of object detected by millimeter wave radar>
Next, an object direction detection method in the FCM method will be described with reference to FIG. FIG. 33 is a diagram for explaining the direction of an object. FIG. 33A shows a top view of an example of the millimeter wave transmitter / receiver 25 and the object 26. In the FCM method, an angle θ of a reflected wave reflected using a horizontal plane is estimated. For example, assuming that the traveling direction of the moving object is 0 degrees, θ is the angle (arrival angle) of the object 26 in the horizontal direction.

FIG. 33B shows a method for calculating θ using a plurality of receiving antennas 31. The angle θ of the received wave 28 can be detected by an array antenna. In FIG. 33 (b), N reception antennas 31 are arranged on a straight line at intervals d. When the reception wave 28 arrives from the angle θ, the arrival propagation becomes a plane if the distance between the object 26 and the reception antenna 31 is sufficiently large. The path difference l (el) of the received wave received by each receiving antenna 31 can be expressed by an integer multiple of dsinθ. Since the path difference appears as a delay r of the reception wave 28, the angle θ can be estimated if the delay amount of the reception wave 28 received by each reception antenna 31 is known.
l = d sin θ
r = l / C = (d / C) sin θ
θ = arcsin (r · C / d) (13)

<About effects>
Below, the effect of the ranging system 100 of this embodiment is demonstrated using an actual experimental result.

<< Experimental result 1 >>
FIG. 34 shows a reference image for explaining the situation. FIG. 34A is a reference image, and FIG. 34B is an enlarged view of the center. A moving body 301, a person 302, and a chart 303 (a signboard indicating the traveling direction, etc.) are shown near the center (far away) of the reference image. The actual measured distance is as follows.
Mobile body: 68.5m
Person: 80m
Chart: 80 m
FIG. 35 shows a comparative example of the overhead view map. FIG. 35A shows an overhead map created based on the distance obtained by block matching, and FIG. 35B shows an overhead map created based on the distance measured by the distance measuring system 100 of the present embodiment. Show. In this embodiment, since the distance measurement accuracy and the surface separation are improved, the moving body 301 and the charts (persons) 302 and 303 are separated. However, the person and the chart at the same distance can not be separated. Although this is a future improvement, the gate 304 that is 100 meters away and the storeroom 305 that is 75 meters away are separated.

The eyelid map is supplemented using FIG. FIG. 36 is an example of a diagram illustrating a method for creating an overhead map. FIG. 36 shows a distance image, and when the distance of pixel coordinates (x, y) of the distance image is z, the parallax d (x, y) of the pixel coordinates (x, y) is
d (x, y) = B × F / z
Is calculated by

Assuming that the center coordinates of the distance image are (x ₀ , y ₀ ), real space coordinates (X, Y, Z) are
X = (x-x ₀ ) x B / d (y, x);
Y = (y-y ₀ ) x B / d (y, x);
Z = B x F / d (y, x) (= z);
The distance calculation unit 14 performs this calculation on each pixel of the distance image for which an overhead map is obtained. Since the eyelid map as shown in FIG. 36 is two-dimensional, if one vote is cast in the mesh space corresponding to (X, Z), the eyelid map shown in FIG. 36 is obtained. If one vote is cast in the mesh space corresponding to (X, Y, Z), a three-dimensional map is obtained.

<< Experimental result 2 >>
FIG. 37 shows a reference image for explaining the situation. FIG. 37A is a reference image, and FIG. 37B is an enlarged view of the center. A plurality of moving bodies 311 and 312 are shown near the center (far away) of the reference image. The actual measured distance is as follows.
Moving body 1: 55 m
Moving body 2: 78 m
FIG. 38 shows a comparative example of the overhead view map. FIG. 38A shows an overhead map created based on the distance obtained by block matching, and FIG. 38B shows an overhead map created based on the distance measured by the distance measuring system 100 of the present embodiment. Show. In the present embodiment, since the distance measurement accuracy and the surface separation are improved, the two moving bodies 311 and 312 are clearly separated. The gate 100m ahead and the storeroom 75m away are separated.

<< Experimental result 3 >>
FIG. 39 shows a reference image at night in which the headlight is imaged. FIG. 39A shows the entire reference image and the processing range of the distance measuring system 100, and FIG. 39B is an enlarged view of the headlight portion of the moving body 321.

  FIG. 40 shows a comparison example of the flooding map. 40A shows an overhead map created based on the distance obtained by block matching, and FIG. 40B shows the overhead map created based on the distance measured by the distance measuring system 100 of the present embodiment. Show. In block matching, the positions of three-dimensional objects are unclear. On the other hand, the distance measuring system 100 according to the present embodiment separates other three-dimensional objects (moving bodies 311 and 322, storage 305, and gates 304) including the moving body 321 at a long distance of 60 m or more even at night It becomes possible to detect.

<< Experimental result 4 >>
An experimental result when the distance of the chart is measured at each distance of 80 m and 30 m will be described.
FIG. 41 is a reference image for explaining the situation. Although a color distance image is actually obtained, color information is lost in the patent drawing, and therefore no distance image is posted. As an alternative, the evaluation result of the surface accuracy is shown as a numerical value by “distance average” and “distance standard deviation” in the surface of the chart 303.

Table 1 shows the detection results of the distance of the chart 303 according to the present embodiment, the SGM propagation method, and the block matching. In both 80 m and 30 m, the distance measuring system 100 of this embodiment can confirm a large improvement in accuracy in all of the average, variance, and standard deviation.

<< Experimental result 5 >>
FIG. 42 is a diagram for explaining a result of suppressing expansion that occurs in the SGM propagation method. 42A shows a reference image, FIG. 42B shows a distance image calculated by the distance measuring system 100 of the present embodiment, and FIG. 42C shows a distance image calculated by the SGM propagation method. In FIG. 42 (c), the distance between the two legs of the chart is the distance of the chart due to the expansion, but in the distance measuring system 100 of the present embodiment shown in FIG. 42 (b), the distance of the background is calculated between the two legs of the chart. It is done.

  In the experiment shown in FIG. 42, the laser radar distance measuring unit 110 in which the irradiation light of the laser is stacked in multiple layers in the vertical direction is used.

<Summary>
As described above, the distance measuring system 100 according to the present embodiment integrates the distance information measured by the LIDAR in the Z space before the stereo camera outputs a distance image by block matching or the like, thereby achieving high density and high resolution. It is possible to realize a wide range image.

  It supplements about the effect of this embodiment. For example, in the related art such as JP-A-2015-143679, the "parallax image" obtained by block matching is integrated in the distance space in the present embodiment, while the distance information measured by LIDAR in the parallax space is integrated.

  In the method of integrating in the parallax space, although the distance resolution of LIDAR is high and the distance accuracy is ensured, the distance resolution becomes coarse especially in the distance, and the parallax space propagation processing is performed using the distance obtained by the coarse distance resolution In sub-pixel estimation. For this reason, distance accuracy can not be secured, and improvement in performance can not be expected.

  On the other hand, the method of integrating in the metric space as in the present embodiment ensures the high distance resolution of LIDAR, and the cost curve and the LIDAR are fused in the metric space. It is possible to generate a dense distance image.

  Due to this difference, in this embodiment, even in a distance range corresponding to a parallax value close to 0, which is the limit of the measurement principle of a stereo camera, for example, at a distant distance of about 100 m, a correct distance and distance dispersion of the object plane occur. No distance image can be generated. In addition, it also becomes possible to restore a wide range of 3D space even in 3D space restoration.

  That is, when viewed from a stereo camera, the distance measurement accuracy is improved, the distance of the object surface is reduced, and the surface separation of the object surface is improved. In addition, it is possible to calculate a more correct distance (object resistance) by integrating the pixels whose cost is low due to repetitive patterns and false matching with a low texture. In addition, it is possible to measure the distance with high accuracy even at night (environmental resistance). In addition, expansion of the object boundary caused by the SGM propagation method is suppressed. In addition, LIDAR also has the effect of increasing spatial resolution.

In this embodiment, a ranging system 100 that integrates stereo matching cost C _ST (p, Z) and LIDAR cost C _LI (p, Z) in consideration of the ambiguity of the ranging value of the laser radar will be described.

  As described in the first embodiment, the accuracy of the distance information decreases as the laser light is extended in the horizontal resolution and the vertical resolution range. In the first embodiment, this is dealt with by increasing the cost of the peripheral portion by the equation (5), but in this embodiment, this is dealt with by considering the ambiguity in the distance information of the laser radar ranging unit 110.

That is, in the first embodiment, the LIDAR cost C _LI (p, Z) is calculated from only the spatial component (the accuracy of the xy plane), but the distance component (the accuracy in the Z direction) is also considered in the present embodiment. In this embodiment, the LIDAR cost C _LI (p, Z) is defined as follows, and a method of calculating the LIDAR cost C _LI (p, Z) in consideration of the distance component cost C _LD (p, Z) will be described. .
LIDAR cost C _LI (p, Z) ∝space component × distance component (14)
The space component may be the equation (5) of the first embodiment. The distance component cost C _LD (p, Z) can be defined as follows as an example.

... (15)
γ: laser distance measurement value Z: deviation from laser distance measurement value v: parameter for determining the spread of the distance component β: parameter for determining the depth of the distance component (0 to 1)

FIG. 43 is a diagram schematically illustrating the distance component cost C _LD (p, Z) of the LIDAR cost C _LI (p, Z) represented by the equation (15). As shown in the figure, the distance component cost of the LIDAR cost C _LI (p, Z) is the smallest among the distance measurement values of the laser, and increases as the deviation from the distance measurement value of the laser increases. This may cause the laser distance measurement value to deviate from the true value depending on the influence of the reflectance of the object and where the object falls. In this case, the object surface that originally forms the surface is split and detected. By including the ambiguity v in the distance component of LIDAR cost C _LI (p, Z) as in equation (15), it is possible to handle the distance measurement value of the laser with a width, so it is easy to suppress the fragmentation. Become.

Note that the shape of the distance component cost C _LD (p, Z) in FIG. 43 is merely an example. For example, a rectangle indicating the minimum cost within a certain range from the LIDAR distance γ may be used, or similar to FIG. 43 but an inverted triangle may be used.

FIG. 44 shows how the spatial component LIDAR cost C _LI (p, Z) and the distance component cost C _LD (p, Z) are integrated into the stereo matching cost C _ST (p, Z) according to equation (15). It is a figure explaining what is done.

A. Among the stereo matching costs C _ST (p, Z) of the pixel r ₀ corresponding to the irradiation direction of the laser radar ranging unit 110, the LIDAR distance γ has the smallest LIDAR cost C _LI (p, Z) and the smallest distance component. The product multiplied by the cost C _LD (p, Z) is added.

B. Among the stereo matching costs C _ST (p, Z) of the pixel r ₀ corresponding to the irradiation direction of the laser radar ranging unit 110, the smallest LIDAR cost C _LI (p, Z) is at a distance Z that is ΔZ away from the LIDAR distance γ. ) And the distance component cost C _LD (p, Z) calculated by the equation (15) according to ΔZ are added.

C. Of the stereo matching costs C _ST (p, Z) of the pixel p around the pixel r ₀ corresponding to the irradiation direction of the laser radar ranging unit 110, the LIDAR distance γ is expressed by an expression (in accordance with the distance between r ₀ and p ( The product of LIDAR cost C _LI (p, Z) calculated in 5) and the smallest distance component cost C _LD (p, Z) is added.

D. Among the stereo matching costs C _ST (p, Z) of the pixel p around the pixel r ₀ corresponding to the irradiation direction of the laser radar ranging unit 110, a distance Z that is ΔZ away from the LIDAR distance γ is set to r ₀ and p. A product obtained by multiplying the LIDAR cost C _LI (p, Z) calculated by the equation (5) according to the distance and the distance component cost C _LD (p, Z) calculated by the equation (15) according to ΔZ. Is added.

<Summary>
As described above, in addition to the effects of the first embodiment, the distance measuring system 100 of the present embodiment can further prevent the object plane from being split.

In this embodiment, a fail-safe process is performed when an abnormality occurs in the distance measuring system 100, and the distance measuring system 100 capable of displaying that an abnormality has occurred will be described. As described with reference to FIGS. 12 and 13, the ECU 20 can detect an abnormality in the laser radar ranging unit 110 and the stereo image calculation unit 120.
When there is an abnormality in the laser radar distance measuring unit 110, the laser radar distance measuring unit 110 can notify the stereo image calculating unit 120 or the stereo image calculating unit 120 can detect and notify the ECU 20. The ECU 20 can continue the driving assistance by the distance image generated by the stereo image calculating unit 120.
When there is an abnormality in the stereo image calculation unit 120, the ECU 20 can detect whether the stereo image calculation unit 120 has notified the ECU 20 or the ECU 20 can detect it. The ECU 20 can continue driving assistance based on the irradiation direction and distance information output by the laser radar distance measuring unit 110.

  However, when the abnormality mode is an abnormality in which the ECU 20 cannot communicate with the stereo image calculation unit 120, the ECU 20 cannot acquire the distance information of the laser radar ranging unit 110 via the stereo image calculation unit 120. For this reason, in the case of an abnormality in which the ECU 20 cannot communicate with the stereo image calculation unit 120, the ECU 20 may acquire the irradiation direction and distance information directly from the laser radar distance measuring unit 110.

  In this way, if either the laser radar distance measuring unit 110 or the stereo image calculation unit 120 is normal, the integration cannot be performed, but the ECU 20 can assist driving using the distance image or the distance information.

  FIG. 45 is an example of a flowchart illustrating a control example of the ECU 20 when there is an abnormality in the laser radar ranging unit 110 or the stereo image calculation unit 120. The process of FIG. 45 is repeatedly performed while the moving object is traveling.

  First, the ECU 20 determines whether or not an abnormality has been detected in both the laser radar distance measuring unit 110 and the stereo image calculation unit 120 (S101). The abnormality may be detected by the laser radar ranging unit 110, detected by the stereo image calculation unit 120, or detected by the ECU 20.

  When the determination in step S101 is Yes, the ECU 20 displays on the display device 21 that there is an abnormality in the laser radar distance measurement unit 110 and the stereo image calculation unit 120 (S102). A display example is shown in FIG.

  If the determination in step S101 is No, the ECU 20 determines whether or not an abnormality of the laser radar distance measuring unit 110 has been detected (S103). The abnormality may be detected by the laser radar distance measurement unit 110 or may be detected by the stereo image calculation unit 120.

  If the determination in step S103 is YES, the ECU 20 performs driving assistance using only the distance image of the stereo image calculation unit 120 (S104). Since the ECU 20 assists the driving with the distance image before the detection of the abnormality, the process is not changed.

  Next, the ECU 20 displays on the display device 21 that there is an abnormality in the laser radar distance measuring unit 110 (S105). A display example is shown in FIG.

  Subsequently, the ECU 20 determines whether an abnormality of the stereo image calculation unit 120 is detected (S106). The abnormality may be detected by the stereo image calculation unit 120 or may be detected by the ECU 20.

  If the determination in step S106 is YES, the ECU 20 performs driving assistance based on only the distance information of the laser radar distance measuring unit 110 (S107). That is, driving assistance is started not based on the distance image but based on the direction (irradiation direction) to the object and the distance.

  Next, the ECU 20 displays on the display device 21 that there is an abnormality in the stereo image calculation unit 120 (S108). A display example is shown in FIG.

  By doing so, the ECU 20 can continue to support driving even if an abnormality occurs in the laser radar distance measuring unit 110 or an abnormality occurs in the stereo image calculation unit 120.

  FIG. 46 is a diagram showing a display example when an abnormality occurs in the laser radar distance measurement unit 110 or the stereo image calculation unit 120.

  FIG. 46 (a) shows that the laser radar distance measuring unit 110 and the stereo image calculating unit 120, which are displayed on the display unit 21 when abnormality occurs in the laser radar distance measuring unit 110 and the stereo image calculating unit 120 A display example of this is shown. In FIG. 46 (a), “Warning: An abnormality has occurred in the radar and camera sensor. Driving support will be terminated.” Is displayed. As a result, the occupant of the moving body can grasp that the driving support is not continued because the laser radar ranging unit 110 and the stereo image calculation unit 120 are abnormal.

  FIG. 46B shows an example of a display indicating that an abnormality has occurred in the laser radar ranging unit 110 displayed on the display device 21 when an abnormality has occurred in the laser radar ranging unit 110. In FIG. 46 (b), “Warning radar has an abnormality. Driving assistance is continued with the camera sensor.” Is displayed. Thereby, the occupant of the moving body can grasp that the driving support is continued although the laser radar ranging unit 110 has an abnormality.

  FIG. 46 (c) shows a display example that an abnormality has occurred in the stereo image calculation unit 120 that is displayed on the display device 21 when an abnormality has occurred in the stereo image calculation unit 120. In FIG. 46 (c), “Warning has occurred in the camera sensor. The radar will continue driving support.” Is displayed. As a result, the occupant of the mobile object can recognize that the driving assistance is continued although there is an abnormality in the stereo image computing unit 120.

In the present embodiment, a ranging system 100 will be described, which partially performs voting for the stereo matching cost C _ST (p, Z) of the LIDAR cost C _LI (p, Z). In FIGS. 20 and 21 of the first embodiment, the LIDAR cost C _LI (p, Z) is compared with the stereo matching cost C _ST (p, Z) of the surrounding pixels centering on the pixel r ₀ corresponding to the irradiation direction of the laser beam. Voted.

In this embodiment, as shown in FIG. 47, the voting of the LIDAR cost C _LI (p, Z) is not performed at the pixel r ₀ corresponding to the irradiation direction of the laser beam and all the surrounding pixels, but the LIDAR cost C _LI This is performed according to the state of (p, Z) and stereo matching cost C _ST (p, Z).

FIG. 47 is a diagram schematically showing the irradiation position of image data and laser light. Two irradiation positions 401 and 402 are shown as an example in FIG. The laser beam spreads on a part of pixel blocks of the image data. The stereo matching cost C _ST (p, Z) at the irradiation position 401 is gentle as shown in the figure (there is no extreme value), and the stereo matching cost C _ST (p, Z) at the irradiation position 402 is convex downward ( Extreme value is clear). Therefore, it is difficult to find the minimum value of the stereo matching cost C _ST (p, Z) at the irradiation position 401, but it is easy to find the minimum value at the irradiation position 401.

FIG. 48 is an example for explaining an integrated example of the light reception level associated with the distance information and the stereo matching cost C _ST (p, Z). FIG. 48A shows an example of voting for the stereo matching cost C _ST (p, Z) at the irradiation position 401, and FIG. 48B shows voting for the stereo matching cost C _ST (p, Z) at the irradiation position 402. An example is shown. In FIG. 48A, the minimum value of the received light level associated with the distance information is clear. For this reason, the minimum value is clarified by integrating the received light level into the stereo matching cost C _ST (p, Z).

On the other hand, in the stereo matching cost C _ST (p, Z) as shown in FIG. 47 (b), voting is effective when the received light level becomes a multi-pulse. In FIG. 48 (b), two minimum values are obtained for the light reception level. Such a waveform can occur when a plurality of objects reflect one laser beam. In FIG. 48 (b), the minimum value is clarified by integrating the received light level into the stereo matching cost C _ST (p, Z).

Therefore, the stereo image calculation unit 120 controls the presence / absence of voting as follows. Incidentally, this integration is performed for the pixels r ₀ and around the pixels corresponding to the irradiation direction of the laser beam in the same manner as in Example 1.
When the stereo matching cost C _ST (p, Z) is flat and the light reception level of the LIDAR is one, the light reception level of the LIDAR and the stereo matching cost C _ST (p, Z) are integrated.
When the LIDAR light reception level is a multi-pulse and the stereo matching cost C _ST (p, Z) has one minimum value, the LIDAR light reception level and the stereo matching cost C _ST (p, Z) are integrated.

In other words, when the stereo matching cost C _ST (p, Z) is flat and the light reception level of the LiDAR is a multi-pulse, it is difficult to obtain an integration effect, so that voting can be omitted. If the minimum value of stereo matching cost C _ST (p, Z) is clear and the light reception level of LIDAR is clear, there is a high possibility that the object has already been reliably captured, so voting is omitted. Yes (in this case, it may be integrated with the meaning of confirmation).

Whether or not the stereo matching cost C _ST (p, Z) is flat can be determined, for example, by comparing the difference between the minimum value and the maximum value with a threshold value. If this difference is sufficiently large, it can be said that the local minimum is clear. Whether or not it is multi-pulse can be determined, for example, based on whether or not there are two or more minimum values whose difference from the maximum value is equal to or greater than a threshold.

In the description of FIG. 48, the expression “integrating the LIDAR light reception level and the stereo matching cost C _ST (p, Z)” is used to explain the waveform of the LiDAR light reception level. It has the same meaning as voting the cost C _LI (p, Z) for the stereo matching cost C _ST (p, Z).

  In this embodiment, a distance image providing system in which the server device executes at least a part of processing will be described. The processing performed by the distance measurement system 100 in the first to fourth embodiments can also be performed by the server device.

  FIG. 49 shows a schematic configuration diagram of the distance image providing system 50. As shown in FIG. As shown in FIG. 49, the distance measurement system 100 mounted on the mobile unit 200 communicates with the server device 51 via the network N. The distance measuring system 100 transmits the distance information, the irradiation direction, the reference image, and the comparison image to the server device 51, and the server device 51 performs the processing of the first to fourth embodiments to generate a distance image and transmits it to the distance measuring system 100. To do.

  FIG. 50 is an example of a functional block diagram of the distance image providing system 50 of the present embodiment. First, although the function of the laser radar ranging unit 110 is not changed, the laser radar ranging unit 110 transmits the irradiation direction and distance information of the laser beam to the communication device 52. The stereo image calculation unit 120 does not need to have the distance calculation unit 14, and the distortion correction unit 13 sends the reference image and the comparison image to the communication device 52. The communication device 52 transmits the irradiation direction, the distance information, the reference image, and the comparison image to the server device 51.

The server device 51 includes a communication device 53 and a distance calculation unit 14, and integrates a stereo matching cost C _ST (p, Z) and a LIDAR cost C _LI (p, Z) to obtain a distance image (three-dimensional high-density high-resolution distance). Create an image). The communication device 53 of the server device 51 transmits the distance image to the mobile unit 200.

  By transmitting the distance image and the reference image to the ECU 20, the moving object 200 can support the driving as in the first to fourth embodiments.

  As described above, in the present embodiment, since the mobile unit 200 and the server device 51 can communicate and generate a distance image, the cost of the distance measurement system 100 can be reduced.

  The server device 51 may transmit the distance image to a mobile other than the mobile 200 that has transmitted the information. For example, when the leading mobile body 200 transmits the irradiation direction, the distance information, the reference image, and the comparison image to the server apparatus 51 during a traffic jam, the server apparatus 51 transmits the distance image to the subsequent mobile body 200. Thereby, the following mobile unit 200 can also grasp the situation of the leading mobile unit 200.

<Other application examples>
The best mode for carrying out the present invention has been described above with reference to the embodiments. However, the present invention is not limited to these embodiments, and various modifications can be made without departing from the scope of the present invention. And substitutions can be added.

  For example, the moving body 200 on which the ranging system 100 is mounted can be a vehicle or an automobile, but the ranging system 100 can be widely applied to the moving body. For example, the present invention is effective for a mobile that moves autonomously even if at least a part of the user does not operate it. For example, it can be applied to an airplane, a drone, a ship, a robot and the like.

  As for the wavelength of the laser light, light in a wavelength range suitable for distance measurement may be used, and visible light, infrared light, and ultraviolet light (in a range that does not affect the human body) may be used. You may catch it.

  In this embodiment, LIDAR has been described as a distance measurement method with high distance resolution. However, millimeter waves, ultrasonic waves (sound waves, sonar), or the like may be used. Such active, distance measurement (distance from water and distance measurement) is collectively called active distance measurement.

  In the present embodiment, the stereo camera has been described as having two image capturing units, but may have three or more image capturing units. The two imaging units may be separated in the vertical direction as well as in the horizontal direction. The imaging unit may not only capture visible light, but may also capture light in a range such as near infrared or infrared. Alternatively, imaging may be performed via a polarization filter.

The stereo image calculation unit 120 is an example of an image processing unit or an image processing unit, the laser radar ranging unit 110 is an example of a ranging unit, and the LIDAR cost C _LI (p, Z) is an example of a distance cost. Yes, LIDAR cost C _LI (p, Z) is an example of a first evaluation value, and distance component cost C _LD (p, Z) is an example of a second evaluation value. The abnormality monitoring unit 601a is an example of a first abnormality detection unit, and the abnormality monitoring unit 14a is an example of a second abnormality detection unit.

8 stereo camera 11 right camera 12 left camera 13 distortion correction unit 14 distance calculation unit 20 ECU
DESCRIPTION OF SYMBOLS 100 Ranging system 110 Laser radar ranging part 120 Stereo image calculating part 200 Mobile body

Hern nn Badino et al., “Integrating LIDAR into Stereo for Fast and Improved Disparity Computation”, China, 2011 International Conference on 3D Imaging, Modeling, Processing, Visualization and Transmission, 16th May 2011

Claims (19)

An image processing method for generating a distance image for measuring a distance to an object, comprising:
A distance measuring unit detecting distance information to an object;
Among the pixels including the matching evaluation value of the stereo image, the image processing unit uses the distance measuring unit to set the matching evaluation value of the pixel corresponding to the position where the object whose distance information is detected by the distance measuring unit is present. And generating the distance image by integrating the distance information detected by
The image processing method, wherein the image processing unit performs the integration process before generating the distance image from the matching evaluation value. The image processing unit calculates a matching cost for each shift amount of pixels when matching the stereo image, converts each shift amount into a distance,
The distance evaluation value related to the distance information is integrated into the matching evaluation value of the distance corresponding to the distance information acquired from the distance measuring unit among the pixel matching evaluation values specified by the electromagnetic wave irradiation direction by the distance measuring unit. The image processing method according to claim 1.   3. The image according to claim 2, wherein the distance evaluation value is a first evaluation value that is the smallest in a pixel specified by an irradiation direction of an electromagnetic wave by the distance measuring unit and increases as the distance from the pixel increases. Processing method.   The image processing unit uses a matching evaluation value of a distance corresponding to the distance information acquired from the ranging unit among matching evaluation values of pixels around the pixel specified by the irradiation direction of the electromagnetic wave by the ranging unit. The image processing method according to claim 3, wherein the first evaluation value that increases as the distance from the pixel increases.   The distance evaluation value is the smallest in the distance corresponding to the distance information acquired by the distance measuring unit, and is multiplied by a second evaluation value that increases as the distance from the distance corresponding to the distance information increases. The image processing method according to claim 3 or 4, characterized in that: The image processing unit is the smallest matching evaluation value of the distance corresponding to the distance information acquired from the distance measuring unit among the pixel matching evaluation values specified by the electromagnetic wave irradiation direction by the distance measuring unit. A product obtained by multiplying the first evaluation value by the smallest second evaluation value is integrated as the distance evaluation value,
The second matching evaluation value that is larger than the distance corresponding to the distance corresponding to the distance information acquired from the distance measuring unit is increased by the distance from the distance corresponding to the smallest first evaluation value and the distance information. The image processing method according to claim 5, wherein a product obtained by multiplying the evaluation value of the image by the image is integrated as the distance evaluation value. The image processing unit uses a matching evaluation value of a distance corresponding to the distance information acquired from the ranging unit among matching evaluation values of pixels around the pixel specified by the irradiation direction of the electromagnetic wave by the ranging unit. Integrating the product of the first evaluation value, which increases as the distance from the pixel specified by the irradiation direction of the electromagnetic wave increases, and the second evaluation value, which is the smallest, as the distance evaluation value;
Corresponding to the first evaluation value and the distance information, which increases as the distance from the pixel specified by the irradiation direction of the electromagnetic wave increases to the matching evaluation value of the distance away from the distance corresponding to the distance information acquired from the distance measuring unit. The image processing method according to claim 5, characterized in that a product obtained by multiplying the second evaluation value, which is increased by the distance from the target distance, is integrated as the distance evaluation value. The image processing unit applies a dense algorithm to an evaluation value obtained by integrating the matching evaluation value and the distance evaluation value for each pixel,
The evaluation value of the pixel of interest and the evaluation value of surrounding pixels are overlapped, and then the distance that minimizes the evaluation value is determined as the distance of the pixel of interest. The image processing method according to any one of 2 to 7. A first abnormality detection unit for detecting an abnormality of the distance measuring unit;
A second abnormality detection unit that detects an abnormality of the imaging unit that captures at least one of the stereo images;
The image processing according to any one of claims 1 to 8, wherein at least one of the first abnormality detection unit or the second abnormality detection unit displays on the display device that an abnormality has been detected. Method. When the first abnormality detection unit detects an abnormality of the distance measuring unit, the image processing unit has distance information on pixels of the stereo image from a matching evaluation value of the stereo image captured by the imaging unit. Generating the associated distance image;
10. The image processing method according to claim 9, wherein when the second abnormality detection unit detects an abnormality in the imaging unit, the distance measurement unit outputs the distance information to the outside. The distance measuring unit detects a light reception level associated with distance information to an object,
The image processing unit integrates the matching evaluation value of the pixel specified from the distance measuring direction of the distance measuring unit and the distance evaluation value when the matching evaluation value is flat and the light reception level has one minimum value. Done
The matching evaluation value of a pixel specified from the distance measuring direction and the distance evaluation value are integrated when the received light level has a multi-pulse and the minimum value of the matching evaluation value is one. Item 2. The image processing method according to Item 2.   The image processing method according to any one of claims 1 to 11, wherein the pixel corresponding to the position where the object exists is a pixel specified by the distance measurement method by the distance measurement unit. .   The image processing method according to claim 1, wherein the image processing unit also associates distance information with pixels of the stereo image. An image processing apparatus for generating a distance image for measuring a distance to an object, comprising:
A distance measuring unit that detects distance information to an object;
Among the pixels including the matching evaluation value of the stereo image, the distance detected by the distance measuring unit to the matching evaluation value of the pixel corresponding to the position where the object whose distance information is detected by the distance measuring unit exists. An image processing unit that integrates information to generate the distance image,
The image processing apparatus, wherein the image processing unit performs the integration process before generating the distance image from the matching evaluation value. An in-vehicle device,
A distance measuring unit that detects distance information to an object;
Among the pixels including the matching evaluation value of the stereo image, the distance detected by the distance measuring unit to the matching evaluation value of the pixel corresponding to the position where the object whose distance information is detected by the distance measuring unit exists. An image processing unit that integrates information to generate a distance image;
The image processing unit performs the integration process before generating the distance image from the matching evaluation value,
An on-vehicle apparatus for transmitting the distance image to a control unit of a mobile unit. A distance measuring unit that detects distance information to an object;
Among the pixels including the matching evaluation value of the stereo image, the distance detected by the distance measuring unit to the matching evaluation value of the pixel corresponding to the position where the object whose distance information is detected by the distance measuring unit exists. An image processing unit that integrates information to generate a distance image;
The image processing unit performs the integration process before generating the distance image from the matching evaluation value,
A control unit for controlling a mobile using the distance image,
Mobile body with. An image processing device that acquires distance information from an object from a distance measuring unit,
Among the pixels including the matching evaluation value of the stereo image, the distance detected by the distance measuring unit to the matching evaluation value of the pixel corresponding to the position where the object whose distance information is detected by the distance measuring unit exists. An image processing unit that integrates information to generate a distance image;
The image processing apparatus, wherein the image processing unit performs the integration process before generating the distance image from the matching evaluation value. Ranging means for measuring the distance to the object;
Imaging means;
The distance information detected by the distance measuring unit in the matching evaluation value of the pixel corresponding to the position where the object whose distance information is detected by the distance measuring unit among the pixels including the matching evaluation value of the stereo image is detected. An image processing unit that generates a distance image by integrating processing of
The system according to claim 1, wherein the image processing unit performs the integration process before generating the distance image from the matching evaluation value. In a system that generates a distance image from distance information and a stereo image,
Image processing for generating a distance image by integrating the distance information with a matching evaluation value of a pixel corresponding to a position where an object for which the distance information is detected exists among pixels including a matching evaluation value of a stereo image Means, and
The system according to claim 1, wherein the image processing means performs the integration process before generating the distance image from the matching evaluation value.