JP2016169001A - Imaging system, image processing system, movable body control system, movable body device, projection device, object detection method, and object detection program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging system capable of acquiring an image effective to detect object information to a dark part.SOLUTION: The imaging system is an imaging system to be mounted on a vehicle body (movable body). The imaging system includes a projection device including a headlight device 10 for projecting random pattern light (pattern light having an irregular brightness distribution), and a stereo camera 20 (an imaging device) including two imaging parts 20a, 20b for imaging a light projection range of the projection device. In this case, the imaging device can acquire an image effective to detect object information to a dark part.SELECTED DRAWING: Figure 5

Description

  The present invention relates to an imaging system, an image processing system, a mobile body control system, a mobile body device, a light projection device, an object detection method, and an object detection program, and more specifically, an imaging system including the light projection device and the imaging device, An image processing system including an imaging system, a mobile control system including the image processing system, a mobile device including the mobile control system, a light projecting device that projects pattern light, and an object detection method including a projecting step, The present invention relates to an object detection program including a procedure for projecting light.

  2. Description of the Related Art Conventionally, a technique for detecting object information such as the presence / absence of an object and a distance to the object using an imaging apparatus including a plurality of imaging units is known.

  Further, in recent years, a light projecting device mounted on a moving body has been actively developed (see, for example, Patent Document 1).

  However, even if the conventional imaging device and the light projecting device are mounted on a moving body, an image effective for detecting object information for a dark part cannot be obtained.

  The present invention is an imaging system mounted on a moving body, and a light projecting device that projects pattern light having an irregular luminance distribution, and a plurality of image capturing units that image a light projecting range of the light projecting device An imaging apparatus including the imaging device.

  According to the present invention, it is possible to obtain an image effective for detecting object information for a dark part.

It is a top view of a vehicle provided with the mobile control system of one embodiment of the present invention. FIG. 2A is a side view of a vehicle including a moving body control system, and FIG. 2B is a front view of the vehicle including a moving body control system. It is a perspective view of the stereo camera of a moving body control system. It is a figure which shows an example of random pattern light. It is a block diagram which shows the hardware constitutions of a mobile body control system. It is a figure for demonstrating the parallax image generation means of an image process part. It is a figure for demonstrating the case of 8 propagation directions. It is a figure for demonstrating the minority parallax by subpixel estimation. It is explanatory drawing of the principle which derives | leads-out the distance from a stereo camera to an object. 10A is a diagram showing a reference image, FIG. 10B is a diagram showing a high-density parallax image with respect to FIG. 10A, and FIG. 10C is an edge parallax with respect to FIG. 10A. It is a conceptual diagram which shows an image. FIG. 11A is a conceptual diagram showing reference pixels in the reference image, and FIG. 11B shows the shift amount while sequentially shifting the corresponding pixel candidates in the comparison image with respect to the reference pixels in FIG. It is a conceptual diagram at the time of calculating. It is a graph which shows the cost value for every shift amount. It is a conceptual diagram for deriving a synthesis cost value. It is a graph which shows the synthetic | combination cost value for every parallax value. It is a flowchart for demonstrating the vehicle control process by the mobile body control system of one Embodiment. It is a flowchart for demonstrating the light projection control process 1 by the object detection apparatus of one Embodiment. It is a flowchart for demonstrating the light projection control process 2 by the object detection apparatus of the modification 1. It is a block diagram which shows the structure of the hardware of the mobile body control system of the modification 2. 10 is a flowchart for explaining a light projection control process 3 by an object detection device according to a second modification. It is a flowchart for demonstrating the light projection control process 4 by the object detection apparatus of the modification 3. FIGS. 21A and 21B are diagrams for explaining a texture amount calculation method. It is a flowchart for demonstrating the light projection control process 5 by the object detection apparatus of the modification 4. It is a flowchart for demonstrating the light projection control process 6 by the object detection apparatus of the modification 5. It is a flowchart for demonstrating the light projection control process 7 by the object detection apparatus of the modification 6.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[Overview]
As shown in FIGS. 1 to 2B, a mobile control system 100 according to an embodiment is mounted on a vehicle body (mobile) of a vehicle 1 as a mobile device, and includes a headlight device 10 and an imaging device. A stereo camera 20 (see FIG. 3), a processing device 30 (see FIG. 5), an ECU 50 (see FIG. 5) as an electronic control device, and the like. Here, the mobile control system 100 is supplied with electric power from an electric battery of the vehicle 1.

  Examples of the vehicle body that is a moving body include an automobile, a train, a motorcycle, a bicycle, a wheelchair, and an agricultural cultivator. Moreover, as a mobile body apparatus provided with the mobile body control system 100 and the mobile body by which this mobile body control system 100 is mounted, not only a vehicle but an aircraft, a ship, etc. may be sufficient, for example.

  The headlight device 10 includes a pair of left and right headlights 10a and 10b provided at the front portion of the vehicle body. The headlights 10a and 10b are turned on or off at the same time. Here, the headlights 10a and 10b are provided as standard equipment on the vehicle 1. Hereinafter, when it is not necessary to distinguish the headlights 10a and 10b, they are collectively referred to as “headlights”.

  Each headlight includes an LED array as a light source, in which a plurality of LEDs are arranged in an array, and a drive circuit that drives the LED array. The headlight can also be turned on / off by manually operating a headlight switch provided near the handle of the vehicle 1. The headlight may include, for example, a white light source instead of the LED as a light source.

  The light projected forward of the vehicle from the headlight is irradiated to the object when the object is within the light projection range of the headlight device 10. Here, the “light projection range” means a range (see FIG. 1) that includes a range in which light can be projected by each headlight of the headlight device 10.

  As an example, as shown in FIGS. 2A and 2B, the stereo camera 20 is attached in the vicinity of the rear-view mirror above the driver's seat inside the vehicle, and both eyes (the left and right imaging units 20a, 20b) so that the front of the vehicle can be imaged. That is, the stereo camera 20 can image the light projection range of the headlight device 10. The left and right images (luminance images) captured by both eyes of the stereo camera 20 are sent to the processing device 30 for each frame. The processing device 30 performs parallax calculation using the left and right images from the stereo camera 20.

  While the headlight device 10 is projecting light, pattern light is emitted to an object (for example, a person, another vehicle, a structure, a road, a tree, etc.) within the light projecting range, and thus the image is picked up with both eyes of the stereo camera 20. The pattern of the pattern light is superimposed (textured) on the objects in the left and right images.

  As shown in FIG. 5, the processing device 30 includes a CPU 31, a memory 29, a stereo camera control CPU I / F 32, an image processing unit 33, an image recognition processing unit 34, brightness information CPU I / F 35, and an image recognition result CPU I / F 37. , A start / end signal transfer I / F 38 and an image recognition result transfer I / F 40.

  The stereo camera control CPU I / F 32 is a transmission / reception interface between the CPU 31 and the stereo camera 20. The CPU 31 performs imaging control of the stereo camera 20 via the stereo camera control CPU I / F 32.

  The CPU 31 and the ECU 50 can transmit and receive via the start / end signal transfer I / F 38.

  The image processing unit 33 calculates brightness information from the left and right images (luminance images) from the stereo camera 20 and transmits the brightness information to the CPU 31 via the brightness information CPU I / F 35. The CPU 31 transmits the received brightness information to the ECU 50 via the brightness data transfer I / F 38.

  In addition, the image processing unit 33 performs gamma correction, distortion correction, and the like on the left and right images captured by the stereo camera 20, and then performs parallax calculation using the left and right images (luminance images). The image and the luminance image are sent to the image recognition processing unit 34.

  The image recognition processing unit 34 recognizes (detects) object information such as the presence / absence of an object, the type (type) of the object, and the distance to the object based on the parallax image and the luminance image from the image processing unit 33. Then, the image recognition result as the recognition result is transmitted to the CPU 31 via the image recognition result CPU I / F 37. The CPU 31 transmits the received image recognition result to the ECU 50 via the image recognition result transfer I / F 40.

  That is, the processing results in the image processing unit 33 and the image recognition processing unit 34 are sent to the CPU 31 of the processing device 30, corrected to a data format that can be received by the ECU 50, and then transferred to the ECU 50. The data format transferred to the ECU 50 mainly includes CAN I / F, LIN I / F, and the like.

  The ECU 50 controls the headlight according to the operation (ON / OFF) of the headlight switch by the driver or the like, and based on the processing information from the processing device 30, the braking device for the vehicle 1 is added to the headlight device 10. Control the steering device.

  The ECU 50 includes a CPU 51, a memory 52, a CPU I / F 53, a CPU I / F 55, a light emission control I / F 56, a braking control I / F 57, and a steering control I / F 58.

  The CPU 51 includes a light projection control unit 51a that controls the headlight device 10, a braking control unit 51b that controls the braking device, and a steering control unit 51c that controls the steering device.

  The CPU 51 acquires the brightness information from the CPU 31 via the start / end signal transfer I / F 38 and the CPU I / F 53.

  Further, the CPU 51 acquires the image recognition result from the CPU 31 via the image recognition result transfer I / F 40 and the CPU I / F 55.

  The braking control unit 51b and the steering control unit 51c acquire the image recognition result from the CPU 31 via the image recognition result transfer I / F 40 and the CPU I / F 55.

  The braking control unit 51b controls the braking device (for example, autobrake for avoiding danger) based on the acquired image recognition result.

  The steering control unit 51c performs control of the steering device (for example, auto steering for avoiding danger) based on the acquired image recognition result.

  Next, a parallax calculation algorithm using an image captured by the stereo camera 20 will be described with reference to FIGS.

  The image data captured by the left and right imaging units of the stereo camera 20 is subjected to processing such as gamma correction and distortion correction by the preprocessing unit of the image processing unit 33, respectively. For the left and right image data that has been subjected to the preprocessing, a parallax calculation algorithm is executed by a parallax image generating means (see FIG. 6) that is a post-processing unit of the image processing unit 33, and a parallax value is calculated for each pixel.

  By the way, the calculation method for generating the parallax image may be a method for calculating the parallax only for a portion having a strong texture (a portion having a strong edge) such as EBM (edge base). Alternatively, as in SGM (semi-global matching), the dissimilarity of matching is propagated from the periphery of each pixel by a recurrence formula, and the decimal disparity is calculated from the integer disparity that gives the minimum dissimilarity and the dissimilarity of the disparity adjacent thereto. It is also possible to calculate the parallax of each pixel of the whole image by using a subpixel estimation method (fitting by an equiangular straight line, a parabola, a high-order polynomial, etc.).

  SGM is more preferable than EBM because it can suppress missing pixels in the parallax data.

  Therefore, the parallax image generation unit of the present embodiment adopts the SGM method, and as an example, as shown in FIG. 7, the high frequency enhancement filter unit for the reference image, the high frequency enhancement filter unit for the comparison image, and the matching A degree calculator, a propagation similarity calculator, an integer parallax calculator, and a minority parallax calculator. The SGM method is disclosed in non-patent literature (Accurate and Efficient Stereo Processing by Semi-Global Matching and Mutual Information).

  Below, the parallax image generation means of this embodiment is demonstrated. In the SGM parallax calculation, as shown in FIG. 7, the dissimilarity is propagated from each direction of the target pixel for which the parallax is to be calculated (see formula (1) or (2)), and the energy S that is the sum of them is calculated. (Refer to equation (3)) is calculated, and the integer parallax D [pixel] that gives the minimum value of the energy S is derived as the integer parallax of the target pixel. Here, C expressed by the expressions (1) and (2) represents the degree of coincidence by the parallax search of the comparison image of the target pixel of the reference image. As an example of C, other stereo matching indices such as SAD, SSD, and NCC may be used. After calculating C, Lr is calculated from equation (1) or (2), and finally S is calculated from equation (3). Increasing the number of propagation directions improves the parallax calculation accuracy, but increases the size of the calculation processing circuit (the relationship between the parallax calculation accuracy and the processing circuit size is a trade-off).

  Further, when pursuing parallax accuracy, sub-parallax is calculated using a sub-pixel estimation method as shown in FIG. For example, in the equiangular straight line method, the decimal parallax D ′ is calculated using three S corresponding to the parallaxes D−1, D, and D + 1. Alternatively, in the parabola fit method, the decimal parallax D ′ is calculated using five S corresponding to the parallaxes D−2, D−1, D, D + 1, and D + 2.

  Note that whether or not the calculated parallax value is reliable is determined by reliability determination. If it is determined that the parallax value is not reliable, the parallax of the pixel is invalid parallax.

<Principles of ranging>
With reference to FIG. 9, the principle of deriving a parallax with respect to an object from a stereo camera by the stereo image method and measuring the distance from the stereo camera to the object with a parallax value indicating the parallax will be described. FIG. 9 is an explanatory diagram of the principle of deriving the distance from the stereo camera 20 to the object. Further, in the following, in order to simplify the description, the description will be made on a pixel-by-pixel basis, not on a predetermined region composed of a plurality of pixels.
When processing is performed in units of a predetermined area including a plurality of pixels instead of a single pixel, the predetermined area including the reference pixel is indicated as the reference area, and the predetermined area including the corresponding pixel is indicated as the corresponding area. Further, this reference area includes the case of only the reference pixel, and the corresponding area includes the case of only the corresponding pixel.

(Parallax value calculation)
First, the images captured by the image capturing unit 20a and the image capturing unit 20b illustrated in FIG. 9 are referred to as a reference image Ia and a comparative image Ib, respectively. In FIG. 9, it is assumed that the imaging unit 20a and the imaging unit 20b are installed in parallel equiposition. In FIG. 9, the point S on the object E in the three-dimensional space is mapped to a position on the same horizontal line of the imaging unit 20a and the imaging unit 20b. That is, the S point in each image is imaged at a point Sa (x, y) in the reference image Ia and a point Sb (X, y) in the comparative image Ib. At this time, the parallax value Δ is expressed by the following equation (4) using Sa (x, y) at the coordinates on the imaging unit 20a and Sb (X, y) at the coordinates on the imaging unit 20b. Is done.
Δ = X−x (4)

  Here, in the case as shown in FIG. 9, the distance between the point Sa (x, y) in the reference image Ia and the intersection of the perpendicular drawn from the imaging lens 11a on the imaging surface is set to Δa, and the comparison image Ib When the distance between the point Sb (X, y) and the intersection of the perpendicular line taken from the imaging lens 11b on the imaging surface is Δb, the parallax value Δ = Δa + Δb.

(Distance calculation)
Moreover, the distance Z between the imaging units 20a and 20b and the object E can be derived by using the parallax value Δ. Specifically, the distance Z is a distance from a plane including the focal position of the imaging lens 11a and the focal position of the imaging lens 11b to the specific point S on the object E. As shown in FIG. 9, using the focal length f of the imaging lens 11a and the imaging lens 11b, the baseline length B that is the length between the imaging lens 11a and the imaging lens 11b, and the parallax value Δ, the following ( The distance Z can be calculated by the equation 5).
Z = (B × f) / Δ (5)
According to the equation (5), the larger the parallax value Δ, the smaller the distance Z, and the smaller the parallax value Δ, the larger the distance Z.

<SGM method>
Subsequently, a distance measuring method using the SGM method will be described with reference to FIGS. 10A is a reference image, FIG. 10B is a conceptual diagram illustrating a high-density parallax image with respect to FIG. 10A, and FIG. 10C is a conceptual diagram illustrating an edge parallax image with respect to FIG.

  Here, the reference image is an image in which an object is indicated by luminance. The high-density parallax image is an image derived from the reference image by the SGM method, and is an image showing the parallax value at each coordinate of the reference image. The edge parallax image is an image derived by a conventionally used block matching method, and is an image showing only a parallax value of a portion having a relatively strong texture such as an edge portion of a reference image.

  The SGM method is a method for appropriately deriving the parallax value even for an object with a weak texture. Based on the reference image shown in FIG. This is a method for deriving a density parallax image. When the block matching method is used, the edge parallax image shown in FIG. 10C is derived based on the reference image shown in FIG. As can be seen by comparing the inside of the dashed ellipse in FIGS. 10B and 10C, the high-density parallax image can represent detailed information such as a road having a weak texture compared to the edge parallax image. More detailed distance measurement can be performed.

This SGM method does not calculate the cost value that is dissimilarity and immediately derives the parallax value, but after calculating the cost value, it further calculates the synthesis cost value that is the synthetic dissimilarity. This is a method of deriving a parallax value and finally deriving a parallax image (here, a high-density parallax image) indicating the parallax value in almost all pixels.
In the case of the block matching method, the cost value is calculated in the same way as the SGM method. However, as in the SGM method, a comparatively strong texture such as an edge portion is calculated without calculating a synthesis cost value. Only the parallax value is derived.

(Calculation of cost value)
First, a method for calculating the cost value C (p, d) will be described with reference to FIGS. FIG. 11A is a conceptual diagram showing reference pixels in the reference image, and FIG. 11B is a diagram showing the corresponding pixel candidates in the comparison image while sequentially shifting (shifting) the reference pixels in FIG. 11A. It is a conceptual diagram at the time of calculating quantity (deviation amount). FIG. 12 is a graph showing the cost value for each shift amount. Here, the corresponding pixel is a pixel in the comparison image that is most similar to the reference pixel in the reference image. In the following description, C (p, d) represents C (x, y, d).

  As shown in FIGS. 11A and 11B, a predetermined reference pixel p (x, y) in the reference image and an epipolar in the comparison image with respect to the reference pixel p (x, y). The cost of each corresponding pixel candidate q (x + d, y) with respect to the reference pixel p (x, y) based on each luminance value with a plurality of corresponding pixel candidates q (x + d, y) on the line (Epipolar Line) A value C (p, d) is calculated. d is the shift amount (shift amount) between the reference pixel p and the corresponding pixel candidate q, and in this embodiment, the shift amount in pixel units is represented. That is, in FIG. 11A and FIG. 11B, the corresponding pixel candidate q (x + d, y) is sequentially shifted by one pixel within a predetermined range (for example, 0 <d <25). A cost value C (p, d), which is a dissimilarity between the luminance values of the pixel candidate q (x + d, y) and the reference pixel p (x, y), is calculated. As a calculation method of the cost value C, when the cost value C indicates dissimilarity, a known method such as SAD (Sum of Absolute Difference) is applied.

The cost value C (p, d) calculated in this way can be represented by a graph of a cost curve that is a collection of cost values C for each shift amount d, as shown in FIG. In FIG. 12, the cost value C is 0 (zero) in the case of the shift amount d = 5, 12, and 19, and therefore the minimum value cannot be obtained. Thus, in the case of an object having a weak texture, it is difficult to obtain the minimum value of the cost value C.
(Computation cost value calculation)
Next, a method for calculating the combined cost value Ls (p, d) will be described with reference to FIGS. 13 and 14. FIG. 13 is a conceptual diagram for deriving a synthesis cost value. FIG. 14 is a graph of a synthesis cost curve showing a synthesis cost value for each parallax value. The calculation method of the synthesis cost value in the present embodiment is not only the calculation of the cost value C (p, d), but also the cost value when pixels around the predetermined reference pixel p (x, y) are used as the reference pixels. The combined cost value Ls (p, d) is calculated by consolidating the cost value C (p, d) at the reference pixel p (x, y).

Next, a method for calculating the synthesis cost value will be described in more detail. In order to calculate the combined cost value Ls (p, d), it is first necessary to calculate the route cost value Lr (p, d). The following expression (6) is an expression for calculating the route cost value Lr (p, d) and is substantially the same as the expression (1), and the expression (7) is the combined cost value Ls. This is an expression for calculating the above expression, and is substantially the same as the expression (3).
Lr (p, d) = C (p, d) + min {(Lr (pr, d), Lr (pr, d-1) + P1, Lr (pr, d + 1) + P1, Lrmin (p) -R) + p2} (6)

  Here, in Expression (6), r represents a direction vector in the aggregation direction, and has two components in the x direction and the y direction. min {} is a function for obtaining the minimum value. Lrmin (p−r) indicates the minimum value of Lr (p−r, d) when the shift amount d is changed in the coordinates obtained by shifting p by one pixel in the r direction. Note that Lr is applied recursively as shown in equation (6). P1 and P2 are fixed parameters determined in advance by experiments, and are parameters that make it easy for the parallax values Δ of adjacent reference pixels on the path to be continuous. For example, P1 = 48 and P2 = 96.

Further, as shown in the equation (6), Lr (p, d) is the cost value C in the reference pixel p (x, y), and the respective values in each pixel in the r direction shown in FIG. It is obtained by adding the minimum value of the path cost value Lr of the pixel. Thus, in order to obtain Lr at each pixel in the r direction, first, Lr is obtained from the endmost pixel in the r direction of the reference image p (x, y), and Lr is obtained along the r direction. .
Then, as shown in FIG. 13, Lr ₀ , Lr ₄₅ , Lr ₉₀ , Lr ₁₃₅ , Lr ₁₈₀ , Lr ₂₂₅ , Lr ₂₇₀ , and Lr ₃₁₅ in eight directions are obtained, and finally based on the equation (7). Thus, the synthesis cost value Ls is obtained.

The composite cost value Ls (p, d) calculated in this way is a graph of the composite cost curve in which the composite cost value Ls (p, d) is shown for each shift amount d as shown in FIG. Can be represented by In FIG. 14, the composite cost value Ls is calculated as a parallax value Δ = 3 because the minimum value is obtained when the shift amount d = 3.
In the above description, the number of r is assumed to be 8. However, the present invention is not limited to this. For example, the eight directions may be further divided into two to be divided into sixteen directions and three into 24 directions.
Further, although the cost value C is shown as “dissimilarity”, it may be expressed as “similarity” as a reciprocal of dissimilarity. In this case, as a method for calculating the cost value C, a known method such as NCC (Normalized Cross Correlation) is applied. Further, in this case, the parallax value Δ where the combined cost value Ls is not “minimum” but “maximum” is derived. In addition, you may represent as a "matching degree" including both dissimilarity and similarity.

  Note that a part of the processing of the processing device may be executed by the ECU, or a part of the processing of the ECU may be executed by the processing device. Moreover, you may comprise one control system which has both functions of a processing apparatus and ECU. In this case, the object detection program and the moving body control program may be stored in one memory.

  Further, instead of the memory, other storage media such as a hard disk may be used.

  Moreover, although the controlled object of a mobile body control system is a vehicle braking device and a steering device, it is not restricted to this. For example, only one of the braking device and the steering device may be set as the control target, or the drive source (for example, engine or motor) of the vehicle may be set as at least one of the control targets.

  In addition, the moving body control system includes a braking control unit and a steering control unit that control the vehicle body, but instead of or in addition to this, based on detection information of an object detection device described later, a driver or the like You may have the alarm device which outputs the alarm sound and warning display for calling attention.

  In addition, the LED array is turned on by a light emission pattern with an irregular light amount distribution and a random pattern light is projected. Instead, for example, a “pattern” represented by a projection mapping image is displayed in front of the host vehicle. It may be projected. In other words, the light projecting device may include an image forming unit that forms an image with light and emits the light that has formed the image as random pattern light. Further, random pattern light may be generated and projected by irradiating light to a mask pattern having an irregular pattern distribution.

  Further, although a headlight device is used as a light source unit of the light projecting device, a dedicated light source unit different from the headlight device may be provided separately. In addition, the headlight device is not limited to the standard equipment in the vehicle body, and may be a custom-made product or an optional product.

  Moreover, although the stereo camera 20 which has two imaging parts (two eyes) is employ | adopted as an imaging device, you may employ | adopt the imaging device which has three or more imaging parts.

[Details]
The light projection control unit 51a turns on the headlight of the headlight device 10 using the light emission pattern data stored in the memory 52 according to the acquired brightness information, and projects random pattern light.

  Therefore, a “light projecting device” that projects the random pattern light includes the headlight device 10 and the light projecting control unit 51a. In addition, an “imaging system” that captures the light projection range of the light projecting device is configured including the light projecting device and the stereo camera 20. Further, an object detection device that detects object information in the light projection range of the light projection device is configured including the image processing system including the imaging system and the image processing unit 33.

  The ECU 50 turns on the headlight when the outside of the vehicle is determined to be “dark” while the headlight is turned off, and turns off the headlight when the outside of the vehicle is determined to be “bright” while the headlight is turned on.

  More specifically, when the ECU 50 determines that the outside of the vehicle is “dark” based on the detection result from the brightness information detecting means for detecting the brightness information outside the vehicle while the headlight is turned off, the light emission pattern data The headlight is turned on using and the pattern light is projected from the headlight. When receiving the light emission pattern data, the headlight drive circuit drives each LED of the LED array with a current value set for each LED in the light emission pattern data.

  More specifically, the light emission pattern data is stored in the memory 52 (see FIG. 5) of the ECU 50. The CPU 51 (see FIG. 5) of the ECU 50 reads the light emission pattern data from the memory 52 at the time of light projection, and drives the LED array of each headlight according to the light emission pattern data.

  Further, when the ECU 50 determines that the exterior of the vehicle has become bright based on the detection result from the brightness information detection means while the headlight is turned on, the ECU 50 turns off the headlight.

  The pattern light projected from the headlight to the front of the vehicle is applied to the object when there is an object in the projection range. Here, the “light projection range” means a range including a range in which light can be projected by each headlight (see FIG. 1).

  The brightness information detection means can be configured to include a stereo camera 20 and an image processing unit 33 (to be described later) that processes an image captured by the stereo camera 20. For example, a combination of a monocular camera and an image processing unit Alternatively, an illuminance sensor or the like may be used. That is, when the brightness information detection unit is configured by the stereo camera 20 and the image processing unit 33, the image processing unit 33 determines that the image is “dark” when the total luminance value of all the pixels in the image is smaller than the threshold value. To do. Further, when the brightness information detecting means is an illuminance sensor, it is determined to be “dark” when the acquired value of the sensor is smaller than the threshold value. Of course, the brightness detection by the brightness information detecting means can take various forms as long as the brightness outside the vehicle can be acquired. The detection result of the brightness information detection means is transmitted to the ECU 50.

  Here, the “light emission pattern data” is data having an irregular light amount distribution (current value distribution), and the corresponding “pattern light” has an irregular luminance distribution (see FIG. 4). Therefore, hereinafter, “pattern light” is also referred to as “random pattern light”.

  In the random pattern light, the binary or multi-value (three or more values) luminance portions are irregularly distributed (see FIG. 4).

  For example, in the case where the light amount distribution of the light emission pattern data is such that the binary light amount portion is irregularly distributed, the random pattern light is also irregularly distributed in the binary luminance portion. Become.

  For example, if the light quantity distribution of the light emission pattern data is an irregular distribution of multi-value light quantity parts, the random pattern light also has an irregular distribution of multi-value luminance parts. It becomes.

  Here, the memory 52 of the ECU 50 may store, for example, a plurality of light emission pattern data having the same pattern and different contrasts (hereinafter abbreviated as “a plurality of light emission pattern data having different contrasts”). A plurality of light emission pattern data having different contrasts can be obtained by uniformly increasing / decreasing the light amount values of all the light amount portions with respect to the reference light emission pattern data.

  In this case, the ECU 50 determines the contrast based on the brightness outside the vehicle, the presence or absence of an object such as another vehicle in front of the vehicle, an obstacle, the texture amount of the environment in front of the vehicle, and the parallax reliability information of the parallax image. It is desirable to select appropriate light emission pattern data from a plurality of different light emission pattern data, and to turn on the LED array of each headlight using the light emission pattern data.

  The object presence / absence detecting means for detecting the presence / absence of an object in front of the vehicle can be configured to include the stereo camera 20 and the image processing unit 33. For example, a combination of a monocular camera, a polarization camera and an image processing unit, a laser radar Etc. may be used. When a combination of a monocular camera and an image processing unit is used as the brightness detection unit, this combination can also be used as the object presence / absence detection unit.

  The texture detection unit that detects the texture amount of the environment in front of the vehicle and the unit that acquires the parallax reliability information of the parallax image can include the stereo camera 20 and the image processing unit 33.

  Therefore, when a plurality of light emission pattern data having different contrasts are stored in the memory 52, the ECU 50 selects one pattern among a plurality of pattern lights having different contrasts according to the light / darkness outside the vehicle or the environment in front of the vehicle. Light can be selectively projected from each headlight.

  By the way, in a dark part such as in a daytime tunnel or at night in a vehicle, the dark part is brightened by projecting light forward from the headlight, but since the contrast of the object originally in the dark part is already low, it is simply a headlight. Even if the light is turned on and uniform light is irradiated, it is difficult to obtain image contrast. Therefore, in a dark part, when acquiring a parallax image ahead of a vehicle with a stereo camera, it becomes difficult to obtain matching (correlation) between left and right images during parallax calculation. As a result, the number of invalid parallax pixels of a subject (a person, another vehicle, an obstacle, a road surface, a background, etc.) in front of the vehicle is increased, and the accuracy of output parallax is reduced.

  Hereinafter, control of the vehicle main body (vehicle control process) by the moving body control system 100 of the present embodiment will be described with reference to the flowchart of FIG. The control here is executed by the ECU 50 and the processing device 30. That is, a subprogram for operating the ECU 50 among the mobile body control programs used for control of the mobile body control system 100 is stored in the memory 52 of the ECU 50, and a subprogram for operating the processing device 30 is stored in the processing device 30. Stored in the memory 29.

  The moving body control program includes all procedures of the object detection program used for controlling the object detection apparatus. Of the object detection program, a subprogram for controlling the light projecting device is stored in the memory 52 of the ECU 50, and a subprogram for controlling the stereo camera 20 and the processing device 30 is stored in the memory 29 of the processing device 30. .

  The control here is performed when a safety mode (danger avoidance mode) is selected (ON) in the operation unit of the vehicle 1 by a driver or a passenger (hereinafter referred to as “driver etc.”). When the safety mode is selected (ON), the stereo camera 20 captures images in real time with both eyes and sends the left and right images to the image processing unit 33 for each frame.

  In the first step S1, the ECU 50 executes “light projection control process 1”. The “light projection control process 1” will be described later.

  In the next step S2, “parallax image generation processing” is executed. The “parallax image generation process” is executed by the processing device 30 according to the subprogram stored in the memory 29. Specifically, the processing device 30 generates a parallax image from the images (luminance images) for the left and right frames captured by the stereo camera 20. More specifically, after the image processing unit 33 performs preprocessing such as gamma correction and distortion correction, parallax calculation is performed, and the preprocessed luminance image and the parallax image obtained by the parallax calculation are input to the image recognition processing unit 34. send.

  In the next step S3, “image recognition processing” is executed. The “image recognition process” is executed by the processing device 30 according to the subprogram stored in the memory 29. Specifically, the processing device 30 performs image recognition (object recognition) using a parallax image. More specifically, the image recognition processing unit 34 recognizes (detects) information (object information) related to an object in the light projection range, and sends the result (image recognition result) to the ECU 50. The “information about the object” includes, for example, the presence / absence of the object, the distance to the object, the type of the object (for example, a person, a structure, a road, etc.). Note that the types of objects can be detected by storing parameters such as the sizes and shapes of a plurality of types of objects in the memory 29 in advance and detecting them based on the parameters.

  In the next step S4, "braking / steering control processing" is executed. The “braking / steering control process” is executed by the ECU 50 in accordance with the subprogram stored in the memory 52. Specifically, the ECU 50 performs autobrake and autosteering (control for avoiding danger) based on the received image recognition result. More specifically, the braking / steering control unit 51b appropriately controls the braking device and the steering device based on the image recognition result.

  In the next step S5, the ECU 50 determines whether or not the safety mode is ON. If the determination here is affirmed, the process returns to step S1, and if the determination is negative, the flow ends.

  Next, “light projection control process 1” in step S1 of the “vehicle control process” will be described with reference to the flowchart of FIG. The light projection control process 1 is executed according to the subprogram stored in the memory 52 by the light projection control unit 51a. Here, brightness information is calculated in real time by the processing device 30 from an image (luminance image) ahead of the vehicle imaged in real time by the stereo camera 20, and the brightness information is transmitted to the light projection control unit 51a in real time. Yes.

  In the first step S11, it is determined whether or not the outside of the vehicle is dark. Specifically, the brightness information from the processing device 30 is compared with a preset reference value, and if the brightness information is less than the reference value, it is determined to be “dark”, and if it is equal to or greater than the reference value. Judge that it is not dark. If the determination in step S11 is negative (determined that it is not dark), the process proceeds to step S12. If the determination in step S11 is affirmative (determined to be dark), the process proceeds to step S13.

  In step S <b> 12, a turn-off trigger signal is transmitted to the headlight device 10. As a result, the headlight is turned off when the headlight is on, and the off state is maintained when the headlight is off. When step S12 is executed, the flow ends.

  In step S <b> 13, the lighting trigger signal and the light emission pattern data are transmitted to the headlight device 10. As a result, when the headlight is turned on, the lighting state is maintained, and when the headlight is turned off, the light is turned on, and the random pattern light is projected forward of the vehicle. At this time, the range in which the random pattern light is projected is picked up by both eyes of the stereo camera 20, and the picked up left and right images are sent to the image processing unit 33 for each frame. When step S13 is executed, the flow ends.

  In the vehicle control processing including the light projection control processing 1, when the safety mode is ON, it is automatically prior to manual operation of the headlight switch by the driver or the like when the brightness outside the vehicle is less than the reference value. The headlight is turned on and the headlight is not turned on when the reference value is exceeded. On the other hand, when the safety mode is OFF, the headlight is turned on / off only by manual operation of the headlight switch by the driver or the like.

  The imaging system of the present embodiment described above is an imaging system mounted on a vehicle main body (moving body), and projects a random pattern light (pattern light having an irregular luminance distribution). And a light projecting device including a light projecting control unit 51a, and a stereo camera 20 (image capturing device) including two image capturing units 20a and 20b that capture the light projecting range of the light projecting device.

  In this case, since the light projection range (range where the random pattern light is projected) can be captured by the stereo camera 20, an image effective for detecting object information for a dark part can be obtained.

  In addition, the light projecting device includes a headlight device 10 that is provided as a standard in the vehicle main body and projects a random pattern light as a light source unit. In this case, the random pattern light becomes illumination light that illuminates the dark part and is also effective light for parallax calculation. That is, it is not necessary to separately provide a light source unit in addition to the headlight device 10.

  In addition, the object detection device of the present embodiment includes an imaging system and a processing device 30 that is mounted on the vehicle body and processes an image captured by the stereo camera 20 of the imaging system.

  In this case, the object information can be detected with high accuracy from the images (left and right images) effective for detecting the object information for the dark part obtained by the imaging system.

  As a result, for example, the matching accuracy of the left and right images in the light projection range can be improved, and the parallax of the object in the light projection range can be calculated with high accuracy.

  That is, since the texture is also attached to the object in the dark part where the contrast is originally low, the contrast appears in the image acquired by the left and right imaging parts of the stereo camera 20, the right and left matching accuracy can be ensured, and the parallax calculation of the object in the dark part Accuracy can be improved.

  As a result, according to the object detection device of the present embodiment, it is possible to improve the detection accuracy of the object information for the dark part.

  Further, since the light projecting device projects a random pattern light when the brightness outside the vehicle body is less than a preset reference value (predetermined value), the light projecting device is random only for the dark part where the accuracy of the parallax calculation is reduced. Pattern light can be projected, unnecessary light projection can be suppressed, and energy saving can be achieved. Since the contrast of the object originally in the bright part is already high, the necessity of projecting the random pattern light is low when the outside of the vehicle is bright.

  Moreover, since the light projector acquires brightness information from the luminance information of the image captured by the stereo camera 20, the number of parts can be increased compared to the case where a dedicated detection unit for detecting brightness information is separately provided. Increase and cost increase can be suppressed.

  Further, since the processing device 30 further includes the image processing unit 33 that generates a parallax image from a plurality of images individually captured by the two imaging units 20a and 20b, it is possible to generate a highly accurate parallax image.

  Further, since the processing device 30 further includes an image recognition processing unit 34 that performs image recognition using the parallax image generated by the image processing unit 33, a highly accurate image recognition result can be obtained.

  In addition, the moving body control system 100 includes an object detection device and an ECU 50 that is mounted on the vehicle main body and controls the vehicle main body based on an image recognition result in the processing device 30 of the object detection device. It is possible to prevent the main body from colliding with the object.

  Moreover, since the vehicle 1 (moving body apparatus) is provided with the moving body control system 100 and the vehicle main body (moving body) in which the moving body control system 100 is mounted, it is excellent in safety.

  Moreover, since the light projection apparatus of this embodiment projects random pattern light on the imaging range of the stereo camera 20 including the two imaging units 20a and 20b mounted on the vehicle body, the light projection is performed even in a dark part. It is possible to cause the stereo camera 20 to acquire an image that can accurately calculate the parallax of an object in the range.

  As can be seen from the above description, the light projection device, the imaging system, and the object detection device of the present invention are particularly effective devices when used in a dark environment, and are not necessarily mounted on a moving body such as a vehicle body. This is not necessary, and is expected to be used in various fields. When the light projecting device, the imaging system, and the object detection device are not mounted on the moving body, a light source unit instead of the headlight device is used.

  In addition, the object detection method using the object detection apparatus according to the present embodiment includes a step of projecting random pattern light from the vehicle body and an image of the range where the random pattern light is projected from two positions on the moving body. And a step of processing the image captured in the imaging step.

  In this case, since the range in which the random pattern light is projected can be imaged from two positions, an image effective for detecting object information for a dark part can be obtained, and the object information can be obtained from the image with high accuracy. Can be detected.

  As a result, according to the object detection method of the present embodiment, it is possible to improve the detection accuracy of the object information for the dark part.

  Moreover, since the object detection method further includes a step of generating a parallax image from a plurality of images individually captured from a plurality of positions in the imaging step, a highly accurate parallax image can be generated.

  Further, the mobile body control method using the mobile body control system of the present embodiment includes a step of projecting random pattern light from the vehicle body, and a range in which the random pattern light is projected from a plurality of positions on the mobile body. A step of capturing, a step of generating a parallax image from a plurality of images individually captured from a plurality of positions in the step of capturing, a step of recognizing an image using the parallax image generated in the step of generating a parallax image, And a step of controlling the vehicle body based on the recognition result in the image recognition step.

  In this case, it is possible to prevent the traveling vehicle body from colliding with the object.

  Further, an object detection program used for controlling the object detection apparatus of the present embodiment is an object detection program for causing a computer to execute the object detection method, a procedure for projecting random pattern light from the vehicle body, and a random It includes a procedure for imaging a range in which pattern light is projected from a plurality of positions on the vehicle body, and a procedure for processing an image captured in the imaging procedure.

  In this case, since the range in which the random pattern light is projected can be imaged from two positions, an image effective for detecting object information for a dark part can be obtained, and the object information can be obtained from the image with high accuracy. Can be detected.

  As a result, according to the object detection program of this embodiment, it is possible to improve the detection accuracy of the object information for the dark part.

  Moreover, since the object detection program further includes a procedure for generating a parallax image from a plurality of images individually captured from a plurality of positions in the imaging step, it is possible to generate a highly accurate parallax image.

  In addition, the moving body control program used for the control of the moving body control system of the present embodiment includes a procedure for projecting random pattern light from the vehicle body, and a plurality of ranges on which the random pattern light is projected on the moving body. A procedure for imaging from a position, a procedure for generating a parallax image from a plurality of images individually captured from a plurality of positions in the imaging procedure, and a procedure for image recognition using a parallax image generated by a procedure for generating a parallax image And a procedure for controlling the vehicle main body based on the recognition result in the image recognition procedure.

  In this case, it is possible to prevent the traveling vehicle body from colliding with the object.

  Further, since part of the object detection program is stored in the memory 29 and the remaining part is stored in the memory 52, the object detection apparatus can be operated according to the object detection program as appropriate.

  In addition, since a part of the moving object control program is stored in the memory 29 and the remaining part is stored in the memory 52, the moving object control system 100 can be appropriately operated according to the moving object control program.

  Hereinafter, some modified examples of the above embodiment will be described.

<< Modification 1 >>
The mobile body control system of the modification 1 has the same configuration as the mobile body control system 100 of the above embodiment. In the first modification, the light projection control process is different from the above embodiment.

  In the first modification, two light emission pattern data (high contrast light emission pattern data and low contrast light emission pattern data) having different contrasts are stored in the memory 52 of the ECU 50, and the light projection control unit 51a turns on the headlight. At this time, one light emission pattern data is selected from these two light emission pattern data according to the brightness outside the vehicle, and the LED array of the headlight is turned on using the one light emission pattern data.

  Below, the light projection control process 2 of the modification 1 is demonstrated with reference to the flowchart of FIG. The control here is executed according to the subprogram stored in the memory 52 by the light projection control unit 51a. Here, brightness information is calculated in real time by the processing device 30 from an image (brightness image) ahead of the vehicle imaged in real time by the stereo camera 20, and the brightness information is transmitted to the light projection control unit 51a in real time. .

  In first step S21, it is determined whether or not the received brightness information is less than the first reference value. If the determination in step S21 is negative, the process proceeds to step S22. If the determination in step S21 is affirmed, the process proceeds to step S23. Here, when the brightness information is greater than or equal to the first reference value (when the process proceeds to step S22), it is a level generally felt “bright”.

  In step S22, an extinction trigger signal is transmitted to the headlight device 10. As a result, the headlight is turned off when the headlight is on, and the off state is maintained when the headlight is off. When step S22 is executed, the flow ends.

  In step S23, it is determined whether or not the received brightness information is less than a second reference value that is smaller than the first reference value. If the determination in step S23 is affirmed, the process proceeds to step S24, and if the determination in step S23 is negative, the process proceeds to step S25. Here, when the brightness information is less than the first reference value and greater than or equal to the second reference value (when the process proceeds to step S25), it is a level generally felt “dim”. When the brightness information is less than the second reference value (when the process proceeds to step S24), it is a level generally felt “dark”.

  In step S <b> 24, the lighting trigger signal and the high contrast light emission pattern data are transmitted to the headlight device 10. As a result, when the headlight is turned on, the lighting state is maintained, and when the headlight is turned off, the headlight is turned on, and a high-contrast random pattern light is projected forward of the vehicle. At this time, the range in which the high-contrast random pattern light is projected is picked up by both eyes of the stereo camera 20, and the picked up left and right images are sent to the image processing unit 33 for each frame. When step S24 is executed, the flow ends.

  In step S <b> 25, the lighting trigger signal and the low contrast light emission pattern data are transmitted to the headlight device 10. As a result, when the headlight is turned on, the lighting state is maintained, and when each headlight is turned off, the headlight is turned on, and low contrast random pattern light is projected forward of the vehicle. At this time, the range in which the low-contrast random pattern light is projected is picked up by both eyes of the stereo camera 20, and the picked up left and right images are sent to the image processing unit 33 for each frame. When step S25 is executed, the flow ends.

  In the vehicle control process including the light projection control process 2, when the safety mode is ON, the brightness information outside the vehicle is less than the first reference value in preference to the manual operation of the headlight switch by the driver or the like. The headlight is automatically turned on, and the headlight is not turned on when the value is equal to or greater than the first reference value. On the other hand, when the safety mode is OFF, the headlight device 10 is turned on / off only by manual operation of the headlight switch during operation or the like.

  In the modification 1 demonstrated above, the light projection apparatus containing the light projection control part 51a and the headlight apparatus 10 adjusts the contrast of random pattern light according to the brightness outside a vehicle, when projecting random pattern light. (Adjust the luminance distribution).

  In this case, it is possible to project random pattern light having a contrast suitable for parallax calculation according to the brightness outside the vehicle.

  For example, when the driver's attention is distracted, it is possible to calculate parallax with high accuracy by projecting a random pattern light with a low contrast, and the driver feels uncomfortable with the ranram pattern light. Can be reduced. As a result, it is possible to achieve both improved safety and visibility by the system.

  For example, parallax can be calculated with high accuracy by projecting random pattern light with high contrast when it is “dark”, which is disadvantageous in calculating parallax. As a result, safety by the system can be ensured.

  In the first modification, two reference values are set, and two random pattern lights having different contrasts are selectively projected when the brightness information falls below at least one of the two reference values. However, three or more reference values may be set, and three or more random pattern lights having different contrasts may be selectively projected with each reference value as a boundary. In this case as well, it is desirable to project random pattern light with higher contrast as the brightness information is smaller (darker).

<< Modification 2 >>
In the second modification, the configuration of the moving body control system and the light projection control process are different from those in the above embodiment. In the processing device 60 of the mobile control system 200 of Modification 2 shown in FIG. 18, vehicle front texture information CPU I / F 36 and texture information transfer I / F 39 are added to the processing device 30 of the above embodiment. Yes. Further, in the ECU 70 of the moving body control system 200, a CPU I / F 54 is added to the ECU 50 of the above embodiment.

  The image processing unit 33 of the processing device 60 calculates vehicle front texture information from the left and right images (luminance images) captured by the stereo camera 20, and transmits the vehicle front texture information to the CPU 31 via the vehicle front texture information CPU I / F 36. The CPU 31 transmits the received vehicle front texture information to the ECU 50 via the texture information transfer I / F 39.

  The CPU 51 of the ECU 70 acquires the texture information from the CPU 31 via the texture information transfer I / F 39 and the CPU I / F 54.

  In the second modification, two light emission pattern data (high contrast light emission pattern data and low contrast light emission pattern data) having different contrasts are stored in the memory 52 of the ECU 70, and the light projection control unit 51 a uses the headlight device 10. When lighting, one light emission pattern data is selected from these two light emission pattern data according to the presence or absence of an object in front of the vehicle, and the LED array of each headlight is lighted using the one light emission pattern data.

  Below, the light projection control process 3 of the modification 2 is demonstrated with reference to the flowchart of FIG. The light projection control process 3 is executed by the light projection control unit 51a according to the subprogram stored in the memory 52. Here, brightness information is calculated in real time by the processing device 60 from an image (luminance image) ahead of the vehicle imaged in real time by the stereo camera 20, and the brightness information is transmitted to the light projection control unit 51a in real time. .

  In the first step S31, it is determined whether or not the vehicle exterior is dark. Specifically, the brightness information from the processing device 60 is compared with a preset reference value. If the brightness information is less than the reference value, it is determined to be “dark”, and if the brightness information is greater than the reference value, “ Judge that it is not dark. If the determination in step S31 is negative (determined as not dark), the process proceeds to step S32. If the determination in step S31 is affirmed (determined as dark), the process proceeds to step S33.

  In step S <b> 32, a turn-off trigger signal is transmitted to the headlight device 10. As a result, the headlight is turned off when the headlight is on, and the off state is maintained when the headlight is off. When step S32 is executed, the flow ends.

  In step S33, it is determined whether there is an object in front of the vehicle. Here, the processing device 60 determines the presence or absence of an object, and outputs the determination result to the ECU 70. Specifically, the image processing unit 33 determines the presence or absence of an object by determining the texture amount of the vehicle front area in the image (luminance image) from the stereo camera 20, and outputs the determination result to the light projection control unit 51a. To do. More specifically, the image processing unit 33 determines that “there is an object” when the texture amount is equal to or greater than a threshold value, and determines that “there is no object” when the texture amount is less than the threshold value. A method for determining the texture amount will be described later. If the determination in step S33 is affirmed, the process proceeds to step S34, and if the determination in step S33 is negative, the process proceeds to step S35. The determination of whether there is an object in front of the vehicle is not limited to the determination based on the texture amount, and various methods can be taken. For example, an object ahead of the vehicle may be recognized using known pattern matching. However, since the calculation speed is also required for determining whether there is an object in front of the vehicle, it is effective to determine the presence / absence of the object based on the texture amount.

  In step S <b> 34, the lighting trigger signal and the high contrast light emission pattern data are transmitted to the headlight device 10. As a result, when the headlight is turned on, the lighting state is maintained, and when the headlight is turned off, the headlight is turned on, and a high-contrast random pattern light is projected forward of the vehicle. At this time, the range in which the high-contrast random pattern light is projected is picked up by both eyes of the stereo camera 20, and the picked up left and right images are sent to the image processing unit 33 for each frame. When step S34 is executed, the flow ends.

  In step S <b> 35, the lighting trigger signal and the low contrast light emission pattern data are transmitted to the headlight device 10. As a result, when the headlight is turned on, the lighting state is maintained, and when the headlight is turned off, the headlight is turned on, and low-contrast random pattern light is projected forward of the vehicle. At this time, the range in which the low-contrast random pattern light is projected is picked up by both eyes of the stereo camera 20, and the picked up left and right images are sent to the image processing unit 33 for each frame. When step S35 is executed, the flow ends.

  In the second modification described above, the light projecting device including the light projecting control unit 51a and the headlight device 10 determines whether there is an object in front of the vehicle (light projecting range of the light projecting device) when projecting the random pattern light. Accordingly, the contrast of the random pattern light is adjusted (luminance distribution is adjusted).

  In this case, it is possible to project random pattern light having a necessary and sufficient contrast according to the presence or absence of an object in front of the vehicle, that is, the risk of collision.

  For example, when there is no object at low collision risk, low-contrast random pattern light can be projected, so that the parallax can be calculated with high accuracy and the driver's sense of discomfort due to the ranram pattern light is reduced. can do. As a result, both safety and visibility by the system can be achieved.

  For example, parallax can be calculated with higher accuracy by projecting high-contrast random pattern light when there is an “object present” with a high risk of collision. As a result, safety by the system can be ensured.

  Further, since the processing device 60 detects the presence / absence of an object from the image captured by the stereo camera 20, the number of parts and the cost are increased compared to a case where a dedicated detection unit for detecting the presence / absence of an object is separately provided. Can be suppressed.

<< Modification 3 >>
The mobile body control system of Modification 3 has the same configuration as that of the mobile body control system 200 of Modification 2. In the third modification, the light projection control process is different from the above embodiment.

  In the third modification, two light emission pattern data (high contrast light emission pattern data and low contrast light emission pattern data) having different contrasts are stored in the memory 52 of the ECU 70, and the light projection control unit 51 a uses the headlight device 10. When turning on the light, one light emission pattern data is selected from these two light emission pattern data according to the sum of the texture amount in front of the vehicle and diagonally forward, and the LED array of each headlight is selected using the one light emission pattern data. Light up.

  Below, the light projection control process 4 of the modification 2 is demonstrated with reference to the flowchart of FIG. The light projection control process 4 is executed by the light projection control unit 51 a according to the subprogram stored in the memory 29. Here, brightness information is calculated in real time by the processing device 60 from an image (luminance image) ahead of the vehicle imaged in real time by the stereo camera 20, and the brightness information is transmitted to the light projection control unit 51a in real time. .

  In first step S41, it is determined whether or not the vehicle exterior is dark. Specifically, the brightness information from the processing device 60 is compared with a preset reference value. If the brightness information is less than the reference value, it is determined to be “dark”, and if the brightness information is greater than the reference value, “ Judge that it is not dark. If the determination in step S41 is negative (determined that it is not dark), the process proceeds to step S42. If the determination in step S41 is affirmative (determined to be dark), the process proceeds to step S43.

  In step S42, a turn-off trigger signal is transmitted to the headlight device 10. As a result, the headlight is turned off when the headlight is on, and the off state is maintained when the headlight is off. When step S42 is executed, the flow ends.

  In step S43, it is determined whether the texture amount of the entire image captured by the stereo camera 20 is large. Here, the processing device 60 determines the texture amount, and outputs the determination result to the ECU 70. Specifically, the image processing unit 33 determines whether or not the sum of the texture amounts (hereinafter also referred to as the total texture amount) of the vehicle front area and the oblique front area in the image (luminance image) from the stereo camera 20 is equal to or greater than a threshold value. Judgment is made, and the judgment result is output to the light projection control unit 51a. More specifically, the image processing unit 33 determines that “the total texture amount is large” when the total texture amount is equal to or greater than the threshold value, and determines that “the total texture amount is not large” when the total texture amount is less than the threshold value. A method for determining the total texture will be described later. If the determination in step S43 is affirmed, the process proceeds to step S44, and if the determination in step S43 is negative, the process proceeds to step S45.

  In step S44, a lighting trigger signal and low contrast light emission pattern data are transmitted to the headlight device 10. As a result, when the headlight is turned on, the lighting state is maintained, and when the headlight is turned off, the headlight is turned on, and low-contrast random pattern light is projected forward of the vehicle. At this time, the range in which the low-contrast random pattern light is projected is picked up by both eyes of the stereo camera 20, and the picked up left and right images are sent to the image processing unit 33 for each frame. When step S44 is executed, the flow ends.

  In step S <b> 45, the lighting trigger signal and the high contrast light emission pattern data are transmitted to the headlight device 10. As a result, when the headlight is turned on, the lighting state is maintained, and when the headlight is turned off, the headlight is turned on, and a high-contrast random pattern light is projected forward of the vehicle. At this time, the range in which the high-contrast random pattern light is projected is picked up by both eyes of the stereo camera 20, and the picked up left and right images are sent to the image processing unit 33 for each frame. When step S45 is executed, the flow ends.

  In the modified example 3 described above, the light projecting device including the light projecting control unit 51a and the headlight device 10 emits a random pattern light, and the sum of texture amounts in front of the vehicle and obliquely forward (of the headlight device 10). The contrast of the random pattern light is adjusted (luminance distribution is adjusted) based on the total amount of texture in the light projection range.

  In this case, it is possible to project random pattern light having a necessary and sufficient contrast according to the sum of texture amounts in front of the vehicle and diagonally ahead (total amount of texture).

  For example, when the total amount of texture is large (when traveling in a busy street, mountain forest, etc.), a random pattern light with a low contrast can be projected, so that the parallax can be calculated with high accuracy and the ranram pattern for the driver The uncomfortable feeling of visibility due to light can be reduced. As a result, both safety and visibility by the system can be achieved.

  For example, when the total amount of texture is small (when traveling on the coast, countryside, wilderness, etc.), the parallax can be calculated with higher accuracy by projecting a random pattern light with a high contrast. As a result, safety by the system can be ensured.

  Further, since the processing device 60 acquires the texture amount of the light projection range from the image captured by the stereo camera 20, the number of parts increases and the case where a dedicated detection means for detecting the texture amount is separately provided. Cost increase can be suppressed.

  In the third modification, one texture total amount threshold value is set, and two random pattern lights having different contrasts are selectively projected with the threshold value as a boundary. However, two texture total amount threshold values are set. It is good also as setting up above and selectively projecting three or more random pattern light from which contrast differs on each threshold. Also in this case, it is desirable to project random pattern light having a higher contrast as the total texture amount is smaller.

  Here, the texture amount determination method in Modification 2 and the texture total determination method in Modification 3 will be described with reference to FIGS. 21A and 21B.

First, in FIG. 21A, a texture definition formula for a certain pixel is Txt = | (A + B) − (C + D) |. || means an absolute value.
A is a pixel value of a pixel adjacent to the left by two pixels from a certain pixel.
Let B be the pixel value of the pixel one pixel to the left of a certain pixel.
Let C be the pixel value of the pixel right one pixel away from a certain pixel.
D is a pixel value of a pixel that is two pixels to the right of a certain pixel.
Also, Th is a texture threshold value.
When Txt> Th is satisfied, it is determined that the pixel has a texture, and the texture flag flag = 1 is set.
If Txt> Th is not satisfied, it is determined that the pixel has no texture, and the texture flag flag = 0 is set.
This process is performed for each pixel of the entire image or the determination target area of the texture amount in the image, and by counting the number of pixels where flag = 1, the texture amount of the entire image or the entire determination target area is calculated. It can be determined (see FIG. 21B).
For example, when it is desired to determine the texture amount of only the front area of the vehicle, a range (for example, four coordinates) of the front area of the vehicle is designated in advance, and the number of flags = 1 in the range may be counted. For example, when it is desired to determine the texture amount of the diagonally forward area of the vehicle, area designation (coordinate designation) is performed for the areas on both sides of the image, and the number of flags = 1 in the range may be counted. Regarding the definition expression of “texture amount”, the above is merely an example, and various correspondences such as Txt = | A−C | and Txt = | B−D | can be taken.

<< Modification 4 >>
The mobile control system of the modification 4 has the same configuration as the mobile control system 100 of the above embodiment. In the fourth modification, the light projection control process is different from the above embodiment.

  In the modified example 4, two light emission pattern data (high contrast light emission pattern data and low contrast light emission pattern data) having different contrasts are stored in the memory 52 of the ECU 50, and the light projection control unit 51 a uses the headlight device 10. When lighting, the LED array of each headlight is turned on using either of these two light emission pattern data in accordance with the number of invalid parallax pixels (parallax reliability information) in the parallax image.

  Below, the light projection control process 5 of the modification 4 is demonstrated with reference to the flowchart of FIG. The light projection control process 5 is executed by the light projection control unit 51a according to the subprogram stored in the memory 52. Here, brightness information is calculated in real time by the processing device 30 from an image (brightness image) ahead of the vehicle imaged in real time by the stereo camera 20, and the brightness information is transmitted to the light projection control unit 51a in real time. .

  In first step S51, it is determined whether or not the outside of the vehicle is dark. Specifically, the brightness information from the processing device 30 is compared with a preset reference value, and if the brightness information is less than the reference value, it is determined to be “dark”, and if it is equal to or greater than the reference value. Judge that it is not dark. If the determination in step S51 is negative (determined that it is not dark), the process proceeds to step S52. If the determination in step S51 is affirmed (determined to be dark), the process proceeds to step S53.

  In step S52, a turn-off trigger signal is transmitted to the headlight device 10. As a result, the headlight is turned off when the headlight is on, and the off state is maintained when the headlight is off. When step S52 is executed, the flow ends.

  In step S <b> 53, the lighting trigger signal and the low contrast light emission pattern data are transmitted to the headlight device 10. As a result, when the headlight is turned on, the lighting state is maintained, and when the headlight is turned off, the headlight is turned on, and low-contrast random pattern light is projected forward of the vehicle. At this time, the range in which the low-contrast random pattern light is projected is picked up by both eyes of the stereo camera 20, and the picked up left and right images are sent to the image processing unit 33 for each frame. When step S53 is executed, the process proceeds to step S54.

  In step S54, it is determined whether or not there are many invalid parallax pixels in the parallax image. Here, the processing device 30 determines the number of invalid parallax pixels in the parallax image, and outputs the determination result to the ECU 50. Specifically, the image processing unit 33 performs preprocessing such as gamma correction and distortion correction on the image (luminance image) captured by the stereo camera 20, and then performs parallax calculation, and is obtained by parallax calculation. The obtained parallax image is sent to the image recognition processing unit 34. Then, the image recognition processing unit 34 determines whether or not the number of invalid parallax pixels in the received parallax image is equal to or greater than a threshold value, and outputs the determination result to the light projection control unit 51a. If the determination in step S54 is affirmed, the process proceeds to step S55, and if the determination in step S54 is negative, the flow ends.

  In step S55, a lighting trigger signal and high contrast light emission pattern data are transmitted to the headlight device 10. As a result, when the headlight is turned on, the lighting state is maintained, and when the headlight is turned off, the headlight is turned on, and a high-contrast random pattern light is projected forward of the vehicle. At this time, the range in which the high-contrast random pattern light is projected is picked up by both eyes of the stereo camera 20, and the picked up left and right images are sent to the image processing unit 33 for each frame. When step S55 is executed, the flow ends.

  In the modified example 4 described above, the light projecting device including the light projecting control unit 51a and the headlight device 10 projects the random pattern light based on the number of invalid parallax pixels in the parallax image, that is, the parallax reliability information. To adjust the contrast of the random pattern light (adjust the luminance distribution).

  In this case, random pattern light with an appropriate contrast can be projected according to the parallax reliability information.

  For example, when the number of invalid parallax pixels is small (when the parallax reliability is high), by projecting a random pattern light with a low contrast, the parallax can be calculated with high accuracy and the ranram pattern light for the driver It is possible to reduce the uncomfortable feeling of visibility. As a result, both safety and visibility by the system can be achieved.

  For example, when the number of invalid parallax pixels is large (when the parallax reliability is low), parallax can be calculated with high accuracy by projecting random pattern light with high contrast. As a result, safety by the system can be ensured.

  Further, since the processing device 30 acquires the number of invalid parallax pixels (parallax reliability information) from the image captured by the stereo camera 20, the configuration of the processing device 30 is larger than that in the case where the number of invalid parallax pixels is acquired by other means. It can be simplified and cost increase can be suppressed.

  In the fourth modification, one threshold value for the number of invalid parallax pixels is set, and two random pattern lights having different contrasts are selectively projected with the threshold value as a boundary. Two or more threshold values may be set, and three or more random pattern lights having different contrasts may be selectively projected with each threshold as a boundary. Also in this case, it is desirable to project random pattern light having higher contrast as the number of invalid parallax pixels is larger.

<< Modification 5 >>
The mobile control system of the modification 5 has the same configuration as the mobile control system 100 of the above embodiment. In the modified example 5, the light projection control process is different from the above embodiment.

  In the modified example 5, two light emission pattern data (high contrast light emission pattern data and low contrast light emission pattern data) having different contrasts are stored in the memory 52 of the ECU 50, and the light projection control unit 51 a uses the headlight device 10. When lighting, the LED array of each headlight is turned on using either of these two light emission pattern data according to the presence or absence of an object in front of the vehicle.

  Below, the light projection control process 6 of the modification 5 is demonstrated with reference to the flowchart of FIG. The light projection control process 6 is executed by the light projection control unit 51a according to the subprogram stored in the memory 52. Here, brightness information is calculated in real time by the processing device 30 from an image (brightness image) ahead of the vehicle imaged in real time by the stereo camera 20, and the brightness information is transmitted to the light projection control unit 51a in real time. .

  In first step S61, it is determined whether or not the vehicle exterior is dark. Specifically, the brightness information from the processing device 60 is compared with a reference value set in advance, and if the brightness information is less than the reference value, it is determined to be “dark”, and if it is equal to or greater than the reference value. Judge that it is not dark. If the determination in step S61 is negative (determined that it is not dark), the process proceeds to step S62. If the determination in step S61 is affirmative (determined to be dark), the process proceeds to step S63.

  In step S62, a turn-off trigger signal is transmitted to the headlight device 10. As a result, the headlight is turned off when the headlight is on, and the off state is maintained when the headlight is off. When step S62 is executed, the flow ends.

  In step S63, a lighting trigger signal and low contrast light emission pattern data are transmitted to the headlight device 10. As a result, when the headlight is turned on, the lighting state is maintained, and when the headlight is turned off, the headlight is turned on, and low-contrast random pattern light is projected forward of the vehicle. At this time, the range in which the low-contrast random pattern light is projected is picked up by both eyes of the stereo camera 20, and the picked up left and right images are sent to the image processing unit 33 for each frame. When step S63 is executed, the process proceeds to step S64.

  In step S64, it is determined whether there is an object in front of the vehicle. Specifically, the processing device 30 generates a parallax image, performs image recognition using the parallax image, determines the presence or absence of an object, and outputs the determination result to the ECU 50. More specifically, after the image processing unit 33 performs preprocessing such as gamma correction and distortion correction on an image (luminance image) captured by the stereo camera 20, parallax calculation is performed, and the preprocessed luminance image, The parallax image obtained by the parallax calculation is sent to the image recognition processing unit 34. The image recognition processing unit 34 performs image recognition using the luminance image and the parallax image, determines the presence / absence of an object, and outputs the determination result to the light projection control unit 51a. If the determination in step S64 is affirmed, the process proceeds to step S65, and if the determination in step S64 is negative, the flow ends.

  In step S <b> 65, the lighting trigger signal and the high contrast light emission pattern data are transmitted to the headlight device 10. As a result, when the headlight is turned on, the lighting state is maintained, and when the headlight is turned off, the headlight is turned on, and a high-contrast random pattern light is projected forward of the vehicle. At this time, the range in which the high-contrast random pattern light is projected is picked up by both eyes of the stereo camera 20, and the picked up left and right images are sent to the image processing unit 33 for each frame. When step S65 is executed, the flow ends.

  In the modified example 5 described above, a low-contrast random pattern light is projected when the outside of the vehicle is dark, a parallax image is generated from the captured image, and image recognition using the parallax image is performed. It is possible to accurately detect the presence or absence of an object. Then, the contrast of the random pattern light is adjusted (the luminance distribution is adjusted) based on the presence / absence of the front of the vehicle. That is, when “no object” is detected in front of the vehicle, the low-contrast random pattern light is continuously projected. When “object” is detected in front of the vehicle, high-contrast random pattern light is emitted. Switch to floodlight.

  In this case, random pattern light with an appropriate contrast can be projected according to the presence or absence of an object in front of the vehicle.

  As a result, it is possible to project random pattern light having an appropriate contrast according to the presence or absence of an object in front of the vehicle, that is, the risk of collision.

  For example, when there is no object at low collision risk, low-contrast random pattern light can be projected, so that the parallax can be calculated with high accuracy and the driver's sense of discomfort due to the ranram pattern light is reduced. can do. As a result, both safety and visibility by the system can be achieved.

  For example, parallax can be calculated with higher accuracy by projecting high-contrast random pattern light when there is an “object present” with a high risk of collision. As a result, safety by the system can be ensured.

  Further, since the processing device 30 detects the presence / absence of an object from the image captured by the stereo camera 20, the number of parts is increased and the cost is increased compared to a case where a dedicated detection unit for detecting the presence / absence of an object is separately provided. Can be suppressed.

  Note that, as in the light projecting control process 7 (steps S71 to S75) of the modified example 6 shown in FIG. 24, a control that substantially replaces step S63 and step 65 in the flowchart of FIG. good. In the sixth modification, the same effect as in the fifth modification can be obtained.

  Examples 1 to 5 (the parallax calculation method used: SGM algorithm) of the present invention are shown below (see Table 1).

<Example 1> (corresponding to the above embodiment)
While driving, he judged that the driver was dark and turned on the headlight manually. The headlight irradiation light was given a non-uniform contrast. As a result, the number of invalid parallax pixels in front of the vehicle decreased.

<Example 2> (corresponding to the above embodiment)
Brightness information detection means was used. During driving, the headlight was automatically turned on automatically judging that it was dark. The headlight irradiation light was given a non-uniform contrast. As a result, the number of invalid parallax pixels in front of the vehicle decreased.

<Example 3> (corresponding to Modifications 2 and 5 above)
Brightness information detection means and object presence / absence detection means were used. During driving, the headlight was automatically turned on automatically judging that it was dark. In addition, while traveling, the presence of another vehicle was detected 40 meters ahead of the vehicle, and thus the headlight irradiation light was given a non-uniform contrast. As a result, the number of invalid parallax pixels in front of the vehicle decreased. In particular, the number of invalid parallax pixels of other vehicles in front of the vehicle has decreased.

<Example 4> (corresponding to Modification 3 above)
Brightness information detection means and texture detection means were used. During driving, the headlight was automatically turned on automatically judging that it was dark. In addition, during the running, the texture judged from the luminance image did not remain so much, so the irradiation light of the headlight was given a non-uniform contrast. As a result, the number of invalid parallax pixels in front of the vehicle decreased.

<Example 5> (corresponding to Modification 4 above)
Brightness detection means and parallax image reliability information (invalid parallax pixel number) acquisition means were used. During driving, the headlight was automatically turned on automatically judging that it was dark. Further, since the number of invalid parallax pixels determined from the parallax image was 65% with respect to the total number of pixels during traveling, non-uniform contrast was given to the irradiation light of the headlight. As a result, the number of invalid parallax pixels in front of the vehicle decreased.

  In the first to fifth embodiments, the stereo camera is used as a detection system, but the present invention is not limited to this. As a means other than the means for acquiring the parallax information, a monocular camera, a polarization camera, a laser radar, or the like can be used.

  The present invention is not limited to the above-described embodiment, each modification, and each example, and can be changed as appropriate. For example, the plurality of random pattern lights may be a plurality of random pattern lights having different contrasts and patterns, or may be a plurality of random pattern lights having different patterns and the same contrast. The switching between the plurality of random pattern lights may be configured to be manually switched by the driver.

  Further, in each of the above modified examples and embodiments, instead of projecting a low-contrast random pattern light, for example, rules such as light with a pattern with no contrast (uniform luminance distribution), light with regularity in contrast, etc. Pattern light having a typical luminance distribution may be projected.

  DESCRIPTION OF SYMBOLS 10 ... Headlight apparatus (a part of light projection apparatus), 20 ... Stereo camera (imaging apparatus), 30 ... Processing apparatus, 50 ... ECU (control apparatus), 51a ... (Light projection control part), 100, 200 ... Movement Body control system.

Japanese Patent No. 5138517

Claims (20)

An imaging system mounted on a moving body,
A light projecting device that projects pattern light having an irregular luminance distribution;
An imaging system comprising: an imaging device including a plurality of imaging units that images a light projection range of the light projecting device.   The imaging system according to claim 1, wherein the light projecting device includes a headlight mounted on the moving body as a light source unit. The imaging system according to claim 1 or 2,
An image processing system comprising: a processing device that is mounted on the moving body and that processes an image captured by an imaging device of the imaging system.   The image processing system according to claim 3, wherein the light projecting device projects the pattern light when brightness information outside the moving body is less than a predetermined value.   The image processing system according to claim 4, wherein the processing device calculates the brightness information based on luminance of an image captured by the imaging device.   The image processing system according to claim 4, wherein the light projecting device adjusts a luminance distribution of the pattern light according to the brightness information when projecting the pattern light.   6. The image processing according to claim 4, wherein when projecting the pattern light, the light projecting device adjusts the luminance distribution of the pattern light according to the presence or absence of an object in the light projection range. system.   The image processing system according to claim 7, wherein the processing device detects the presence or absence of the object based on an image captured by the imaging device.   The image processing system according to claim 4, wherein when projecting the pattern light, the light projecting device adjusts a luminance distribution of the pattern light based on a texture amount in the light projection range. .   The image processing system according to claim 9, wherein the processing device calculates the texture amount from an image captured by the imaging device. The processing device generates a parallax image from a plurality of images individually captured by the plurality of imaging units when the pattern light is projected from the light projecting device, and detects reliability information of the parallax image And
The image processing system according to claim 4, wherein the light projecting device adjusts a luminance distribution of the pattern light based on the reliability information.   The image processing system according to claim 6, wherein adjusting the luminance distribution of the pattern light includes adjusting a pattern of the pattern light.   The image processing system according to any one of claims 6 to 11, wherein adjusting the luminance distribution of the pattern light includes adjusting a contrast of the pattern light.   The processing device generates a parallax image from a plurality of images individually captured by the plurality of imaging units when the pattern light is projected from the light projecting device. The image processing system according to any one of the above.   The image processing system according to claim 14, wherein the processing device performs image recognition using the parallax image. An image processing system according to claim 15;
A moving body control system comprising: a control device mounted on the moving body and controlling the moving body based on an image recognition result in a processing device of the image processing system. The mobile control system according to claim 16,
A moving body apparatus comprising: the moving body on which the moving body control system is mounted. In the light projection device mounted on the moving body,
A light projecting device that projects pattern light having an irregular luminance distribution in an imaging range of an imaging device including a plurality of imaging units mounted on the moving body. Projecting pattern light having an irregular luminance distribution from the moving body;
Imaging a range in which the pattern light is projected from a plurality of positions on the moving body;
An object detection method comprising: processing the image captured in the imaging step. An object detection program for causing a computer to execute the object detection method according to claim 19,
A procedure for projecting pattern light having an irregular luminance distribution from the moving body,
A procedure for imaging the range in which the pattern light is projected from a plurality of positions on the moving body;
An object detection program comprising: a procedure for processing an image captured in the imaging procedure;