JP7096701B2 - Imaging device - Google Patents

Description

The present invention relates to an image pickup device that acquires a distance to an object using structured light.

As a technique for acquiring the distance to an object, there is a technique for measuring the distance to the object by irradiating the object with light and measuring the light reflected from the object. Further, the object may be irradiated with a plurality of types of light, and the distance may be measured using the results of measuring each type of light.

The following Patent Document 1 describes an object recognition device using a stereo camera. The document is based on "an active range finder 100 having a light projecting unit 110 that projects light onto the target object 1 and determining the distance to the target object 1 from the reflected light, and image information from the target object 1. The stereo camera 200 for obtaining the distance to the target object 1 and the object recognition means 310 for recognizing the target object 1 based on the output signal S1 from the active range finder 100 and the output signal S2 from the stereo camera 200. The main feature is that the auxiliary light control means 320 that controls the operation of the floodlight unit 110 so as to irradiate the light projected by the floodlight unit 110 of the active range finder 100 as the auxiliary light of the stereo camera 200 is provided. It is supposed to be. 』Disclosures the technology (see summary).

The following Patent Document 2 describes a technique of irradiating uniform pattern light and random pattern light to suppress deterioration of visibility. The document describes "a light projecting device including a headlight device 10 mounted on a vehicle body and projecting uniform pattern light and random pattern light having different brightness distributions at different time zones, and a light projecting range of the light projecting device. It is provided with a stereo camera 20 including two image pickup units on the left and right to capture an image. In this case, it is possible to obtain an image effective for detecting object information while suppressing a decrease in visibility. 』Disclosures the technology (see summary).

Japanese Unexamined Patent Publication No. 2005-077130 Japanese Unexamined Patent Publication No. 2016-170163

When the pattern light emitted from the light source provided in the vehicle is imaged by the camera, the brightness of the pattern decreases and the contrast of the pattern image decreases as the distance increases. Further, when a pattern consisting of a straight line pattern having a monospaced width or a parallel straight line is imaged with a camera, an image in which the pattern converges toward the vanishing point in perspective can be obtained. Then, the farther the pattern is, the narrower the pattern width and the interval between the straight lines are, so that the contrast of the pattern image is lowered due to the spatial resolution of the camera. Due to these effects, the distance measurement performance of an object deteriorates even if structured pattern light is used.

The present invention has been made in view of the above problems, and when the distance to an object is acquired by using structured light, the distance can be accurately measured even if the object is distant. The purpose is to provide technology.

The image pickup apparatus according to the present invention changes at least one of the light intensity of the structured light and the pattern size of the structured light according to the distance from the light source to the light projecting point.

According to the image pickup apparatus according to the present invention, even in an environment where the contrast of an image is low such as in a dark place, distance measurement can be performed accurately at a distance. Thereby, for example, the vehicle equipped with the image pickup device according to the present invention can determine the presence / absence of a travelable route and the necessity of avoidance with high reliability.

It is a block diagram of the stereo camera 100 which concerns on Embodiment 1. FIG. An example of a projection pattern when the headlight 200 projects structured light is shown. This is an example of a complementary pattern. It is the result of time averaging the structured pattern of FIG. 2 and the structured pattern of FIG. This is an example of pattern switching timing assuming a global shutter method in which the exposure timings of each row of the image sensor are simultaneous. This is an example assuming a rolling shutter type image sensor in which the exposure timing of each line of the image sensor is deviated. This is an example in which the exposure time is relatively long compared to the frame period. This is an example in which the positional relationship between the image sensor of the camera 1 and the image sensor of the camera 2 is shifted up and down. It is a figure which exemplifies the light projection range when the traveling speed is a predetermined speed or more. It is a figure which exemplifies the light projection range when the traveling speed is a predetermined speed or less. This is an example of detecting that there is a preceding vehicle in front of the own vehicle. This is an example of the light projection range when the steering wheel is turned to the left. It is explanatory drawing of the matching by the area matching method. This is an example of changing the light intensity for each floodlight area. This is an example of the light intensity of each projection area of the structured pattern and the interpolation pattern. This is another example of the light intensity of each projection area of the structured pattern and the interpolation pattern. It is a block diagram of the stereo camera 700 which concerns on Embodiment 2. FIG. This is an example of a light projection pattern when the laser light source 707 projects structured light. It is a figure which shows the floodlight pattern for discriminating two similar straight lines. This is an example of a criterion for identifying which straight line the linear pattern on the captured image is. This is an example of a method of distinguishing a straight line by the light intensity / straight line width / arrangement of a straight line pattern. This is an example of a method for distinguishing straight lines by the color / arrangement of the straight line pattern. It is a figure which exemplifies the light projection range when the traveling speed is a predetermined speed or more. It is a figure which exemplifies the light projection range when the traveling speed is a predetermined speed or less. This is an example of the light projection range when the steering wheel is turned to the left. This is a configuration example in which the stereo camera 700 is equipped with the acceleration sensor 708. It is a block diagram of the camera 1000 which concerns on Embodiment 3. It is a block diagram of the camera 1400 which concerns on Embodiment 4. FIG.

<Embodiment 1>
FIG. 1 is a configuration diagram of a stereo camera 100 according to the first embodiment of the present invention. The stereo camera 100 is an image pickup system that projects structured light using a headlight mounted on a vehicle. The stereo camera 100 includes an optical unit 101, a signal processing unit 102, a distance calculation unit 103, an object recognition unit 104, a vehicle control unit 105, and a control unit 106. The vehicle equipped with the stereo camera 100 includes a headlight 200, a headlight 300, and a vehicle controller 400. The signals a, b1 and b2 will be described later.

When the accessory power of the vehicle equipped with the stereo camera is turned on, the stereo camera 100 starts a range-finding operation after performing a start-up operation and a self-diagnosis. The optical unit 101 has two pairs of a lens and an image pickup element. The optical unit 101 determines exposure settings such as shutter speed and gain based on the information from the control unit 106, performs image pickup by each image sensor, and sends the two acquired image data to the signal processing unit 102. .. The signal processing unit 102 corrects the brightness / color / distortion of the two image data sent from the optical unit 101, and sends the corrected two image data to the distance calculation unit 103. The distance calculation unit 103 searches for the corresponding points of both images from the image data sent from the signal processing unit 102, calculates the distance to the corresponding points based on the disparity of the corresponding points, and uses the calculated distance information as the object recognition unit. Send to 104. The object recognition unit 104 detects and recognizes an object based on the distance information sent from the distance calculation unit 103, and sends the obtained object information to the vehicle control unit 105. The vehicle control unit 105 performs vehicle acceleration / deceleration / steering, vehicle light source control, wiper control, attention to passengers, and notification of additional information based on the object information sent from the object recognition unit 104. Is generated, and the generated control signal is sent to the vehicle controller 400. The vehicle controller 400 controls the operation of the vehicle based on the transmitted control signal.

FIG. 2 shows an example of a light projection pattern when the headlight 200 emits structured light. For the sake of explanation, only the projection of one of the headlights 200 is shown. It is shown that the illuminance is higher in the filled part closer to white, and the illuminance is lower in the filled part closer to black. The low-light portion and the high-light portion are alternately projected along the circumferential direction centered on the headlight 200. When this projection pattern is captured by the stereo camera 100, parallax is generated at the edge portion generated at the boundary between the low illuminance portion and the high illuminance portion even if the surroundings are dark or the image has low contrast. Obtainable.

The projection pattern of FIG. 2 is darker as it is closer to the center (headlight 200) along the radial direction and brighter as it is farther from the center. Generally, the brightness on the captured image is lower as the distance increases, but by increasing the illuminance as the distance increases as shown in FIG. 2, the edges formed on the image can have the same brightness difference regardless of the distance. As a result, parallax can be stably obtained even at a distance.

The floodlight pattern of FIG. 2 is formed in a substantially fan shape centered on the headlight 200, and the pattern width of the low illuminance portion and the pattern width of the high illuminance portion are wider as they are farther away. Due to the effect of the perspective method, the pattern width on the captured image generally becomes narrower as the distance increases and the distinctiveness deteriorates. However, by widening the pattern width as the distance increases as shown in FIG. Become. As a result, parallax can be stably obtained even in the distance.

FIG. 3 is an example of a complementary pattern. The headlight 200 projects light while alternately switching between the structured pattern shown in FIG. 2 and the interpolation pattern shown in FIG. As a result, parallax can be obtained at the edge portion generated at the boundary between the low illuminance portion and the high illuminance portion in any of the projection periods of the structured pattern and the interpolation pattern.

FIG. 4 is a result of time averaging the structured pattern of FIG. 2 and the structured pattern of FIG. By averaging these structured patterns over time, the dropped light has a uniform luminance distribution in which the structured pattern cannot be seen, or a smooth luminance distribution in which the structured pattern cannot be seen, as shown in FIG. As a result, it is possible to realize light projection that does not give a sense of discomfort to passengers and pedestrians.

5 to 8 are time charts for explaining the timing of switching the projection pattern. It is desirable to avoid the exposure period of the camera when switching between the structured pattern and the interpolation pattern. This is because when the pattern is switched during the exposure period, the difference in brightness between the edges of the low-light portion and the high-light portion may decrease due to the exposure of the two patterns to the acquired image, or a similar pattern may occur. This is because erroneous matching in which the edges that should be originally matched are mistaken in the image matching process is likely to occur. Mismatching will be described later with reference to FIG.

FIG. 5 is an example of pattern switching timing assuming a global shutter method in which the exposure timings of each row of the image sensor are simultaneous. This assumes a CCD or a global shutter type CMOS in which the exposure timings of all the rows of the image sensor are the same. In this case, since the timing at which the exposure of all the pixels ends is the same, the patterns 501 and 502 are switched at the end of the exposure.

FIG. 6 is an example assuming a rolling shutter type image sensor in which the exposure timing of each row of the image sensor is deviated. If there is a period during which the exposure is not performed in any of the rows between the end of the exposure in the last row and the start of the exposure in the first row, the patterns 511 and 512 are switched in that period.

FIG. 7 is an example in which the exposure time is relatively long compared to the frame period. In this case, as shown in FIG. 7, there is no period during which the exposure of any row is not performed. In this case, the headlight 200 switches the patterns 521 and 522 every two frames, for example, and the stereo camera 100 obtains parallax using the image acquired in the frame immediately after the pattern is switched. Strictly speaking, when the time from the exposure start time of the first row to the exposure end time of the last row is t1 and the interval between the dotted lines which is the frame period is t2, t1 · (n-1) <t2≤t1 · n The patterns 521 and 522 are switched every n frames with respect to n. In the case of FIG. 7, since n = 2, the patterns 521 and 522 are switched every two frames.

FIG. 8 is an example in which the positional relationship between the image sensor of the camera 1 and the image sensor of the camera 2 is shifted up and down. In this case, as shown in FIG. 8, the exposure timing of the camera 1 and the exposure timing of the camera 2 may be intentionally shifted. In this case, parallax can be detected only in a part of the row of the image sensor. Therefore, the patterns 531 and 532 are switched while avoiding the period from the start of exposure in the first row of the image sensor whose exposure start is late to the end of exposure in the last row of the image sensor whose exposure end is early.

The signals b1 and b2 shown in FIG. 1 are control signals in which the stereo camera 100 indicates a light projection pattern to the headlights 200 and 300. The timing for switching the projection pattern can be instructed based on, for example, the exposure method / frame period / exposure time. Regarding the light intensity / shape pattern / color, etc., a control signal indicating the contents themselves is output from the stereo camera 100, and the headlights 200 and 300 interpret this to generate the structured light instructed by themselves. Alternatively, the stereo camera 100 may output a control signal for individually controlling the elements of the headlights 200 and 300. In any case, it is sufficient that the headlights 200 and 300 can output the desired structured light. The same applies to the laser light source 707 in the second embodiment described later.

In the above description, a floodlight pattern that can accurately detect even a distant object has been described. On the other hand, the range of objects to be detected differs depending on the traveling speed of the own vehicle. For example, when the vehicle is traveling at a speed of 100 km / h, it is desirable to be able to detect an object 100 m or more ahead from the viewpoint of collision avoidance, but when the vehicle is stopped or traveling on a narrow road, the range is wider. It is desirable to be able to detect an object. Comprehensive irradiation of these areas consumes unnecessary power or gives unnecessary discomfort to surrounding vehicles and pedestrians. Therefore, the stereo camera 100 acquires the traveling speed of the vehicle from the vehicle controller 400 by the signal a, and instructs the headlight 200 and the headlight 300 of the projection range.

FIG. 9 is a diagram illustrating a light projection range when the traveling speed is equal to or higher than a predetermined speed. In this case, the structured pattern is projected farther along the traveling direction as compared with the projection range of FIG.

FIG. 10 is a diagram illustrating a light projection range when the traveling speed is a predetermined speed or less. In this case, the structured pattern is projected at a wider angle than the projection range of FIG. The difference between FIGS. 9 and 10 is due to the fact that the range of the object to be recognized differs depending on the traveling speed of the vehicle. The stereo camera 100 can more accurately acquire the surrounding conditions in a required range by changing the projection range according to the speed of the own vehicle.

FIG. 11 is an example of detecting that a preceding vehicle is in front of the own vehicle. In this case, the stereo camera 100 projects a structured pattern within a range that does not reach the preceding vehicle. It is desirable to control the projection range so that the passengers of the preceding vehicle do not feel the glare due to the reflection of the rearview mirror, for example, the structured light is emitted below the bumper or the license plate.

FIG. 12 is an example of the light projection range when the steering wheel is turned to the left. The stereo camera 100 acquires the steering angle of the vehicle from the vehicle controller 400 by the signal a, and acquires the traveling direction of the vehicle according to the steering angle. The stereo camera 100 instructs the headlights 200 and 300 to change the projection range toward the traveling direction of the vehicle. Here, the light projection range is rotated to the left side as compared with FIG. 10 in which the light travels straight at a low speed. By changing the projection range according to the steering angle in this way, it is possible to more accurately acquire the surrounding conditions in the required range.

The stereo camera 100 uses parallax when the object is viewed from two viewpoints in order to measure the object distance. In order to determine that one object is the same between two viewpoints, for example, a process of matching reference points is required. At this time, if the images obtained from the two viewpoints have a plurality of similar feature points or patterns, there is a possibility that so-called erroneous matching occurs in which the part that is not the reference point that should be originally matched is matched as the reference point. There is.

FIG. 13 is an explanatory diagram of matching by the area matching method. To search for an area that matches the area shown by the square in the left figure from the right figure, search on the epipolar line. When the floodlight pattern is a monotonous repeating pattern, there are multiple areas with similar patterns (white dotted lines in the right figure) in addition to the areas that should be matched originally (white solid lines in the right figure), so erroneous matching May occur. In order to prevent this, it is conceivable to change at least one of the light intensity / the projection angle of the low illuminance portion and the high illuminance portion / the pattern width / color of the low illuminance portion and the high illuminance portion for each projection area.

FIG. 14 is an example of changing the light intensity for each floodlight area. The stereo camera 100 sets, for example, the light intensities of the floodlight areas 601/602/603 so as to be different from each other. As a result, the difference in brightness between the low illuminance portion and the high illuminance portion differs for each flooded area, so that erroneous matching can be prevented.

FIG. 15 is an example of the light intensity of each projection area of the structured pattern and the interpolation pattern. In order from the top row of the table, the light intensity in each floodlight area is shown with the passage of time. In this example, the sum of the intensities of the structured pattern and the interpolation pattern is set to be the same in all the floodlight areas.

FIG. 16 is another example of the light intensity of each projection area of the structured pattern and the interpolation pattern. In this example, the sum of the light intensities in areas 601 to 603 is set to be the same in the period from t = 1 to t = 6. The sum of the light intensities from areas 611 to 614 is set to be the same in the period from t = 1 to t = 8. Further, the intensity of each area may be offset according to the brightness of the imaging range. This makes it possible to realize robust distance measurement performance against changes in the environment. The same applies to the second embodiment described later.

Instead of or in addition to changing the light intensity for each projection area, the emission angles of the projection areas 601 to 603 may be different. Further, the colors of the floodlight areas 601 to 603 may be different from each other. In this case, the structured pattern and the interpolation pattern are illuminated by alternately projecting red / green / blue light, which is close to the three primary colors of light, so that the entire flooded area is white and the brightness is uniformly distributed. .. When different colors are projected for each area, the stereo camera 100 can prevent erroneous matching due to similar patterns by performing matching processing for each pixel type of the image sensor.

When the headlight 200 and the headlight 300 have an invisible light source in addition to the visible light source and the structured pattern is projected by the invisible light, the pixel type of the image pickup element having high sensitivity to the invisible light is selected. By performing the matching process using the light source, the parallax calculation and the distance measurement of the object can be performed more quickly.

<Embodiment 1: Summary>
The stereo camera 100 according to the first embodiment changes at least one of the light intensity of the structured light, the pattern width of the structured light, and the pattern size of the structured light according to the distance from the headlight 200. As a result, the pattern edge can be clearly identified on the captured image even if the object is distant, so that the identification accuracy can be improved.

The stereo camera 100 according to the first embodiment changes at least one of light intensity / pattern width / pattern size / color for each projection area. This makes it possible to suppress erroneous matching between stereo images when a similar pattern is included on the captured image.

In the first embodiment, for example, the light intensity of the structured pattern can be smoothly changed according to the projection distance, but the present invention is not limited to this. For example, the light intensity can be changed discretely by dividing the light projection distance into a plurality of ranges and setting different light intensities for each range. Even in this case, the same effect can be obtained. The same applies to the second embodiment described later.

In the first embodiment, the traveling speed of the own vehicle can be divided into three speed groups of low speed / medium speed / high speed, and the projection range can be set for each speed group, but the present invention is limited to this. Not done. For example, the light projection range can be continuously changed according to the speed of the own vehicle. Even in this case, the same effect can be obtained. The same applies to the second embodiment described later.

In the first embodiment, the headlights 200 and 300 are supposed to emit structured light, but the present invention is not limited to this. For example, the same effect can be obtained even with a fog lamp / stop lamp / tail lamp or the like. The same effect can be obtained by using a side turn lamp when turning and a rear turn lamp when moving backward.

In the first embodiment, both the left and right headlamps are used to project the structured light, but the present invention is not limited to this. For example, only the right headlamp or the left headlamp may be used to project the structured light. Alternatively, even if an area where parallax cannot be detected or an area where the detection accuracy is unstable is detected and structured light is projected only in the area, the same effect can be obtained.

In the first embodiment, the object recognition process and the vehicle control signal generation are performed inside the stereo camera 100, but the present invention is not limited to this. For example, the same effect can be obtained even if the distance calculation result or the information obtained by simplifying the distance calculation result is output to the calculation processing unit or the control unit mounted on the vehicle. The same applies to the second embodiment described later.

In the first embodiment, it has been described that the light intensity / shape pattern of the structured light is changed according to the projection distance. The projection distance here is the distance from the light source to the projection point, which is determined by the height from the road surface to the light source and the depression angle of the structured light. The stereo camera 100 stores in advance the relationship between the projection direction of the structured light and the projection distance, and according to the relationship, the structured light corresponding to the projection distance is directed to the headlights 200 and 300. Just point to it. The same applies to the laser light source 707 in the second embodiment described later.

<Embodiment 2>
FIG. 17 is a block diagram of the stereo camera 700 according to the second embodiment of the present invention. The stereo camera 700 projects structured light using a laser light source. The optical unit 701 to the control unit 706 are functional units similar to the optical unit 101 to the control unit 106 in the first embodiment. The vehicle equipped with the stereo camera 100 includes a laser light source 707 and a vehicle controller 400. The laser light source 707 controls the pattern / projection area / light intensity of the structured light based on the signal sent from the control unit 706. The signals a and b will be described later.

FIG. 18 is an example of a light projection pattern when the laser light source 707 projects structured light. For the sake of explanation, the structured light is shown by two straight lines extending in the traveling direction. The darker the color, the higher the illuminance, and the lighter the color, the lower the illuminance. The light intensity of each straight line is increased as the distance from the vehicle or the stereo camera 700 increases. The width of each straight line is made thicker as the distance from the vehicle or the stereo camera 700 increases.

When the stereo camera 700 captures the projection pattern of FIG. 18, parallax is generated at the edge portion generated at the boundary between the low-light portion and the high-light portion even in an environment where the surroundings are dark or the captured image has low contrast. Obtainable. In particular, by increasing the illuminance as the distance increases, the edges formed on the image can have the same luminance difference regardless of the distance, so that parallax can be stably obtained even at a distance. Further, by increasing the width of the pattern as the distance increases, it is possible to prevent the edge spacing formed on the image from becoming smaller according to the distance, the edges can be identified farther, and the parallax can be stably obtained over the distance. be able to.

In FIG. 18, the structured pattern is two straight lines, but since the luminance difference can be obtained only at the edge portion of the straight line, if the number of straight lines is small, the parallax is partially detected. On the other hand, as the number of straight lines increases, the probability of erroneous matching increases, so it is necessary to devise ways to prevent similar patterns from appearing in the floodlight pattern. This device will be described with reference to FIG.

FIG. 19 is a diagram showing a floodlight pattern for distinguishing two similar straight lines. Each straight line is projected onto two areas outside the optical axis of both cameras. However, as shown in FIG. 19, by arranging a straight line outside the optical axis of the two cameras of the optical unit 701, the vanishing point (the point where the parallel lines in perspective converge) and the straight line on the captured image are aligned with each other. It is possible to identify which straight line is based on the relative position between them. The judgment criteria will be described with reference to FIG. 20 below.

FIG. 20 is an example of a determination criterion for identifying which straight line the linear pattern on the captured image is. The straight line 901 is to the left of the vanishing point when viewed from the left camera and the right camera. The straight line 902 is on the right side of the vanishing point when viewed from the left camera and the right camera. According to these criteria, the stereo camera 700 can identify which straight line is on the captured image.

As another method for identifying the linear pattern, a method of projecting a straight line at a different position for each projection pattern and switching the pattern over time can be considered. In addition, a method of forming a pattern in which similar patterns do not occur at the same time can be considered by appropriately setting the light intensity / straight line width / color / arrangement of the straight line pattern.

In the method of switching patterns over time, it is desirable to avoid the exposure period of the camera when switching patterns. When the patterns are switched during the exposure period, the acquired image is exposed to two patterns, so that the brightness difference between the edges of the low-light portion and the high-light portion may decrease, or the image may be similar due to the occurrence of similar patterns. This is because erroneous matching is likely to occur in the matching process. Specific examples of pattern switching have been described with reference to FIGS. 5 to 8 of the first embodiment and will be omitted.

FIG. 21 is an example of a method of distinguishing a straight line by the light intensity / straight line width / arrangement of the straight line pattern. In FIG. 21, a straight line 903/904/905 with three light intensities of weak / medium / strong is projected on the right side of the optical axis of the left and right cameras, and three light intensities of weak / medium / strong are also projected on the left side. The straight line 906/907/908 is projected. As a result, six straight lines can be displayed in the captured image of one frame, and parallax can be obtained simultaneously and with finer spatial resolution than the example shown in FIG. Furthermore, the number of straight lines displayed at the same time can be further increased by displaying lines having different light intensities so as to be adjacent to each other and changing the order. When monochromatic light is used, the parallax calculation and distance measurement of the object can be performed more quickly by performing the matching process in advance using the pixel type of the image sensor having high sensitivity to the light of the wavelength.

FIG. 22 is an example of a method of distinguishing a straight line by the color / arrangement of the straight line pattern. In FIG. 22, a red / green / blue three-color straight line 909/910/911 is projected on the right side of the optical axes of the left and right cameras, and a red / green / blue three-color straight line 912/913 is also projected on the left side. / 914 is flooded. As a result, six straight lines can be displayed in the captured image of one frame, and parallax can be obtained simultaneously and with finer spatial resolution than the example shown in FIG. In this case, it is desirable that the matching process in parallax detection be performed independently for each red / green / blue pixel type. Furthermore, by combining light intensity / arrangement order / color, parallax can be obtained simultaneously and with finer spatial resolution than the examples shown in FIGS. 21 and 22.

FIG. 23 is a diagram illustrating a light projection range when the traveling speed is equal to or higher than a predetermined speed. The stereo camera 700 acquires the traveling speed of the vehicle from the vehicle controller 400 via the control unit 706 by the signal a, and indicates the display range to the laser light source 707. When the traveling speed is equal to or higher than the predetermined speed, the structured pattern is projected farther along the traveling direction than the projection range of FIG.

FIG. 24 is a diagram illustrating a light projection range when the traveling speed is a predetermined speed or less. In this case, the structured pattern is projected at a wider angle than the projection range of FIG. The difference between FIGS. 23 and 24 is for the same reason as the difference between FIGS. 9 and 10.

When it is detected that there is a preceding vehicle or a pedestrian in front of or in the vicinity, the projection distance determined by the traveling speed is limited as described with reference to FIG. 11 of the first embodiment. That is, at least limit the projection distance of the structured light in the direction in which the preceding vehicle is present, and prevent the laser light from being seen by the passengers of the preceding vehicle through the rearview mirror, for example, below the bumper or license plate. The projection range is controlled so that structured light is emitted. In addition, at least the projection distance of the structured light in the direction in which the surrounding pedestrians are present is limited so that the structured light is irradiated to the lower half of the pedestrian, for example, so that the laser does not enter the pedestrian's eyes. Control the projection range.

FIG. 25 is an example of the light projection range when the steering wheel is turned to the left. The stereo camera 700 acquires the steering angle of the vehicle from the vehicle controller 400 by the signal a, and indicates the display range to the laser light source 707. Here, the projection range is rotated to the left as compared with FIG. 18 which is traveling straight at a low speed. By changing the projection range according to the steering angle in this way, it is possible to more accurately acquire the surrounding conditions in the required range.

FIG. 26 is a configuration example in which the stereo camera 700 is equipped with the acceleration sensor 708. By controlling the projection direction of the structured light in the pitch direction according to the acceleration detected by the acceleration sensor 708, the attitude in the pitch direction of the vehicle varies depending on the difference in the boarding of the rear seats and the storage of luggage in the trunk. Therefore, it is possible to realize floodlight range control with high robust performance.

The laser light source 707 may project linear structured light so as to overlap the optical axis of either of the two camera lenses of the stereo camera 700. That is, structured light is projected on the intersection of the vertical surface including the optical axis and the road surface. In this case, since the linear pattern of the structured light coincides with the optical axis of the camera, the X coordinate of the structured light does not change even if the pitch angle of the vehicle changes (the X coordinate is always substantially the same as the vanishing point). ). Therefore, it is possible to improve the robustness of the matching accuracy with respect to the posture change in the pitch direction of the vehicle. As a result, the X coordinate detected by the other camera can be regarded as parallax as it is, so that the matching process can be simplified.

Since the stereo camera 700 measures the distance using the parallax on the captured image, the distance can be measured if the parallax occurs. For example, instead of using (or in addition to) two cameras, it is possible to measure the distance by a combination of one camera and a laser light source 707. However, in the distance measurement using a stereo camera, in principle, the longer the baseline length of the camera, the farther the object can be measured. The same applies to the case where the distance is measured by the combination of one camera and the laser light source 707. Therefore, in this case, it is desirable that the distance between one camera and the laser light source 707 is equal to or larger than the baseline length of the stereo camera 700. This is because if the interval is smaller than this, the distance measurement accuracy is inferior to that of the stereo camera 700. When measuring the distance using one of the cameras and the laser light source 707, the structured light may be projected so as to be included in the vertical plane parallel to the straight direction of the vehicle.

<Embodiment 2: Summary>
In the second embodiment, the structured light is projected by using the laser light source 707 instead of the headlight mounted on the vehicle. Also in the second embodiment, the same effect as that of the first embodiment can be exhibited. In the second embodiment, the structured light is projected by the laser light source 707, but the present invention is not limited to this. For example, a similar effect can be obtained by using an LED (Light Emitting Diode) light.

In the second embodiment, the laser light source 707 may be arranged at a position away from the stereo camera 700 main body. For example, the same effect can be obtained by arranging the windshield on the upper right or upper left, the rear-view mirror, the headlight, the bumper, or the like.

<Embodiment 3>
FIG. 27 is a block diagram of the camera 1000 according to the third embodiment of the present invention. In the third embodiment, the optical unit 1001 has one pair of a lens and an image pickup element. The signal processing unit 1002 to the control unit 1006 are functional units similar to the signal processing unit 102 to the control unit 106 in the first embodiment. The third embodiment differs from the first embodiment in that the relative position between the edge portion generated at the boundary between the low-light portion and the high-light portion and the optical portion 1001 is known. Other than that, it is the same as that of the first embodiment. Also in the third embodiment, the distance can be measured by the same principle as that of the first embodiment.

<Embodiment 4>
FIG. 28 is a block diagram of the camera 1400 according to the fourth embodiment of the present invention. In the fourth embodiment, the optical unit 1401 has one pair of a lens and an image pickup element. The signal processing unit 1402 to the laser light source 1407 are functional units similar to the signal processing unit 702 to the laser light source 707 in the second embodiment. In the fourth embodiment, the point where the relative position between the edge portion and the optical portion 1401 generated at the boundary between the low illuminance portion and the high illuminance portion is known or the line spacing of the structured pattern is known is the same as the second embodiment. different. Other than that, it is the same as that of the second embodiment. Also in the fourth embodiment, the distance can be measured by the same principle as that of the second embodiment.

<About a modification of the present invention>
The present invention is not limited to the above-described embodiment, and includes various modifications. For example, the above-described embodiment has been described in detail in order to explain the present invention in an easy-to-understand manner, and is not necessarily limited to the one including all the described configurations. Further, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Further, it is possible to add / delete / replace a part of the configuration of each embodiment with another configuration.

Each of the above configurations, functions, processing units, processing means and the like may be realized by hardware, for example, by designing a part or all of them by an integrated circuit or the like. Further, each of the above configurations, functions, and the like may be realized by software by the processor interpreting and executing a program that realizes each function. Information such as programs, tables, and files that realize each function can be placed in a memory, a hard disk, a recording device such as an SSD (Solid State Drive), or a recording medium such as an IC card or an SD card. In addition, the control lines and information lines indicate those that are considered necessary for explanation, and do not necessarily indicate all the control lines and information lines in the product. In practice, it can be considered that almost all configurations are interconnected.

100: Stereo camera 101: Optical unit 102: Signal processing unit 103: Distance calculation unit 104: Object recognition unit 105: Vehicle control unit 106: Control unit 200: Headlight 300: Headlight 400: Vehicle controller 707: Laser light source 708: Acceleration sensor

Claims (24)

An image pickup device that acquires the distance to an object.
A light source that emits structured light,
A camera that acquires an image of the projected portion where the structured light is projected,
Equipped with
The light source changes at least one of the light intensity of the structured light and the shape pattern size of the structured light according to the distance from the light source to the projection point.
The camera acquires the distance from the camera to the object by using the pattern of the structured light contained in the image .
The light source casts the structured light so that the pattern of the structured light has a first portion having a first illuminance and a second portion having a second illuminance higher than the first illuminance. ,
The camera is a stereo camera having a first camera and a second camera.
The camera acquires the distance from the camera to the object by using the parallax generated between the first camera and the second camera at the boundary between the first part and the second part.
An imaging device characterized by this. The light source projects the structured light while changing the light intensity of the structured light between two or more light intensities.
When the distance is the first distance, the light source projects the structured light having the first light intensity.
The light source is characterized in that when the distance is a second distance longer than the first distance, the light source emits the structured light having a second light intensity stronger than the first light intensity. Item 1. The image pickup apparatus according to Item 1. The image pickup apparatus according to claim 1, wherein the light source projects the structured light having a shape pattern in which the width becomes wider as the distance from the light source to the projection point increases. The image pickup device further includes a speed acquisition unit for acquiring the moving speed of the image pickup device.
The light source projects the structured light while switching the shape pattern of the structured light between two or more shape patterns.
When the moving speed is the first speed, the light source projects the structured light having the first shape pattern.
The structured light having a second shape pattern arriving at a position farther from the light source than the first shape pattern when the moving speed is a second speed faster than the first speed. The image pickup apparatus according to claim 1, wherein the light source is projected. The light source projects the first structured light to the first floodlight region and the second structured light to the second floodlight region.
The light intensity of the first structured light and the light intensity of the second structured light, or the shape pattern size of the first structured light and the shape pattern size of the second structured light, or the first. The image pickup apparatus according to claim 1, wherein at least one of the color of the first structured light and the color of the second structured light are different from each other. The light source is characterized in that at least one of the light intensity of the structured light and the shape pattern size of the structured light is switched during a period other than the period in which the image sensor included in the camera is exposed. The image pickup apparatus according to claim 1. When the difference between the exposure start time of the pixel to be exposed first and the exposure end time of the pixel to be exposed last in the predetermined range of the image sensor is t1 and the frame period is t2.
When t1> t2, for n such that t2 × (n-1) <t1 <t2 × n (n is a natural number)
The image pickup apparatus according to claim 1, wherein the light source switches at least one of the light intensity of the structured light and the shape pattern size of the structured light for each n-frame. The image pickup device further includes a steering angle acquisition unit that acquires the steering angle of the vehicle on which the image pickup device is mounted.
The image pickup apparatus according to claim 1, wherein the light source acquires the traveling direction of the vehicle according to the steering angle and changes the direction of projecting the structured light toward the traveling direction. The image pickup device further includes an illuminance acquisition unit that acquires the illuminance of the flooded portion.
The imaging device according to claim 1, wherein the light source changes the light intensity of the structured light according to the illuminance. The light source is
The first structured light having the first illuminance is projected onto the first floodlight region, and the second structured light having a second illuminance higher than the first illuminance is projected onto the second floodlight region. The first flooding mode to flood and
A third structured light having a third illuminance higher than the first illuminance is projected onto the first floodlight region, and a fourth structured light having a fourth illuminance lower than the second illuminance is emitted. A second floodlight mode in which light is projected onto the second floodlight region, and
2. Imaging device. The image pickup device according to claim 1, wherein the light source is a headlight included in a vehicle equipped with the image pickup device. The image pickup apparatus according to claim 1, wherein the light source is a laser light source or an LED light source. The light source is a laser light source or an LED light source.
The light source according to claim 1 , wherein the light source projects the structured light having a linear shape on a straight line including the line of intersection between the vertical surface including the optical axis of the lens of the first camera and the road surface. Imaging device. The light source is a laser light source or an LED light source.
The image pickup apparatus according to claim 1 , wherein the light source is arranged at a position separated from the stereo camera by a base line length or more of the stereo camera. An image pickup device that acquires the distance to an object.
A control unit that controls the light source that emits structured light,
A camera that acquires an image of the projected portion where the structured light is projected,
Equipped with
The control unit causes the light source to change at least one of the light intensity of the structured light and the shape pattern size of the structured light according to the distance from the light source to the projection point. Outputs a control signal to instruct,
The camera acquires the distance from the camera to the object by using the pattern of the structured light contained in the image .
The control unit projects the structured light so that the pattern of the structured light has a first portion having a first illuminance and a second portion having a second illuminance higher than the first illuminance. The control signal to be caused is output to the light source, and the control signal is output.
The camera is a stereo camera having a first camera and a second camera.
The camera acquires the distance from the camera to the object by using the parallax generated between the first camera and the second camera at the boundary between the first part and the second part.
An imaging device characterized by this. The control unit outputs the control signal instructing the light source to emit the structured light while changing the light intensity of the structured light between two or more light intensities.
When the distance is the first distance, the light source projects the structured light having the first light intensity.
The light source is characterized in that when the distance is a second distance longer than the first distance, the light source emits the structured light having a second light intensity stronger than the first light intensity. Item 15. The image pickup apparatus according to Item 15. The control unit outputs the control signal instructing the light source to emit the structured light having a shape pattern in which the width becomes wider as the distance from the light source to the projection point increases. 15. The image pickup apparatus according to claim 15 . The image pickup device further includes a speed acquisition unit for acquiring the moving speed of the image pickup device.
The control unit outputs the control signal instructing the light source to emit the structured light while switching the shape pattern of the structured light between two or more shape patterns.
When the moving speed is the first speed, the light source emits the structured light having the first shape pattern.
The structured light having a second shape pattern arriving at a position farther from the light source than the first shape pattern when the moving speed is a second speed faster than the first speed. 15. The image pickup apparatus according to claim 15 , wherein the light source is projected. The control unit instructs the light source to project the first structured light to the first floodlight region and to flood the second structured light to the second floodlight region. Output the control signal to be
The light intensity of the first structured light and the light intensity of the second structured light, or the shape pattern size of the first structured light and the shape pattern size of the second structured light, or the first. The image pickup apparatus according to claim 15 , wherein at least one of the color of the first structured light and the color of the second structured light are different from each other. The control unit switches at least one of the light intensity of the structured light and the shape pattern size of the structured light during a period other than the period during which the image sensor included in the camera is exposed. The image pickup apparatus according to claim 15 , wherein the control signal instructed to the light source is output. The image pickup device further includes a steering angle acquisition unit that acquires the steering angle of the vehicle on which the image pickup device is mounted.
The control unit acquires the traveling direction of the vehicle according to the steering angle, and obtains the traveling direction of the vehicle.
The imaging according to claim 15 , wherein the control unit outputs the control signal instructing the light source so as to change the direction in which the structured light is projected toward the traveling direction. Device. The image pickup device further includes an illuminance acquisition unit that acquires the illuminance of the flooded portion.
The imaging device according to claim 15 , wherein the control unit outputs the control signal instructing the light source to change the light intensity of the structured light according to the illuminance. The control unit
The first structured light having the first illuminance is projected onto the first floodlight region, and the second structured light having a second illuminance higher than the first illuminance is projected onto the second floodlight region. The first flooding mode to flood and
A third structured light having a third illuminance higher than the first illuminance is projected onto the first floodlight region, and a fourth structured light having a fourth illuminance lower than the second illuminance is emitted. A second floodlight mode in which light is projected onto the second floodlight region, and
Is instructed to the light source to smooth the illuminance of the structured light in the first floodlight region and the illuminance of the structured light in the second floodlight region, respectively. The image pickup apparatus according to claim 15 , wherein the control signal is output. The image pickup device according to claim 15 , wherein the light source is a headlight included in a vehicle equipped with the image pickup device.

Images