JPH01274214A - Image processor for mobile vehicle - Google Patents

Abstract

PURPOSE:To easily perform an image processing in a short time by providing a light projecting means which projects the image pickup area of an image input means, and performing irradiation by detecting the state of the image pickup area and controlling the projecting means by a control means. CONSTITUTION:On an image processor, the light projecting means 6 to irradiates the image pickup area 4 of the image input means 2 is provided, and the irradiating state of the light projecting means 6 is controlled by the control means 8. Such light projection control is performed based on output from a detecting means 10, and also, detection is performed by processing an image from the image input means 2. And the light projecting means 6 is provided with a wide area projector 6a which performs the irradiation on the whole of the image pickup area and a spot projector 6b which performs the irradiation on a part of the area, and also, the detecting means 10 is provided with an image processing means 10a for detecting an illuminating state and an optical sensor 10b. Also, the speed and the travel direction, etc., of a detected image are controlled via an image processing means 16 for recognizing an outside field and a controller 22, etc. In such a way, the adaptability of the illuminating state can be detected, and the image pickup area can be illuminated appropriately.

Description

DETAILED DESCRIPTION OF THE INVENTION (Field of Industrial Application) The present invention relates to an image processing device for a moving vehicle such as an automobile or a robot, which is equipped with an image input means for recognizing the outside world.

(Prior art) For example, Japanese Patent Application Laid-Open No. 61-240307 discloses that a moving vehicle is imaged by a camera serving as an image input means installed at a predetermined point, and the image input to the camera is processed to locate the moving vehicle. A technique has been disclosed in which the travel of a moving vehicle is controlled based on the calculation results.

In addition, unlike the above case, an image input means is installed in the moving vehicle, the image input means captures an image of the outside world of the moving vehicle (the actual driving environment), and the image input to the image input means is used to Techniques have been considered in the past for use in, for example, processing images to recognize the state of the outside world, and controlling the travel of a moving vehicle based on the recognition results.

(Problem to be Solved by the Invention) By the way, in the case where the image input means is installed in a moving vehicle, for example, when the moving vehicle runs indoors, the image captured by the image input means The illumination condition of the area (usually the outside area in front of the moving vehicle) by indoor lighting equipment etc. is not always good. There may be cases where the area is partially dark due to the shadow of some object, or where the illumination is inappropriate for the color of the imaging area.

However, when performing external world recognition by processing images captured by the image input means as described above, it is very important whether the lighting condition of the imaging area is good or not. If the conditions are not good, there will be a problem with the image processing of the input image, making it difficult to accurately recognize the external world, and even if you try to deal with this problem by image processing, the image processing will become complicated and the processing time will increase. Furthermore, there is a problem in that even if such a method is used, a good processing result is not necessarily obtained.

Further, the above-mentioned problem regarding the illumination state of the imaging area may also occur in the case of a moving vehicle such as a general automobile traveling outdoors.

In view of the above circumstances, an object of the present invention is to provide an image processing device for a moving vehicle that can always keep the imaging area in an appropriate illumination state and obtain high-quality images that can easily perform necessary external world recognition. .

(Means for Solving the Problems) In order to achieve the above object, an image processing device for a moving vehicle according to the present invention is an image processing device for a moving vehicle that has an image input means for external world recognition, and includes the following: light projecting means for illuminating the imaging area of the image input means;
and a control means for variably controlling the light irradiation state of the light projection means based on the illumination state of the imaging region detected by the detection means. Features.

The light projecting means may be, for example, a wide area light projector that irradiates the entire imaging area with light, or a spot light projector that irradiates a part of the imaging area with light, or It may be formed by combining a plurality of light projecting devices.

The illumination state of the imaging area mentioned above refers to the state of light in the imaging area that affects the image, and specifically includes, for example, the overall brightness and darkness of the imaging area, the partial brightness and darkness within the imaging area, such as the presence or absence of shadows, and the imaging area. Examples include the suitability of illumination light for the color of the area.

The detection means may be of any type as long as it can detect the illumination state of the imaging area, but for example, it may detect the illumination state of the imaging area by appropriately processing the image input by the image input means. The image processing means may be an image processing means, an optical sensor, or both may be provided.

The above-mentioned light irradiation state that is subject to variable control is the light irradiation state that affects the image, and specifically includes, for example, on/off of light irradiation, the amount of light irradiated, the color of the irradiated light, and in the case of spot lighting. Examples include the irradiation location and the type of light projection, such as whether to perform wide area light projection, spot light projection, or both.

(Function) By providing the above-mentioned light projecting means, detection means, and control means, the illumination condition of the imaging area, for example, when the imaging area is dark, there is a partial shadow within the imaging area, or the color of the imaging area If the illumination is not appropriate for the object, the detection means detects this fact, and based on the detection result, the control means controls the light projecting means to emit wide-area light of an appropriate amount of light or an appropriate color. Irradiation or spot light irradiation can be carried out, so that the imaging area can always be kept in a suitable illumination state, ie, in a state in which an image can easily be obtained that allows the necessary external world recognition.

(Example) Hereinafter, an example of the present invention will be described with reference to the drawings.

FIG. 1 is a block diagram showing an outline of an example of the apparatus according to the present invention, in which a light projecting means 6 for irradiating light onto the imaging area 4 of the image input means 2 is provided. The light irradiation state is controlled (light projection control) by the control means 8 in order to appropriately irradiate the imaging area 4 with light and bring the imaging area 4 into an illumination state in which an image that can be well recognized as the outside world is obtained. The detection of the illumination state of the imaging area by the detection means 10 is performed by appropriately processing the image output from the image input means 2, for example.

FIG. 2 is a block diagram specifically showing an embodiment of the image processing apparatus according to the present invention, and this embodiment will be described in detail below.

In this embodiment, the present invention is applied to an image processing apparatus for a mobile vehicle that uses an image sensor 2 using COD as an image input means and that travels on an indoor running path under automatic control.

This embodiment includes a wide-area light projector 6a that irradiates the entire imaging region with light and a spot light projector 6b that irradiates a part of the imaging region with light as the light projecting means 6, and also detects the illumination state of the imaging region. The detection means 10 includes an image processing means 10a for detecting illumination conditions and an optical sensor lOb.

FIG. 3 shows the image sensor 2 and both light projectors Oa.

6b and a light sensor 10b;
The image sensor 2 and both light projecting devices 8a and '6b are configured to have variable imaging areas 4 and light projecting areas, respectively. The wide-area light projection device 6a is a device that irradiates the entire imaging region 4 with light, and is driven to change the light irradiation region as the image sensor 2 moves and the imaging region changes. Therefore, for example, the wide-area projector 6a may be fixedly attached to the image sensor 2 and moved together with the image sensor 2. Further, both of the light projecting devices 6a and 6b are configured so that not only the light projecting area but also the amount of irradiated light and the color (wavelength) of the irradiated light can be changed and controlled.

Image output from the above image sensor 2 (image signal)
is input to the external world recognition image processing means 16, as shown in FIG. In the processing means 1B, the image signal input from the image sensor 2 is processed by various image processing algorithms to extract edges, areas, etc. indicating the running route.

The extracted edge/area information is input to the local map generation means 18, which generates a local map based on the edge/area information, that is, the state of the outside world in the immediate vicinity of the vehicle. A map is created. This local map information is used by the optimal route generating means 20.
The optimal route to the destination is determined based on the above local map information and information such as the destination that has been input in advance, and the travel direction and travel speed for traveling on the optimal route are determined. Ru. This information regarding the traveling direction and traveling speed is input to the controller 22, and the controller 22 controls the steering and speed of the automobile based on the traveling direction and traveling speed information.

On the other hand, the image output from the image sensor 2 is input to the image processing means 10a for illumination state detection, and the image processing means LO
Light projection control of the light projecting device [ta, 8b] is performed based on the processing result by a and the output from the optical sensor lob.

This control will be explained with reference to the flowchart shown in FIG.

First, the illumination state of the imaging area is detected in Sl. In this embodiment, this lighting state detection is performed by appropriately processing the image output from the image sensor 2 by the lighting state detection image processing means 10a to determine the brightness of the entire imaging area, for example, the average brightness of all pixels of the image. This is done by detecting whether or not the color of each part in the imaging area is appropriately illuminated. In addition,
In this embodiment, an optical sensor tob is also provided as a detection means, but this is an auxiliary provision.For example, when a moving vehicle enters a dark place while it is running, the sensor tob detects the It is conceivable to detect this and omit the calculation of the average luminance by the image processing means 10a.

After the illumination state of the imaging area is detected in this way, the control means 8 controls the light projection of the wide area light projecting device 6a in S2 based on the detection result. That is, if the imaging area is dark based on the above detection results, the imaging area is irradiated with an amount of light corresponding to the darkness, and if the illumination is not appropriate for the color of each part of the imaging area, the illumination is adjusted accordingly. Light projection control is performed to emit light of a different color.

Then, after the wide-area light projecting device 6a projects light over a wide area onto the imaging area under the above control, the image output from the image sensor 2 in the wide-area light projecting state is again subjected to image processing by the illumination state detection image processing means 10a, and S3 In step S4, it is determined whether there is a disturbance in a part of the image, that is, whether there is a shadow part, and if there is no shadow, the external world recognition image processing means 16 performs image processing for external world recognition in step S4. will be carried out.

If there is a shadow, the control means 8 controls the light projection of the spot light projector 6b in S5. In this light projection control, spot light is projected, for example, at the boundary between a shaded area and an unshaded area on the road.

Then, the image output from the image sensor 2 in this spot light projection state is subjected to image processing for external world recognition at 84 in the same manner as described above.

Since the above-mentioned illumination state detection of the imaging area and external world recognition are basically the same, it is possible to share a part of the image processing for the above-mentioned illumination state detection with a part of the image processing for external world recognition. possible, in which case both image processing means 10a, L
[i may be used as one dual-purpose image processing means, and light projection control may be performed according to the flowchart shown in FIG. 5, for example.

In this case, first, in S6, the illumination state of the imaging area is detected by the dual-purpose image processing means in the same manner as 51 in FIG. 4, and then in 87, the control means 8 Wide-area projector control is performed by the following steps, and then image processing for external world recognition is performed in S8. This image processing is performed by inputting the image of the imaging area output from the image sensor 2 and illuminated over a wide area under the control of S7 to the dual-purpose image processing means. Next, in S9, it is determined whether or not there is a disturbance, that is, a shadow, in a part of the image as a result of the image processing for external world recognition. If there is no shadow, the process returns to S6, and if there is a shadow, the process proceeds to SlO and the process is performed there. The same spot light projection device control as 85 in FIG. 4 is performed, and the image in the state where the spot light has been projected is again subjected to external world recognition image processing by the dual-purpose image processing means in S8. This image processing is performed, for example, only on the spot-projected portion, and the results that have already been performed are used for the other portions.

In addition, whether or not there is a shadow at 83 in FIG. 4 above is also determined by Sl.
, and in S2, the light projection control of both the wide area light projector 6a and the spot light projector 6b may be performed, and this point is also the same in the case of the flowchart shown in FIG.

Next, the irradiation amount control and irradiation color control of the wide area light projecting device 6a and the light projection control of the spot light projecting device 6b will be described in more detail while showing specific examples.

First, the irradiation amount control of the wide-area projector 6a will be explained.

FIG. 6(a) is a diagram showing the image 30 output by the image sensor 2. In the map reading, area I is the wall, area ■ is the running route, line B is the boundary between the wall and the running route, and arrow A is the Indicates the direction in which the moving vehicle is traveling.

Now, if the brightness of the imaging area that is the subject of this image is appropriate, for example, the brightness of each pixel on the XX line of the image 30 will be as shown by the solid line in FIG. 6(b), Wall area I
The brightness difference ΔL between the road area 1. II can be reliably classified. However, if the illumination of the imaging area is not appropriate and is too dark, the brightness of each pixel on the x-X line becomes lower overall as shown by the broken line in FIG. 6(b), and as a result, the side area I, The brightness difference ΔL between area II becomes extremely small, making it difficult to distinguish both areas I and II based on the difference in brightness, making it impossible to accurately recognize the external world. Therefore, when the illumination of the imaging area is too dark, the amount of irradiation light is selected so as to provide good brightness, and the illumination is controlled to project over a wide area.

Next, light projection control of the spot light projection device 6b will be explained. FIG. 7(a) is a diagram showing an image 30 output by the image sensor 2, in which areas I,
n, line B1 arrow A is the same as in FIG. 6(a),
The hatched areas are shadow areas. When a shadow exists in a part of such an image, normal image processing can determine whether the shadow is a shadow and does not interfere with driving, or whether it is caused by a wall or the like that interferes with driving. It is difficult to determine whether the In such a case, spot light is projected onto the boundary between the darkened shadow area and the bright area of the driving road area (■), and the spot light is projected onto the boundary area C.
The brightness of each pixel on the xx-ray including Then, as shown by the solid line in FIG. 7(b), the brightness of each pixel before spot light projection is such that there is a large brightness difference ΔL between the shadowed part and the non-shadowed part. Although it is difficult to determine what it is, by performing spot light projection, the brightness of the spot light projection area C becomes as shown by the broken line in Fig. 7(b). Compare this broken line with the solid line. In other words, the brightness of the shadow area after spot lighting is almost the same as that of the driving path area H where spot lighting is not performed, so this darkened area is continuous with the driving path ■ and is part of the driving path. In other words, it can be determined that the dark areas are shadows.

In the above spot lighting control, turning on/off the spot lighting
In addition to off control and light amount control, since light must be projected onto the boundary portion of the shadow area, it is also necessary to control the location of the light projection.

Note that although the irradiation amount and irradiation color are approximately appropriate for the entire imaging area as described above, if there is a disturbance such as a shadow in a part of the imaging area, spot light is applied only to that part as described above. is desirable. This is because in such cases, only spot lighting is sufficient, and wide area lighting is a waste of energy. This is because if you do so, the amount of irradiation in areas that were previously appropriate may become too large and become inappropriate.Furthermore, if there is a large difference in brightness between the shadow area and other areas, wide-area projection is necessary. This is because there are many cases in which the difference in brightness does not become very small even if

Next, color control of the irradiation light from the wide-area projector 6a will be explained.

For example, assume that the image 30 currently output from the image sensor 2 is in the state shown in FIG. This image 30 middle area 1. n, line B, and arrow A are the same as in FIG. 6(a), and a unique color area (2), which is painted in a different color, exists in a part of the travel path area H.

Then, the information that the unique color area ■ exists in this travel road area ■ and that the unique color area ■ can be judged as a part of the travel road area, that is, a driveable area when it is detected, is sent to the outside world. It is assumed that the information has been input in advance to the recognition image processing means (6).

If so, the image processing means 16 only needs to be able to reliably classify the regions I, n, and I[[; for example, if the classification is performed using a histogram of the R-dimensional signal and G-dimensional signal of the image. think of.

In this case, if the illumination state is also in R dimension, G
If the dimensions are also good, the histogram of the R-dimensional signal of all pixels will be as shown in Figure 9A, and the histogram of the G-dimensional signal of all pixels will be as shown in Figure 9B. If you decide the threshold value, from the R-dimensional histogram, area I, area ■, ■
From the G-dimensional histogram, it is possible to distinguish between region I, ■ and region ■. Therefore, from both histograms, region 1
．． n, III can be accurately classified.

However, for example, there are cases where the illumination condition is not appropriate and the R-dimensional light is sufficient but the G-dimensional light is insufficient. In this case, the R-dimensional histogram has no particular problem as shown in Figure 10A, but the G-dimensional histogram has insufficient light in the G-dimensional area as shown in Figure 10B.
, III are all small, and there is no large valley between them on the histogram. Therefore, even if both histograms are combined, it is impossible to distinguish between regions (1) and (2). In such a case, the problem will not be solved even if, for example, general white light irradiation including all wavelength components is performed. This is because when such white light irradiation is performed, the light in the G dimension is appropriate, but the light in the R degree dimension is too strong, so the histogram in the G dimension is divided into areas I and ■ as shown in Figure 11B. Area■
However, the R-dimensional histogram is the first
Area I as shown in Figure 1A.

Both ① and ② shift to the high concentration side, and area 1. 11. H
This is because it becomes impossible to differentiate. Therefore, in such a case, the light projection should be controlled to emit an appropriate amount of only the G-dimensional light, and then the R-dimensional and G-dimensional histograms would be similar to the 12th A.
This is the same as when the illumination condition is good as shown in FIGS. 9A and 9B above, and therefore, from both histograms, each region 1. n,
m can be divided.

Note that color images are generally separated into three components, R, G, and B, and then captured, so if there are many parts of the image that need to be segmented, the three histograms of R, G, and B dimensions may be used to segment the image. In this case, the irradiation color may be controlled so that these three dimensions of irradiation light are appropriate. Further, the irradiation color may be controlled based on the color of the imaging area, for example, with reference to the R, G, or B-dimensional histogram of the image for initial imaging area state detection.

In the present invention, the light projection means irradiates light only when necessary, that is,
This need only be done when the illumination state of the imaging area is inappropriate and when an image is captured by the image input means.

In addition, the control of the light irradiation state of the light projecting means in the present invention can be simply carried out in such a way that light is irradiated to the imaging area so that an image that allows good recognition of the outside world is obtained based on the state of the area. The specific method is not limited to that described in the examples, and it is not necessarily necessary to control all the light irradiation states described in the examples, but only some of them, for example, controlling the irradiation amount in a wide area floodlighting device. It may be only for control.

The image output from the image input means in the present invention may be used for recognizing the outside world, and the manner in which it is used is not particularly limited. In other words, while the above embodiment performs autopilot by recognizing the external world from images, it is also possible to perform partial autopilot, which forcibly performs steering or braking only when an obstacle is detected in front, for example. The image may be processed as necessary and displayed on a CRT, etc. in the moving vehicle to make the driver aware of the outside world. It may be something.

(Effects of the Invention) As described above, the image processing device for a moving vehicle according to the present invention includes a light projection means for projecting light onto an imaging region of an image input means, detects the state of the imaging region by a detection means, Based on the detection result, the control means causes the light projecting means to appropriately illuminate the imaging area, so that the imaging area can always be kept in a good illumination state, thereby facilitating image processing for external world recognition. It is possible to avoid the complexity and lengthening of time, and it is also possible to always accurately recognize the external world based on the input image.

[Brief explanation of the drawing]

Fig. 1 is a block diagram showing an overview of an example of the present invention, Fig. 2 is a block diagram showing an embodiment of the invention, and Fig. 3 is a conceptual side view of an automobile equipped with the embodiment shown in Fig. 2. , FIG. 4 is a flowchart showing the light projection control procedure of the embodiment shown in FIG. 2, FIG. 5 is a flowchart showing another light projection control procedure, and FIGS. 6(a) and 7(a) are respectively Figures showing images, Figure 6(b) and Figure 7(b) are respectively Figure 6(a) and Figure 7(b).
Figure 7 is a diagram showing the brightness of each pixel on the x-X line of the image in (a), Figure 8 is a diagram showing other types of images, Figures 9A and 9B are when the illumination condition is appropriate Figures 10A and 10B are histograms of the R-dimensional signal and G-dimensional signal when the illumination condition is inappropriate, and Figures 11A and 11B are histograms of the R-dimensional signal and G-dimensional signal when the illumination condition is inappropriate. Histograms of R-dimensional signals and G-dimensional signals when light is projected. FIGS. 12A and 12B are histograms of R-dimensional signals and G-dimensional signals when light of appropriate colors is projected. 2... Image input means 4... Imaging area 6... Light projection means 8... Control means 10... Detection means Fig. 1 Fig. 2 Fig. 3 Fig. 4 Fig. 5 Fig. 6 Genus Figure 7

Claims (1)

[Scope of Claims] An image processing device for a moving vehicle equipped with an image input means for external world recognition, comprising: a light projection means for irradiating an imaging area of the image input means;
and a control means for variably controlling the light irradiation state of the light projection means based on the illumination state of the imaging region detected by the detection means. Features: Image processing device for moving vehicles.