JP4625418B2 - Shadow detection method - Google Patents

Description

  The present invention relates to a shadow determination method for improving image processing technology when performing vehicle parking assistance, robot object detection, or the like based on a captured image captured by an imaging device.

In recent years, a parking assistance technique for assisting parking at a parking target position when a vehicle is parked, a control technique for allowing a robot to automatically advance to a target position, and the like have been devised. For example, Patent Literature 1 below discloses a parking assist technology that can park a vehicle at a parking target position in a short time.
JP 2005-313710 A

  In the conventional technology, for example, an imaging device is mounted on a moving body such as a robot or a vehicle, and the moving direction of the moving body is imaged by the imaging device, and the captured image is visually recognized or object detection is performed on the captured image. The safety in the direction of travel is confirmed by performing processing. However, when the shadow of the moving object is projected on the periphery of the moving object such as the traveling direction, the shadow appears in the captured image, and the visibility of the moving object such as the traveling direction becomes poor. There is a problem that an object existing in a shadow or a road surface unevenness cannot be detected. In this case, there is a possibility that the moving object may collide with the object or ride on the road surface unevenness without noticing the object existing in the shadow and the road surface unevenness, and there is also a problem in terms of safety. .

  In order to solve the above problem, the present invention determines a shadow of a moving body (self-shadow) from shadows present in a captured image captured by a camera installed on the moving body. It aims to provide a method.

In order to achieve the above object, the self-shadow determination method of the present invention determines a shadow of the moving object in an image captured by an imaging device installed on the moving object so as to move with the moving object. A shadow determination method executed by a computer for
An image acquisition step of acquiring the image captured by the imaging device;
A shadow recognition result acquisition step of acquiring a shadow existing in the image recognized by the shadow recognition device as a shadow recognition result;
From the shadow recognition result acquired in the shadow recognition result acquisition step, it is determined whether or not there is a shadow in a predetermined self-shadow determination area in the image, and a shadow exists in the predetermined self-shadow determination area. A shadow determination step of determining a group of shadows including a shadow of the predetermined shadow determination area as a shadow of the moving object ,
Have.
This makes it possible to determine the shadow (self-shadow) of the moving object from the shadows present in the captured image captured by the camera installed on the moving object. For example, the influence of the self-shadow on the captured image. It is possible to improve the safety of movement of the moving body by performing a process such as reducing the amount of movement.

Furthermore, in addition to the above-described self-shadow determination method, the self-shadow determination method of the present invention determines a part of an image area where the moving body is imaged as a cut-out area and performs a process of expanding the cut-out area. The predetermined shadow determination area is determined.
Thereby, it is possible to reliably specify the shadow projected by the moving body in the captured image captured by the camera installed on the moving body.

Furthermore, in addition to the above-described self-shadow determination method, the self-shadow determination method of the present invention deploys a part of the image region used for creating the predetermined shadow determination region at the lower center of the photographed image. This corresponds to the deformation of the shadow due to the light source position.
Thus, it is possible to reliably specify the shadow projected by the moving body without being affected by the change of the light source position or the deformation of the shadow.

  The present invention can determine a shadow (self-shadow) of a moving body from shadows present in a captured image captured by a camera installed on the moving body, and thereby the influence of the shadow of the moving body. This has the effect that the image can be corrected so as to be reduced.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  First, the configuration of a shadow correction system using the shadow determination method according to the embodiment of the present invention will be described with reference to FIG. FIG. 1 shows an example of the configuration of a shadow correction system using the shadow determination method according to the embodiment of the present invention.

  The shadow correction system illustrated in FIG. 1 includes a camera (imaging device) 11, a shadow recognition device 12, a shadow determination device 13, a shadow correction device 14, a display monitor 15, a parking support device 16, and an object detection device 17. Have. The shadow determination device 13 includes a shadow determination unit 13a and a reference information storage unit 13b.

  In FIG. 1, the camera 11 has a function of performing imaging in an arbitrary imaging direction, and is installed on the moving body so as to move together with the moving body. The moving body is, for example, a robot that moves autonomously or by remote control in addition to a vehicle, a ship, an airplane, or the like. The captured image captured by the camera 11 is supplied to the shadow recognition device 12 and the shadow correction device 14. Note that the captured image captured by the camera 11 may be a still image or a moving image.

  In addition, the shadow recognition device 12 has a function of recognizing a shadow present in a captured image captured by the camera 11. Note that the present invention is not particularly limited to a method for recognizing a shadow present in a captured image, that is, the shadow recognition device 12 recognizes an arbitrary shadow in order to recognize a shadow present in the captured image. Technology can be used. The shadow recognition result by the shadow recognition device 12 is supplied to the own shadow determination device 13.

  The shadow determination device 13 has a function for executing the shadow determination method according to the present invention. The shadow determination unit 13a of the shadow determination device 13 is based on the shadow recognition result supplied from the shadow recognition device 12, and the shadow of the moving body on which the camera 11 is mounted (referred to as shadow in this specification). And a shadow other than that (a shadow created by another object) is identified. The shadow determination function by the shadow determination unit 13a can be realized by hardware and / or software executed by a computer.

  Further, the reference information storage unit 13b of the self-shadow determination device 13 stores reference information used when the self-shadow determination unit 13a determines the shadow. In addition, reference information is the information etc. which show the below-mentioned own vehicle partial image and self-shadow determination area | region, for example. The shadow determination result by the shadow determination unit 13 a is supplied to the shadow correction device 14.

  The shadow correction device 14 identifies a shadow in the captured image supplied from the camera 11 by referring to the shadow determination result supplied from the shadow determination device 13, and determines the image quality of the shadow region in the captured image. It has a function to correct (improve). The shadow correction device 14 can reduce the influence of the shadow on the captured image by correcting the shadow area. Note that the present invention is not particularly limited to the correction method of the self-shadow existing in the captured image, that is, the shadow correction device 14 is not limited to the arbitrary method for correcting the self-shadow existing in the captured image. Shadow correction techniques can be used. The captured image (corrected captured image) whose shadow has been corrected by the shadow correction device 14 can be used for various purposes.

  For example, the corrected captured image generated by the shadow correction device 14 is sent to the display monitor 15 and displayed. In the captured image after correction, the image quality of the self-shadow region is improved, and the visibility of the image region that should be difficult to be seen by the self-shadow is improved.

For example, the corrected captured image generated by the shadow correction device 14 is sent to the parking assist device 16. In the parking assistance device 16 , for example, based on the image captured by the camera 11, assistance for parking the moving body (vehicle) at a desired parking target position is performed, but instead of the image captured from the camera 11 including the shadow, By using the corrected captured image from which the influence of the shadow is removed, safer and more reliable parking assistance is performed.

For example, the corrected captured image generated by the shadow correction device 14 is sent to the object detection device 17 . Also in the object detection device 17 , the object (obstacle) existing around the moving body including the traveling direction is detected based on the image captured by the camera 11 as in the parking assistance device 16 . Therefore, also in the object detection device 17 , the object detection is performed more reliably by using the corrected captured image from which the influence of the shadow is removed instead of the captured image from the camera 11 including the shadow. It becomes like this.

  Next, an example of the operation of the shadow correction system shown in FIG. 1 will be described with reference to FIG. FIG. 2 is a flowchart showing an example of the operation of the shadow correction system using the shadow determination method according to the embodiment of the present invention. In the operation example described below, the shadow correction system shown in FIG. 1 is applied to a parking assistance system, and the moving body is a vehicle 31 as shown in FIG. A case where the rear camera 32 (corresponding to the camera 11 in FIG. 1) installed in the vehicle 31 is installed in the rear part of the vehicle 31 so as to capture the rear of the vehicle 31 will be described.

  As shown in FIG. 2, first, for example, when the shift lever of the vehicle 31 is operated to the back position, the rear camera 32 starts imaging the rear of the vehicle 31 (step S100).

  Here, it is assumed that the captured image captured by the rear camera 32 is, for example, an image as illustrated in FIG. 3B. That is, the rear camera 32 images a part 34 a of the host vehicle, a three-dimensional object 35 a, two white lines 36, and two garage stops 37. Note that a light source such as the sun is present on the front right side of the vehicle 31, and the shadow 34b of the host vehicle and the shadow 35b of the three-dimensional object are also captured. Further, as shown in FIG. 3C, the captured image captured by the rear camera 32 can be converted into a bird's-eye view image and displayed on the display monitor.

  The shadow recognition device 12 acquires a captured image captured by the rear camera 32, and performs a process of recognizing a shadow present in the captured image (step S101). As described above, the shadow recognition device 12 can perform shadow recognition using an arbitrary shadow recognition technique, but here, as a shadow recognition result, a shadow as illustrated in FIG. 4A is used. 41 and a binary image (shadow image) representing a non-shadow are obtained.

  The shadow determination unit 13a of the shadow determination device 13 that has acquired the shadow recognition result first reads out the own vehicle partial image stored in the reference information storage unit 13b. In addition, the own vehicle partial image is a binary image including a part 42 of the own vehicle and the outside 43 in a state where nothing exists, as illustrated in FIG. 4B, for example. Then, the shadow determination unit 13a of the shadow determination device 13 extracts a shadow candidate by comparing the vehicle partial image with the shadow image (step S102).

  Specifically, in step S102, a difference image as illustrated in FIG. 4C is obtained by taking a difference between the shadow image and the vehicle partial image using, for example, the following equation (1). The shadow included in the difference image can be used as a candidate for the own shadow.

  Here, in Expression (1), Id (x, y), Is (x, y), and Ib (x, y) are the luminance information of the difference image, the luminance information of the shadow image, and the luminance of the own vehicle partial image, respectively. Value.

  If the accuracy of the shadow recognition result by the shadow recognition device 12 is high and shadows and non-shadows can be reliably recognized, the shadow recognition result by the shadow recognition device 12 and the above difference image are almost equivalent. It is. Therefore, when the accuracy of the shadow recognition result by the shadow recognition device 12 is high, the process of step S102 is not necessarily performed.

  Then, the shadow determination unit 13a of the shadow determination device 13 uses the difference image obtained in step S102 to divide the shadow into a lump of shadows (a group of shadows in one connection), and assign an ID (identification information). The shadow analysis process is performed to analyze the candidate for the own shadow (step S103). Specifically, in step S103, shadow analysis is performed using a labeling process. As a result, as shown in FIG. 4C, an ID (label number 1 and label number 2 in FIG. 4C) is assigned to each shadow (shadow 44a and shadow 44b in FIG. 4C).

  Next, the shadow determination unit 13a of the shadow determination device 13 selects a shadow using information indicating the shadow determination area stored in the reference information storage unit 13b (step S104). For example, as illustrated in FIG. 5A, a shadow overlapping with the own shadow determination area 51 is determined as the shadow 52 of the own vehicle, and the other is determined as a shadow 53 of another object. Specifically, for example, the self-shadow is determined using the following equation (2).

Here, in Expression (2), S _N is a shadow block, N is a shadow ID (label number), and S _REF is a shadow determination area.

  Note that the self-shadow determination area can be set from, for example, the vehicle partial image. For example, as shown in FIG. 5B, the self-shadow determination area is a cut-out area 54 obtained by cutting out a part of an area where a part 42 of the own vehicle in the own-car partial image is imaged. By performing the above process, the region 55 obtained by the expansion can be set as the shadow determination region.

  Further, it is desirable that the self-shadow determination area is arranged on the lower center side of the captured image captured by the rear camera 32 for the following reason described with reference to FIGS. 6A to 6D.

  FIG. 6A illustrates a self-shadow (shadow of the own vehicle) 63 projected by the vehicle 62 when the light source 61 is present on the right front side of the vehicle 62. When a shadow as illustrated in FIG. 6A is formed, the captured image captured by the rear camera 32 is largely reflected by the shadow 63 of the host vehicle in the captured image as illustrated in FIG. 6B. It will be.

  On the other hand, FIG. 6C shows a self-shadow (shadow of the own vehicle) 65 projected by the vehicle 62 when the light source 61 is present on the right side of the vehicle 62. When a shadow as illustrated in FIG. 6C is formed, the captured image captured by the rear camera 32 is projected horizontally in the vicinity of a part 64 of the host vehicle as illustrated in FIG. 6D. become.

  As can be seen by comparing the captured images in FIGS. 6B and 6D, in the captured image captured by the rear camera 32, the area of the shadow of the host vehicle varies greatly depending on the position of the light source. Regardless, when a self-shadow appears in the captured image, the vicinity of part 64 of the own vehicle (directly above part 64 of the own vehicle) is always a self-shadow area. 6A to 6D, the case where the light source 61 is present on the right side of the vehicle has been described. However, when the light source 61 is present on the left side of the vehicle, a symmetrical shadow is generated. In fact, since the probability that the light source exists on the right side and the left side of the vehicle is the same, it can be said that the self-shadow is basically reflected in the lower center of the captured image. Therefore, it is effective to arrange a self-shadow determination area for detecting a self-shadow on the lower center side of the captured image.

  In step S <b> 104, the selection result of the shadow by the shadow determination unit 13 a of the shadow determination apparatus 13 is supplied to the shadow correction apparatus 14. For example, the shadow correction device 14 corrects the luminance information of the self-shadow region (increases the luminance of the self-shadow region) using luminance information of pixels around the self-shadow region (pixels of the non-shadow region). To correct the image so as to reduce the influence of the shadow on the captured image (step S105).

  The corrected captured image in which only the self-shadow area is corrected so that the influence of the self-shadow is reduced in step S105 is displayed from the display monitor 15, for example. As a result, as shown in FIG. 7A, an image with little influence of the shadow is displayed on the display monitor 15. For example, the obstacle 70 that is difficult to see due to the shadow can be clearly confirmed. Further, as illustrated in FIG. 7B, the image in FIG. 7A can be converted into a bird's-eye view image and then displayed on the display monitor 15.

  As described above, according to the present invention, the moving object is determined by determining the shadow (self-shadow) of the moving object from the shadows present in the captured image captured by the camera installed on the moving object. The image can be corrected so that the influence of the shadow of the body is reduced. In particular, when the camera is installed so as to capture the moving direction of a moving body, the influence of the shadow in the captured image is reduced, so that objects existing in the traveling direction and unevenness of the road surface are more reliably detected. Thus, it is possible to confirm (or detect), and to improve the safety of the moving body.

According to the present invention, there is provided a self-shadow determination device for determining a shadow of the moving body in an image captured by an imaging device installed on the moving body so as to move with the moving body. ,
Image acquisition means for acquiring the image captured by the imaging device;
Shadow recognition result acquisition means for acquiring a shadow present in the image recognized by the shadow recognition device as a shadow recognition result;
From the shadow recognition result acquired by the shadow recognition result acquisition means, a self-shadow determination means for determining the shadow of the moving body in the image,
A self-shadowing determination apparatus is also provided.

  Moreover, according to this invention, the program for making a computer perform the shadow determination method of this invention is also provided.

  According to the present invention, the influence of the shadow of the moving object is reduced by determining the shadow (self-shadow) of the moving object from the shadows present in the captured image captured by the camera installed on the moving object. Image processing technology for performing vehicle parking assistance, robot object detection, and the like based on the captured image captured by the imaging device, and user assistance technology for moving the moving body It can be applied to the movement control technology of a moving body.

It is a figure which shows an example of a structure of the shadow correction system using the shadow determination method in embodiment of this invention. It is a flowchart which shows an example of operation | movement of the shadow correction system using the shadow determination method in embodiment of this invention. It is a figure which shows typically the vehicle to which the shadow determination method in embodiment of this invention is applied. It is a figure which shows an example of the captured image imaged with the rear camera shown by FIG. 3A. It is a figure which shows an example of the bird's-eye view image which concerns on the captured image imaged with the rear camera shown by FIG. 3A. It is a figure which shows an example of the shadow recognition result (shadow image) obtained by step S101 of the flowchart shown in FIG. It is a figure which shows an example of the own vehicle partial image used by step S102 of the flowchart shown in FIG. It is a figure which shows an example of the difference image produced by step S102 of the flowchart shown in FIG. It is a figure which shows the concept of the shadow determination area | region used in the case of selection of the shadow in step S104 of the flowchart shown in FIG. It is a figure which shows the concept of the cut-out area | region used when setting the self-shadow determination area | region shown by FIG. 5A, and an expansion process. It is a figure which shows the shadow of the vehicle projected when a light source exists in the front right side of a vehicle. It is a figure which shows an example of the captured image imaged with the rear camera in the condition of FIG. 6A. It is a figure which shows the shadow of the vehicle projected when a light source exists in the side right side of a vehicle. It is a figure which shows an example of the captured image imaged with the rear camera in the condition of FIG. 6C. It is a figure which shows an example of the captured image correct | amended by the self-shadow determination method in embodiment of this invention. It is a figure which shows an example of the bird's-eye view image which concerns on the captured image correct | amended by the self-shadow determination method in embodiment of this invention.

Explanation of symbols

11 Camera (imaging device)
DESCRIPTION OF SYMBOLS 12 Shadow recognition apparatus 13 Self shadow determination apparatus 13a Self shadow determination part 13b Reference information storage part 14 Shadow correction apparatus 15 Display monitor 16 Parking assistance apparatus 17 Object detection apparatus 31, 62 Vehicle 32 Rear camera 34a, 42, 64 Part of own vehicle 34b, 52, 63, 65 Shadow of own vehicle (self-shadow)
35a Three-dimensional object 35b Three-dimensional object shadow 36 White line 37 Garage stop 41 Shadow 43 External 44a Shadow (label number 1)
44b Shadow (label number 2)
51 Self-shadow determination area 53 Shadow of other object 54 Cut-out area 55 Area obtained by expansion 61 Light source 70 Obstacle

Claims (3)

A shadow determination method executed by a computer to determine a shadow of the moving body in an image captured by an imaging device installed on the moving body so as to move with the moving body,
An image acquisition step of acquiring the image captured by the imaging device;
A shadow recognition result acquisition step of acquiring a shadow existing in the image recognized by the shadow recognition device as a shadow recognition result;
From the shadow recognition result acquired in the shadow recognition result acquisition step, it is determined whether or not there is a shadow in a predetermined self-shadow determination area in the image, and a shadow exists in the predetermined self-shadow determination area. A shadow determination step of determining a group of shadows including a shadow of the predetermined shadow determination area as a shadow of the moving object ,
Self-shadow determination method. Defined as part cutout region of an image region where the moving object is imaged, by performing processing for expanding the cutout region, the self-shadow determining method according to claim 1 for determining the predetermined self shadow determination area. 3. The method for determining a shadow according to claim 2 , wherein a part of an image area used for creating the predetermined shadow determination area is arranged at a lower center of the image to cope with a deformation of a shadow due to a light source position. .

Images