JP4632992B2 - Image correction method - Google Patents

Description

  The present invention relates to an image correction method for correcting an image having a region with poor visibility (a region that is difficult to visually recognize) due to an influence of a shadow or the like reflected in the image.

  Some conventional imaging devices (cameras) have an auto iris mechanism for automatically adjusting the brightness of a captured image. This auto iris mechanism is based on the brightness of the captured image (“average brightness value of the entire image (corresponding to the brightness of the image)” or “average brightness value of the center of the captured image”). This is a mechanism for adjusting the amount of light by adjusting the aperture of an aperture provided or controlling the increase or decrease of the shutter speed of an electronic shutter.

  By using this auto iris mechanism, the captured image is adjusted to the optimum brightness even when the brightness of the subject changes during the day or at night, and the captured image or image processing device that is easy for humans to see is highly accurate. Thus, it is possible to obtain a captured image that can be subjected to image processing.

  Patent Document 1 below discloses a technique that improves the control method of the auto iris mechanism. In the technique disclosed in Patent Literature 1, when a region that is difficult to visually recognize (a region with poor visibility) exists in a captured image obtained by imaging the periphery of the vehicle, the brightness of the region that is difficult to visually recognize The amount of light is adjusted accordingly. Thus, for example, even in a case where the visibility of a part of the captured image is poor, such as when a part of the captured image is locally bright or dark, an area that is difficult to visually recognize has an optimal brightness. By capturing an image, it is possible to improve the visibility related to the region that is difficult to visually recognize.

  Note that with the technique disclosed in Patent Document 1, the visibility of a captured image is improved by adjusting the amount of light in the imaging device, so that only real-time visibility can be improved. That is, with the technique disclosed in Patent Document 1, for example, it is not possible to improve the visibility related to a locally poor region existing in a captured image that has already been captured.

On the other hand, by using image processing software (photo retouch tool), the image quality of a captured image captured by the imaging apparatus can be adjusted afterwards. When image processing software (photo retouch tool) is used, it is possible to adjust various characteristics of an image including color tone and contrast in addition to brightness.
JP-A-9-142210

  However, in the technique disclosed in Patent Document 1, the amount of light in the imaging apparatus is adjusted in order to improve the visibility related to the region with poor local visibility in the captured image. With the improvement of the visibility related to the region, the brightness of the other region (the region with good visibility) is also changed, and as a result, the visibility related to the region with good visibility may be deteriorated. .

  For example, when the amount of light entering the imaging device is increased using the technique disclosed in Patent Document 1 in order to improve the visibility of the shadow area reflected in the image, the brightness of the shadow area increases. Although it becomes visible, the brightness of non-shadow areas (non-shadow areas) also increases, and the non-shadow areas that were previously visible may become too bright and become invisible. .

  In addition, for example, a user who is skilled in image processing can perform an image processing operation that improves the visibility related to a locally invisible region existing in a captured image using a photo retouch tool. However, this image processing operation requires labor (labor) and time. For example, improving the visibility of a captured image (moving image) captured by a video camera in real time is not practical. Is possible.

  In order to solve the above problem, the present invention provides an image correction method that makes it possible to improve the visibility of a region having poor visibility locally while maintaining the visibility of the entire image. Objective.

In order to achieve the above object, an image correction method of the present invention is an image correction method executed by a computer to perform image correction,
An image acquisition step of acquiring the image captured by the imaging device;
A visibility recognition result acquisition step of acquiring, as a visibility recognition result, a region with poor visibility present in the image detected by the visibility determination device;
Based on the visibility recognition result acquired in the visibility recognition result acquisition step, an area specifying step for specifying the poor visibility area in the image acquired in the image acquisition step ;
A first feature amount calculating step of calculating a feature amount of the region with poor visibility;
A second feature amount calculating step of calculating a feature amount of a region other than the region with poor visibility;
An image correction step of correcting an image of the region with poor visibility based on the feature amount calculated in the first and second feature amount calculation steps ;
Have
An object detection step of detecting an object moving in the plurality of images that are continuously imaged in time series;
When the object detected in the object detection step exists in a region other than the poor visibility region as a feature amount of the region other than the poor visibility region, an area around the object including the object is included. A feature amount storing step of calculating a feature amount and storing a feature amount of a region around the object including the calculated object;
An object tracking step of following the movement of the object in the plurality of images continuously imaged in time series and determining whether or not the object has entered the poor visibility region;
Have
When it is determined that the object has entered the region with poor visibility, in the image correction step, the feature amount of the region around the object including the object stored in the feature amount storage step is visually recognized. Setting the feature amount of the region other than the poor region to correct the image of the poor visibility region,
Have.
As a result, it is possible to improve the visibility related to the region having poor visibility locally while maintaining the visibility of the entire image. In addition, it is possible to calculate a feature amount representing a feature of a region with poor visibility and a feature amount representing a feature of a region other than the region with poor visibility by statistical processing. In addition, when an object moving in the image locally enters an area with poor visibility, the visibility of the object can be improved.

Furthermore, in the image correction method of the present invention, in addition to the image correction method described above, in the image correction step, a pixel value of a pixel belonging to the poor visibility region is newly calculated, and the poor visibility region is calculated. A pixel value changing step of changing the pixel value of the pixel to which the pixel belongs to the calculated pixel value;
Thereby, only the pixel value of the pixel belonging to the region with poor visibility can be changed, and the visibility related to the region with poor visibility can be improved locally.

Furthermore, in addition to the image correction method described above, the image correction method of the present invention changes the pixel value based on the coordinate position of a pixel belonging to the poor visibility region in the pixel value changing step.
Thereby, for example, it is possible to perform image correction in consideration of the conditions under which the imaging device is installed and the imaging characteristics of the imaging device.

Furthermore, in addition to the above-described image correction method, the image correction method of the present invention belongs to the poor visibility region when the calculated pixel value exceeds a predetermined threshold value in the pixel value changing step. The pixel value of the pixel is set to the original pixel value before the change or the predetermined pixel value.
As a result, it is possible to prevent an excessive correction more than necessary from being performed for the correction of the locally invisible region.

Furthermore, in the image correction method of the present invention, in addition to the image correction method described above, the pixel value changes smoothly at the boundary between the poorly visible region and the region other than the poorly visible region. A pixel value correcting step for correcting the pixel value.
As a result, it is possible to eliminate the uncomfortable feeling given by the corrected image.

Furthermore, the image correction method of the present invention includes, in addition to the image correction method described above, a first feature amount calculation step of calculating a feature amount of the region with poor visibility,
A second feature amount calculating step of calculating a feature amount of a region other than the region with poor visibility, and the feature amount calculated in the first and second feature amount calculating steps in the image correction step. Based on this, the image in the poor visibility region is corrected.
Thereby, it is possible to correct the image of the region with poor visibility based on the comparison result between the feature amount representing the feature of the region with poor visibility and the feature amount representing the feature of the region other than the region with poor visibility. It becomes possible.

Furthermore, in addition to the image correction method described above, the image correction method of the present invention calculates an average value of pixel values of each pixel as the statistical processing result in the first and second feature amount calculation steps.
As a result, it is possible to calculate a feature amount representing a feature of a region with poor visibility and a feature amount representing a feature of a region other than the region with poor visibility by a quick and easy average value calculation process.

Furthermore, in the image correction method of the present invention, in addition to the image correction method described above, in the image correction step, the feature amount of the poor visibility region is changed to the feature amount of the region other than the poor visibility region. In addition, the pixel value of the pixel belonging to the poor visibility region is calculated, and the pixel value of the pixel belonging to the poor visibility region is changed to the calculated pixel value.
Thereby, it is possible to improve the visibility related to the poorly visible region with reference to the visibility related to the region other than the poorly visible region.

Furthermore, in the image correction method of the present invention, in addition to the image correction method described above, the pixel value is a predetermined value when the feature amount of the poor visibility region is calculated in the first feature amount calculation step. Ignore pixels greater than the threshold.
As a result, it is possible to calculate the feature amount of the region with poor visibility by eliminating the influence of pixels having exceptional pixel values.

Furthermore, in the image correction method of the present invention, in addition to the image correction method described above, in the second feature value calculation step, the pixel value is calculated when calculating the feature value of a region other than the region with poor visibility. Ignore pixels smaller than a predetermined threshold.
Thereby, it becomes possible to calculate the feature amount of the region other than the region with poor visibility by eliminating the influence of the pixel having an exceptional pixel value.

Furthermore, in addition to the image correction method described above, the image correction method of the present invention sets a region around the poor visibility region as a region other than the poor visibility region.
Thereby, it is possible to improve the visibility related to the region with poor visibility in accordance with the feature amount of the region around the region with poor visibility.

Furthermore, in the image correction method of the present invention, in addition to the image correction method, the image correction step includes:
An overall image correction step for correcting the entire image acquired in the image acquisition step so as to improve the visibility of the image in the poor visibility region;
An image cutout step of cutting out an image of the region with poor visibility from the image in which the entire image is corrected in the whole image correction step;
An image pasting step of pasting the image of the poor visibility region cut out in the image clipping step on the poor visibility region of the image obtained in the image obtaining step;
Have.
As a result, it is possible to improve the visibility related to the region having poor visibility locally while maintaining the visibility of the entire image.

Furthermore, in addition to the image correction method described above, the image correction method of the present invention includes an image display step of displaying the corrected captured image corrected in the image correction step on a display device installed in the moving body. Have.
Thereby, for example, it becomes possible to provide a captured image with good visibility to a driver who drives the mobile body, and it is possible to improve the safety of driving the vehicle.

Further, the image correction method of the present invention is an image correction method executed by a computer to perform image correction,
An image acquisition step of acquiring the image captured by an imaging device installed in the mobile body so as to move with the mobile body;
A visibility recognition result acquisition step of acquiring, as a visibility recognition result, a region with poor visibility present in the image detected by the visibility determination device;
Based on the visibility recognition result acquired in the visibility recognition result acquisition step, the poor visibility region in the image acquired in the image acquisition step is specified, and the image of the poor visibility region An image correction step for correcting
Have
The three-dimensional object detection means for detecting a three-dimensional object existing in a peripheral region including the imaging direction imaged by the imaging device is mounted on the moving body, and the three-dimensional object detected by the three-dimensional object detection means Is present in the poorly visible region in the image, the image correction step corrects the image of the poorly visible region where the three-dimensional object is present.
Thereby, in order to ensure that the three-dimensional object can be visually recognized when the three-dimensional object overlaps an area where the visibility of the image is poor and it is difficult to visually recognize the three-dimensional object, the visibility is poor. It becomes possible to improve the visibility related to the region. In addition, by improving the visibility related only to the poorly visible area that is reducing the visibility of the three-dimensional object, the poorly visible area is improved only when necessary, thereby reducing the processing load. It becomes possible.

Furthermore, in addition to the above-described image correction method, the image correction method of the present invention provides the visual recognition that the solid object is present when the user exists in the poorly visible area in the image. It is possible to switch between an operation mode for correcting an image in an inferior area and an operation mode for correcting all the images in an invisible area.
Thereby, the user can set the aspect of the improvement process of the visibility himself.

  The present invention has an effect of making it possible to improve the visibility of a region having poor visibility locally while maintaining the visibility of the entire image. In addition, according to the image correction method of the present invention, it is possible to perform processing for improving the visibility of a captured image captured by the imaging device in real time, and for an already captured image, It is also possible to perform processing for improving visibility.

  Hereinafter, first to third embodiments of the present invention will be described with reference to the drawings. Below, a shadow appears in a part of the image, and the shadowed area (shadow area) is darker than the non-shadowed area (non-shadow area). A case where the visibility is deteriorated will be described as an example. In this specification, “visibility” represents not only the degree of whether or not a human can visually recognize (that is, visibility or difficulty in viewing) but also the degree of influence on the processing accuracy of the image processing apparatus. ing.

  First, the configuration of an image correction system using the image correction method according to the first to third embodiments of the present invention will be described with reference to FIG. FIG. 1 shows an example of the configuration of an image correction system using the image correction method according to the first to third embodiments of the present invention.

  An image correction system illustrated in FIG. 1 includes a camera (imaging device) 10, a shadow recognition device (visibility determination device) 11, a three-dimensional object sensor 15, a shadow correction device (image correction device) 20, a display monitor 30, and imaging. An image utilization device 40 is included.

  In FIG. 1, the camera 10 has a function of performing imaging in an arbitrary imaging direction. Note that the captured image captured by the camera 10 may be a moving image or a still image. Further, the installation mode of the camera 10 is arbitrary, and may be fixed at a predetermined position such as a surveillance camera, for example, so as to move together with a moving body such as a vehicle, a ship, an airplane, or a movable robot. It may be installed in the moving body. The camera 10 may be a camera that can be carried by a human.

  As shown in FIG. 1, the captured image captured by the camera 10 is supplied to the shadow recognition device 11 and the shadow correction device 20. FIG. 1 illustrates a configuration in which the captured image captured by the camera 10 is directly supplied to the shadow recognition device 11 and the shadow correction device 20, but the captured image captured by the camera 10 is temporarily recorded. It may be supplied to the shadow recognition device 11 and the shadow correction device 20 after being stored in a medium or subjected to image processing by another image processing device, or may be supplied to the shadow recognition device 11 and the shadow correction device through a network or the like. 20 may be supplied.

  In addition, the shadow recognition device 11 has a function of recognizing a shadow present in a captured image captured by the camera 10. Note that the present invention does not particularly limit the method for recognizing a shadow present in a captured image. That is, the shadow recognition device 11 recognizes an arbitrary shadow in order to recognize a shadow present in the captured image. Technology can be used. For example, the shadow recognition device 11 can output the coordinates of a pixel in which a shadow exists in the captured image (a pixel belonging to the shadow) as a shadow recognition result. The shadow recognition result by the shadow recognition device 11 is supplied to the shadow correction device 20.

  Further, the shadow correction device 20 has a function of correcting (improving) the image quality of the shadow region shown in the captured image supplied from the camera 10 based on the shadow recognition result supplied from the shadow recognition device 11. ing. The shadow correction device 20 can reduce the influence of the shadow in the captured image by correcting the image quality of the shadow region. The detailed configuration of the shadow correction device 20 will be described later with reference to FIG. The captured image (captured image after correction) whose shadow has been corrected by the shadow correction device 20 can be used for various purposes.

  For example, the corrected captured image generated by the shadow correction device 20 is sent to the display monitor 30 and displayed. In the corrected captured image, the image quality of the shadow area is improved, and the visibility of the image area that should be difficult to see due to the shadow is improved.

  Further, for example, the corrected captured image generated by the shadow correction device 20 is sent to the captured image utilization device 40 such as a parking assistance device or an object detection device. In the parking support device, for example, support for parking the vehicle at a desired parking target position is performed based on an image captured by the camera 10 mounted on the vehicle. However, instead of the captured image from the camera 10 including a shadow, By using the corrected captured image from which the influence of the above is removed, safer and more reliable parking assistance is performed. Also in the object detection device, an object (obstacle) existing in a peripheral region including the traveling direction of the moving body is detected based on the image captured by the camera 10 as in the parking assistance device. Therefore, in the object detection apparatus, the object detection can be performed more reliably by using the corrected captured image from which the influence of the shadow is removed instead of the captured image from the camera 10 including the shadow. .

  Further, the shadow correction device 20 refers to the three-dimensional object detection result supplied from the three-dimensional object sensor 15 for detecting the three-dimensional object existing in the peripheral area including the traveling direction of the moving object, and applies the shadow object in the captured image. Only in the case where a three-dimensional object exists, the shadow area where the three-dimensional object exists may be corrected. The three-dimensional object sensor 15 has a function of detecting a three-dimensional object using an arbitrary technique. For example, a technique such as a clearance sonar using ultrasonic waves or a millimeter wave radar using millimeter waves is used. It is possible. Also, the installation mode of the three-dimensional object sensor 15 is arbitrary. For example, the three-dimensional object sensor 15 may be installed on a moving body that moves together with the camera 10, and further set for the purpose of notifying the driver of the detection of an obstacle. It is also possible to use the obstacle sensor being used as the three-dimensional object sensor 15 according to the present invention.

  Also, in the shadow correction device 20, whether or not the shadow area where the three-dimensional object exists is corrected only when the three-dimensional object exists in the shadow area in the captured image, or in all cases, the shadow area is corrected. It is also possible to allow the user to selectively set various operation modes such as whether to perform the operation.

Here, the corrected captured image generated by the shadow correction device 20 will be specifically described. For example, it is assumed that the camera 10 of the image correction system illustrated in FIG. 1 is installed in a vehicle and captures a predetermined imaging direction around the vehicle. Then, it is assumed that the captured image captured by the camera 10 is displayed on the display monitor 30 installed in a place where the driver can visually recognize the captured image. In this case, a captured image as illustrated in FIG. 14 is captured by the camera 10, for example. FIG. 15 is a diagram schematically showing the contour of the object in the captured image of FIG.

According to the conventional technique, even when a shadow is captured in the captured image captured by the camera 10, the captured image including the shadow (the image illustrated in FIG. 14) is not performed without performing image correction processing for correcting the shadow region. Image) is displayed on the display monitor 30 as it is.

On the other hand, when the image correction method of the present invention is used, the shadow recognition process in the captured image captured by the camera 10 is performed by the shadow recognition device 11 and the shadow correction device 20 improves the visibility related to the shadow area. For example, instead of the captured image captured by the camera 10, a corrected captured image in which a shadow as illustrated in FIG. 16 is corrected (removed) is displayed on the display monitor 30 . Is displayed. Note that FIG. 17 schematically illustrates the contour of an object in the captured image after correction in FIG.

  14 is compared with the captured image after correction shown in FIG. 16 (or the schematic diagram of the captured image before correction shown in FIG. 15 and the schematic diagram of the corrected image shown in FIG. 17). As can be seen from the comparison with the above, the captured image before correction has poor visibility related to the shadow area, and it is not easy to visually recognize an object existing in the shadow area. And the visibility of the object existing in the shadow area becomes easy.

<First Embodiment>
Next, the configuration of a shadow correction apparatus that executes the image correction method according to the first embodiment of the present invention will be described with reference to FIG. FIG. 2 shows an example of the configuration of a shadow correction apparatus that executes the image correction method according to the first embodiment of the present invention. Note that the shadow correction device 20 illustrated in FIG. 2 is a configuration example of the shadow correction device 20 illustrated in FIG.

  The shadow correction apparatus 20 illustrated in FIG. 2 includes a shadow region feature amount calculation unit 21, a non-shadow region feature amount calculation unit 22, a shadow region image quality improvement unit 23, a shadow recognition result acquisition unit 26, an image acquisition unit 27, and a correction. An image output unit 28 is provided. Note that the functions of the shadow correction device 20 represented by the shadow area feature quantity calculation unit 21, the non-shadow area feature quantity calculation unit 22, and the shadow area image quality improvement unit 23 are performed by hardware and / or software executed by a computer. It is feasible.

  In FIG. 2, the shadow recognition result acquisition unit 26 has a function of acquiring the shadow recognition result of the captured image obtained by the shadow recognition device 11, and the image acquisition unit 27 captures the captured image captured by the camera 10. It has the function to acquire. It should be noted that, for example, synchronization control of information acquisition timing in each of the shadow recognition result acquisition unit 26 and the image acquisition unit 27 is performed, and the captured image acquired by the image acquisition unit 27 and the shadow recognition result of the captured image Response needs to be clearly identified. That is, the shadow correction device 20 is configured to correct a shadow in the captured image using a shadow recognition result obtained from a certain captured image.

  The shadow recognition results and captured images acquired by the shadow recognition result acquisition unit 26 and the image acquisition unit 27 are supplied to the shadow region feature amount calculation unit 21, the non-shadow region feature amount calculation unit 22, and the shadow region image quality improvement unit 23. Is done.

  The shadow area feature amount calculation unit 21 has a function of specifying a shadow area in the captured image with reference to the shadow recognition result and calculating a feature amount representing the characteristics of the specified shadow area. The shadow area feature value calculation unit 21 uses, for example, a pixel value of each pixel belonging to the shadow area (for example, a part or all of an arbitrary parameter in the color space used in each pixel is used as the pixel value). The statistical processing result obtained by performing the statistical processing can be used as the feature amount of the shadow region.

  In addition, when there are a plurality of shadow regions in the captured image, the shadow region feature amount calculation unit 21 may calculate the feature amounts of each of the plurality of shadow regions. The feature amount of the entire shadow area included may be calculated. The shadow region feature amount calculated by the shadow region feature amount calculation unit 21 is supplied to the shadow region image quality improvement unit 23. Note that the shadow area feature value calculation unit 21 can perform a shadow area feature value calculation process (see FIG. 4), which will be described later.

  Further, the non-shadow area feature quantity calculation unit 22 identifies an area other than the shadow area in the captured image (referred to as a non-shadow area in this specification) with reference to the shadow recognition result, and the identified non-shadow area It has a function of calculating a feature amount that represents the feature of. The non-shadow area feature quantity calculation unit 22 is, for example, similar to the shadow area feature quantity calculation unit 21, for example, the pixel value of each pixel belonging to the non-shadow area (the pixel value is, for example, the color space used in each pixel) The statistical processing result obtained by performing the statistical processing of a part or all of the arbitrary parameters in (2) can be used as the feature amount of the non-shadow region. The feature amount of the non-shadow region calculated by the non-shadow region feature amount calculation unit 22 is supplied to the shadow region image quality improvement unit 23. Note that the non-shadow region feature value calculation unit 22 can perform non-shadow region feature value calculation processing (see FIG. 5) described later.

  The shadow area image quality improvement unit 23 has a function of correcting an image area (pixel) in a captured image corresponding to the shadow area. The shadow area image quality improvement unit 23 changes the feature quantity of each pixel belonging to the shadow area so that the feature quantity of the shadow area is complemented with the feature quantity of the non-shadow area as a reference, for example. It is possible to improve the visibility related to the shadow area without making any changes. The captured image (corrected captured image) corrected by the shadow area image quality improvement unit 23 is supplied to the corrected image output unit 28. The shadow area image quality improvement unit 23 performs a shadow area image quality improvement process (see FIG. 6), a shadow area image quality improvement process (see FIG. 7) excluding the vicinity of the shadow boundary, and a shadow area near the shadow boundary. Image quality improvement processing (see FIG. 8) and pixel averaging processing (see FIG. 9) in the vicinity of the shadow boundary can be performed.

  The corrected image output unit 28 has a function of outputting the corrected captured image supplied from the shadow area image quality improving unit 23 to the outside (for example, the display monitor 30 or the captured image utilization device 40).

  Next, the outline of the image correction process according to the image correction method in the first embodiment of the present invention will be described with reference to FIG. FIG. 3 is a flowchart showing an outline of the image correction processing according to the image correction method in the first embodiment of the present invention. Note that the image correction processing illustrated in FIG. 3 is executed by the shadow correction device 20 illustrated in FIGS. 1 and 2.

In the image correction process illustrated in FIG. 3, first, the shadow area feature quantity calculation unit 21 performs a shadow area feature quantity calculation process based on the shadow area and the captured image specified from the shadow recognition result, A feature amount of the region is calculated (step S210). On the other hand, in the non-shadow area feature quantity calculation unit 22 , the non-shadow area feature quantity calculation process is performed based on the non-shadow area and the captured image specified from the shadow recognition result, and the non-shadow area feature quantity is calculated. (Step S220). Then, the shadow area image quality improvement unit 23 corrects the shadow area in the captured image based on the feature quantities of the shadow area and the non-shadow area calculated by the processes in steps S210 and S220 (step S230). ).

  Note that the feature quantities of the shadow area and the non-shadow area calculated by the processes in step S210 and step S220 are used as basic data for the shadow area image improvement process in step S230 to be performed next. In FIG. 3, the process in step S220 is illustrated to be performed after the process in step S210. However, the processes in step S210 and step S220 can be performed independently and are performed in parallel. May be.

  Next, details of the shadow region feature amount calculation processing in step S210 of FIG. 3 will be described with reference to FIG. FIG. 4 is a flowchart showing an example of a detailed operation of the shadow region feature amount calculation processing according to the image correction method of the first embodiment of the present invention.

  In the shadow area feature amount calculation processing illustrated in FIG. 4, first, the shadow area feature amount calculation unit 21 acquires a captured image (step S2101) and acquires the coordinates of the shadow area from the shadow recognition result (step S2101). Step S2102). At this time, for example, information specifying the coordinates of the pixels belonging to the shadow region as illustrated in FIG. 18 is acquired.

  Then, the shadow area feature amount calculation unit 21 selects one pixel (that is, a pixel belonging to the shadow area) in the captured image corresponding to the coordinates of the shadow area acquired in step S2102 (step S2103), and step It is determined whether or not the pixel value of the pixel selected in S2103 is equal to or less than a predetermined threshold (step S2104). An arbitrary value set in advance is used as the predetermined threshold. Note that as a pixel value handled in step S2104, it is possible to use a part or all of an arbitrary parameter in the color space used in the pixel.

  In addition, the determination process in step S2104 described above is a process performed to eliminate the influence of a pixel having an extremely high pixel value (for example, a pixel representing a white object) from the finally calculated feature value of the shadow region. It is. For example, when a white line exists in the shadow area, the pixel representing the white line has a high pixel value even though it belongs to the shadow area. It is desirable to calculate the quantity. Note that the feature amount of the shadow region may be calculated from the pixel values of all the pixels belonging to the shadow region without ignoring pixels having a pixel value larger than the predetermined threshold. Alternatively, an arbitrary pixel belonging to the shadow area may be selected, and the feature amount of the shadow area may be calculated from the pixel value of the selected pixel.

  The pixel whose pixel value is determined to be equal to or smaller than the predetermined threshold value in step S2104 is set as a shadow region feature amount calculation target (step S2105), and it is determined in step S2104 that the pixel value is not equal to or smaller than the predetermined threshold value. These pixels are excluded from the shadow region feature quantity calculation target.

  The processing in steps S2103 to S2105 is performed for each pixel belonging to the shadow area. In other words, when the determination of whether to select or exclude the selected pixel as a feature amount calculation target of the shadow region is completed, it is confirmed whether all the pixels in the captured image corresponding to the coordinates of the shadow region have been selected. (Step S2106). If an unselected pixel exists, the process returns to step S2103, and the processes of steps S2103 to S2105 are performed on the unselected pixel.

  On the other hand, when all the pixels belonging to the shadow region are selected, for example, the statistical processing of the pixel value of the pixel to be calculated for the feature amount in the shadow region set in step S2105 is performed to obtain the statistical processing result. Thus, the feature value of the shadow area is calculated (step SS2107). Note that any statistical processing algorithm can be used for the statistical processing performed in step S2107. For example, a statistical process for calculating an average of pixel values of pixels for which a shadow area feature value is to be calculated may be performed, and an average value of pixel values obtained as a result of the statistical process may be used as the feature quantity of the shadow area.

  Next, the details of the non-shadow area feature quantity calculation processing in step S220 of FIG. 3 will be described with reference to FIG. FIG. 5 is a flowchart showing an example of detailed operation of the feature amount calculation processing of the non-shadow region according to the image correction method in the first embodiment of the present invention.

  In the non-shadow region feature amount calculation process illustrated in FIG. 5, first, the non-shadow region feature amount calculation unit 22 acquires a captured image (step S2201). Further, the non-shadow area feature amount calculation unit 22 acquires a shadow recognition result, and acquires coordinates around the shadow area (coordinates belonging to the non-shadow area) (step S2202). Note that, for example, a non-shadow area within an arbitrary distance (for example, 5 to 10 pixels) from the boundary (shadow boundary) between the shadow area and the non-shadow area can be defined as the periphery of the shadow area. At this time, for example, an area around the shadow area as shown in FIG. 18 is set as the non-shadow area.

  The process in step S2202 is a process performed to calculate the feature amount of the non-shadow area in consideration of only the pixels around the shadow area. If no shadow is generated, the shadow area is expected to have a feature amount immediately around the shadow area. Therefore, it is desirable that the feature amount of the non-shadow region used for correcting the feature amount of the shadow region is calculated from the pixel values of the pixels around the shadow region. In the process of step S2202, not only the periphery of the shadow area but also the coordinates of other non-shadow areas (or all non-shadow areas) may be acquired. Further, only the coordinates of a part of the area around the shadow area may be acquired.

  Then, the non-shadow region feature value calculation unit 22 sets 1 in the captured image (that is, pixels belonging to the periphery of the shadow region) corresponding to the coordinates of the non-shadow region (the periphery of the shadow region) acquired in step S2202. Is selected (step S2203), and it is determined whether the pixel value of the pixel selected in step S2203 is equal to or greater than a predetermined threshold (step S2204). An arbitrary value set in advance is used as the predetermined threshold. For example, an average value of the pixel values in the shadow area can be used as the predetermined threshold value. Further, as the pixel value handled in step S2204, a part or all of an arbitrary parameter in the color space used in the pixel can be used, as in the above-described shadow area feature amount calculation processing. is there.

  The determination process in step S2204 is performed to eliminate the influence of pixels having extremely low pixel values (for example, pixels representing a black object) from the finally calculated feature amount of the non-shadow region. It is processing. For example, a pixel representing a black object existing in a non-shadow area has a low pixel value even though it belongs to the non-shadow area. Therefore, this pixel is ignored and the feature amount of the non-shadow area is ignored. It is desirable that the calculation is performed. Note that the feature amount of the non-shadow region may be calculated from the pixel values of all the pixels belonging to the non-shadow region without ignoring pixels having a pixel value smaller than the predetermined threshold. Alternatively, an arbitrary pixel belonging to the non-shadow area may be selected, and the feature amount of the non-shadow area may be calculated from the pixel value of the selected pixel.

  A pixel whose pixel value is determined to be greater than or equal to a predetermined threshold in step S2204 is set as a feature amount calculation target for a non-shadow region (step S2205), and it is determined in step S2204 that the pixel value is not equal to or greater than a predetermined threshold. The excluded pixels are excluded from the feature amount calculation target of the non-shadow region.

  The processing in steps S2203 to S2205 is performed for each pixel belonging to the periphery of the shadow area. That is, whether or not all the pixels in the captured image corresponding to the coordinates around the shadow area have been selected when the determination of whether or not to set the selected pixel as the feature amount calculation target of the non-shadow area is completed Is confirmed (step S2206). If there is an unselected pixel, the process returns to step S2203, and the processes from step S2203 to step S2205 are performed on the unselected pixel.

  On the other hand, when all the pixels belonging to the periphery of the shadow area are selected, for example, the statistical processing of the pixel value of the pixel for the feature amount calculation target of the non-shadow area set in step S2205 is performed, and the statistical processing result Thus, the feature amount of the non-shadow area is calculated (step SS2207). Note that any statistical processing algorithm can be used for the statistical processing performed in step S2207. For example, statistical processing may be performed to calculate the average of pixel values of pixels for which feature amounts are to be calculated in the non-shadow region, and the average value of pixel values obtained as a result of the statistical processing may be used as the feature amount of the non-shadow region.

  Next, details of the image quality improvement processing of the shadow area in step S230 of FIG. 3 will be described with reference to FIG. FIG. 6 is a flowchart showing an example of detailed operation of the image quality improvement processing of the shadow area according to the image correction method in the first embodiment of the present invention.

  In the non-shadow area feature amount calculation process illustrated in FIG. 6, the shadow area image quality improvement unit 23 performs the image quality improvement process for the shadow area excluding the vicinity of the shadow boundary (step S231). In addition, the shadow area image quality improvement unit 23 performs an image quality improvement process for the shadow area near the shadow boundary (step S232). Further, in order to eliminate the uncomfortable feeling at the time of visual recognition that occurs near the shadow boundary, Pixel averaging processing is performed (step S233).

  In the non-shadow region feature amount calculation process illustrated in FIG. 6, the process in step S231 is performed after the process in step S230. However, the entire shadow area is divided into a shadow area excluding the vicinity of the shadow boundary and a shadow area of the shadow boundary, and the image quality improvement processing for each area can be executed independently, and the processes in steps S231 and S232 are performed in parallel. May be performed.

  Further, the image quality improvement processing of the shadow area near the shadow boundary in step S232 is a process performed to eliminate the uncomfortable feeling at the time of visual recognition that occurs near the shadow boundary as described later. The image improvement process for the entire shadow region may be performed using one image improvement algorithm without dividing the shadow region except for the vicinity of the boundary and the shadow region of the shadow boundary. It is also possible to omit the averaging process for pixels near the shadow boundary in step S233.

  Here, the basic concept of the image quality improvement processing of the shadow area in step S230 will be described. In the image correction method according to the first embodiment of the present invention, the shadow region image is corrected (improved) based on the feature amounts of the shadow region and the non-shadow region.

  In the image quality improvement processing of the shadow region in step S230, processing for generating the pixel value of each pixel of the corrected captured image is performed by changing the pixel value of each pixel belonging to the shadow region of the captured image. That is, the pixel value of the pixel (pixel belonging to the shadow area) at the coordinate position (x, y) in the captured image before correction is expressed as Vin (x, y), and the coordinate position (x, y in the corrected image after correction). When the pixel value of the pixel y) is represented by Vout (x, y), in the image quality improvement processing of the shadow region in step S230, the pixel of each pixel of the captured image before correction as shown in the following equation (1) By calculating the value Vin (x, y) using an arbitrary function F, a pixel value Vout (x, y) of each pixel of the corrected captured image is generated.

  Vout (x, y) = F (Vin (x, y)) (1)

  Further, the function F of the equation (1) is determined by, for example, the feature amount Xm of the shadow region calculated in step S210 and the feature amount Ym of the non-shadow region calculated in step S220. A conversion expression such as Expression (2) can be used.

  Vout (x, y) = Vin (x, y) × (Ym / Xm) (2)

  For example, when the average value of the pixel value of the shadow area is set as the feature value Xm of the shadow area, and the average value of the pixel value of the non-shadow area is set as the feature value Ym of the non-shadow area, A value obtained by increasing the pixel value Vin (x, y) of each pixel of the captured image before correction to the average value level of the non-shadow area by the calculation using (2) is obtained for each pixel of the corrected captured image. The pixel value Vout (x, y) can be set.

  Hereinafter, conversion of the pixel value using the above equation (2) will be described with reference to FIGS. 19 and 20. FIG. 19 is a diagram schematically illustrating pixels in the vicinity of the shadow boundary of the captured image before correction according to the present invention. FIG. 20 is a graph showing pixel values before and after correction on the line segment AB in FIG.

  FIG. 19 illustrates pixels near the shadow boundary. In FIG. 19, a line segment (line segment A-B crossing the shadow boundary) between the position A in the shadow area and the position B in the non-shadow area is defined. At this time, as shown in FIG. 20, the pixel value of the pixel of the captured image before correction on the line segment AB is generally a high value (pixel value of the non-shadow region) for pixels belonging to the non-shadow region. For the pixels belonging to the shadow area, generally low values (values close to the feature quantity Xm of the shadow area).

  Here, the pixel value of the pixel belonging to the shadow area is converted by the above formula (2) to match the level of the feature quantity Xm of the shadow area with the level of the feature quantity Ym of the non-shadow area. It is possible to obtain pixel values belonging to the shadow area in the captured image after correction. Note that the visibility of the non-shadow region can be maintained by not changing the pixel value of the pixel in the non-shadow region.

  In Expression (2), the function F is defined as a function that does not depend on the coordinate position (x, y) in the captured image by using the statistical processing result (average value), but the function F is captured. The function F (x, y) depending on the coordinate position (x, y) in the image may be used. An example of the function F (x, y) is, for example, an expression (3) described later (however, in the expression (3), it depends only on one coordinate y), but a conversion expression using a more complicated function. Can also be defined.

  Next, the details of the image quality improvement processing of the shadow area excluding the vicinity of the shadow boundary in step S231 in FIG. 6 will be described with reference to FIG. FIG. 7 is a flowchart showing an example of detailed operation of the image quality improvement processing of the shadow area excluding the vicinity of the shadow boundary according to the image correction method in the first embodiment of the present invention.

  Note that, for example, the pixel values of the pixels belonging to the shadow area excluding the vicinity of the shadow boundary can be converted using the above-described equation (2). Here, the captured image of the imaging device installed in the vehicle is used. In consideration of this characteristic, a method using Equation (3), which will be described later, can reproduce the tendency of the image to become brighter from the bottom to the top in the corrected shadow region.

  In the image quality improvement processing of the shadow area excluding the vicinity of the shadow boundary illustrated in FIG. 7, first, the shadow area image quality improvement unit 23 acquires a captured image (step S2311), and also detects the vicinity of the shadow boundary from the shadow recognition result. The coordinates of the excluded shadow area are acquired (step S2312).

  Then, the shadow area image quality improvement unit 23 calculates pixels in the captured image corresponding to the coordinates of the shadow area excluding the vicinity of the shadow boundary acquired in step S2312 (that is, pixels belonging to the shadow area excluding the vicinity of the shadow boundary). One is selected (step S2313), and a parameter based on the coordinates of the pixel selected in step S2313 is set (step S2314). Then, the shadow area image quality improvement unit 23 calculates the pixel value of the selected pixel using the parameters set in step S2314 (step S2315), and calculates the pixel value of the pixel as the pixel value calculated in step S2315. (Step S2316).

  The processing in steps S2313 to S2316 is performed for each pixel belonging to the shadow area excluding the vicinity of the shadow boundary. That is, when the pixel value of the selected pixel is changed, it is confirmed whether or not all the pixels in the captured image corresponding to the coordinates of the shadow area excluding the vicinity of the shadow boundary have been selected (step S2317). If an unselected pixel exists, the process returns to step S2313, and the processes of step S2313 to step S2316 are performed on the unselected pixel. On the other hand, when all the pixels belonging to the shadow area excluding the vicinity of the shadow boundary are selected, the image quality improvement processing for the shadow area excluding the vicinity of the shadow boundary is completed.

  As described above, the parameters set in step S2314 take into consideration the characteristics of the captured image of the imaging device installed in the vehicle, and the image becomes brighter from the bottom to the top in the corrected shadow region. It is set to reproduce the trend. Specifically, in step S2314 and step S2315, the pixel value can be converted by a conversion equation such as the following equation (3).

Vout (x, y) = Vin (x, y) × (α− (H−y) × β) (3)
However, α and β are arbitrary positive values, and H is the height of the captured image.

  Hereinafter, conversion of the pixel value using the above equation (3) will be described with reference to FIGS. 21 and 22. FIG. 21 is a diagram showing the definition of coordinates for explaining the expression (3) used in the image correction method according to the first embodiment of the present invention. FIG. 22 is a graph for explaining parameters (weights) set depending on the y direction of FIG.

  As shown in FIG. 21, the upper left of the captured image is the origin (0, 0), the horizontal direction is the x axis, the vertical direction is the y axis, and the height of the captured image is expressed as H. At this time, the parameter (α− (H−y) × β) in the above equation (3) is a weight having a gradient (−β) in the y-axis direction as shown in FIG. The weight is set to the pixel value. The parameter (α− (H−y) × β) increases as the value of y increases (that is, toward the lower direction of the image). Therefore, Vout (x, y) obtained by integrating the parameters (α− (H−y) × β) is a value whose brightness is adjusted depending on the y direction. Note that the values of α and β vary depending on, for example, the characteristics of the camera 10 and the vehicle shape. Further, although the weight that depends only on the y direction is set in Expression (3), it is possible to set a weight that depends on the x direction or a weight that depends on both the x direction and the y direction.

  Next, the details of the image quality improvement processing of the shadow area near the shadow boundary in step S232 of FIG. 6 will be described with reference to FIG. FIG. 8 is a flowchart showing an example of detailed operation of the image quality improvement processing of the shadow area near the shadow boundary according to the image correction method in the first embodiment of the present invention.

  In the image quality improvement processing of the shadow area near the shadow boundary illustrated in FIG. 8, first, the shadow area image quality improvement unit 23 acquires a captured image (step S2321), and from the shadow recognition result, the shadow area near the shadow boundary. Are obtained (step S2322).

  Then, the shadow region image quality improvement unit 23 selects one pixel in the captured image corresponding to the coordinates of the shadow region near the shadow boundary acquired in step S2322 (that is, a pixel belonging to the shadow region near the shadow boundary). (Step S2323), a parameter based on the coordinates of the pixel selected in step S2323 is set (step S2324). Then, the shadow area image quality improvement unit 23 calculates the pixel value of the selected pixel using the parameters set in step S2324 (step S2325). However, the pixel value calculated in step S2325 is still only a pixel value change candidate at this point.

  Here, in step S2323 and step S2324, the pixel value is calculated by a conversion equation having a coordinate-dependent parameter such as equation (3). However, for example, the coordinate-dependent value as shown in equation (2) is used. The pixel value may be calculated using a conversion formula having no parameter.

  Here, the shadow area image quality improvement unit 23 checks whether or not the pixel value calculated in step S2325 is equal to or less than a maximum value (or a predetermined set value) that the pixel value can take (step S2326). Then, when the pixel value is equal to or less than the maximum value, the shadow area image quality improvement unit 23 changes the pixel value of the pixel to the pixel value calculated in step S2325 (step S2327). If the value exceeds the maximum value, the current pixel value is set as the changed pixel value so that the pixel value of the selected pixel is used as it is (step S2328).

  In the processing in step S2326 described above, the shadow area near the shadow boundary tends to have a higher pixel value than the other shadow areas, and the shadow area near the shadow boundary in the corrected captured image is emphasized in white (brighter). This is a process performed in consideration of the fact that there are many cases where it becomes too much. For example, the maximum value of the luminance value in hexadecimal notation is 256, but the pixel value (luminance value) calculated in step S2325 may be 256 or more. In consideration of such a case, in the image quality improvement processing of the shadow region near the shadow boundary illustrated in FIG. 8, if a value exceeding the maximum pixel value is calculated in step S2325, the calculation result The original pixel value is adopted without adopting. For example, in step S2326, the pixel value calculated in step S2325 is compared with a predetermined setting value (for example, 200), and for a pixel value exceeding the predetermined setting value, the predetermined setting value is uniformly set to that pixel. It can also be set as a value. The predetermined set value may be a value that changes depending on the coordinates.

  The processing in steps S2323 to S2328 is performed for each pixel belonging to the shadow region near the shadow boundary. That is, when the pixel value of the selected pixel is changed, it is confirmed whether or not all the pixels in the captured image corresponding to the coordinates of the shadow area near the shadow boundary have been selected (step S2339). If there is an unselected pixel, the process returns to step S2323, and the processes of step S2323 to step S2328 are performed on the unselected pixel. On the other hand, when all the pixels belonging to the shadow area near the shadow boundary are selected, the image quality improvement processing for the shadow area near the shadow boundary is completed.

  Next, the details of the averaging process for pixels near the shadow boundary in step S233 in FIG. 6 will be described with reference to FIG. FIG. 9 is a flowchart showing an example of detailed operation of the averaging process of pixels near the shadow boundary according to the image correction method in the first embodiment of the present invention.

  In the averaging process for pixels in the vicinity of the shadow boundary illustrated in FIG. 9, the shadow region image quality improvement unit 23 acquires a captured image (step S2331), and acquires the coordinates of the pixel at the shadow boundary from the shadow recognition result. (Step S2332).

  Next, the shadow area image quality improvement unit 23 calculates the feature amount of the non-shadow area near the shadow boundary based on the coordinates of the pixel at the shadow boundary acquired in step S2332 (step S2333), and at the same time, near the shadow boundary. The feature amount of the shadow area is calculated (step S2334). Note that the feature amount calculation processing of the non-shadow region near the shadow boundary in step S2333 may be performed simultaneously in, for example, the non-shadow region statistic calculation processing illustrated in FIG. In addition, in the process of calculating the feature value of the shadow area near the shadow boundary in step S2334, the pixel value (corrected pixel value) obtained by the image quality improvement process of the shadow area near the shadow boundary shown in FIG. It is desirable to calculate the feature amount of the shadow area near the shadow boundary.

  Then, the shadow area image quality improvement unit 23 uses the feature quantity of the non-shadow area near the shadow boundary obtained by the process of step S2333 and the feature quantity of the shadow area near the shadow boundary obtained by the process of step S2334. Then, a pixel value averaging process for averaging the pixel values of the pixels in the vicinity of the shadow boundary is performed (step S2335).

  The pixel value averaging process in step S2335 is based on the non-shadow area near the shadow boundary and the feature amount of the shadow area, and changes in discontinuous pixel values that may occur near the shadow boundary (the difference between the shadow area and the non-shadow area). This is a process for correcting the pixel values in the vicinity of the shadow boundary so that the difference in pixel values between them becomes smooth (continuous). By this pixel value averaging process, it becomes possible to blur the outline of the shadow that existed in the captured image, and it is possible to generate a corrected captured image that does not feel strange as if there was no shadow from the beginning. It becomes. Here, the pixel value of the pixel belonging to the shadow area near the shadow boundary is changed to smooth the change in the pixel value between the non-shadow area and the shadow area. You may change the pixel value of the pixel which belongs, and the pixel value of the pixel of both the non-shadow area | region and shadow area | regions near a shadow boundary.

  As described above, according to the first embodiment of the present invention, by performing conversion that improves the visibility of the pixels in the area of the area with poor visibility in the image, It is possible to improve the visibility related to the locally poor visibility region while maintaining the visibility of the entire image.

<Second Embodiment>
Next, the configuration of a shadow correction apparatus that executes the image correction method according to the second embodiment of the present invention will be described with reference to FIG. FIG. 10 shows an example of the configuration of a shadow correction apparatus that executes the image correction method according to the second embodiment of the present invention. Note that the shadow correction device 20 illustrated in FIG. 10 is a configuration example of the shadow correction device 20 illustrated in FIG.

  The shadow correction apparatus 20 illustrated in FIG. 10 includes a shadow recognition result acquisition unit 26, an image acquisition unit 27, a corrected image output unit 28, an image improvement unit 51, a shadow region image cutout unit 52, and a shadow region image pasting unit 53. have. Note that the functions of the shadow correction apparatus 20 represented by the image improvement unit 51, the shadow region image cutout unit 52, and the shadow region image pasting unit 53 can be realized by hardware and / or software executed by a computer. .

  The shadow recognition result acquisition unit 26, the image acquisition unit 27, and the corrected image output unit 28 in FIG. 10 have the same functions as the shadow recognition result acquisition unit 26, the image acquisition unit 27, and the corrected image output unit 28 illustrated in FIG. have.

  In addition, when there is a region with poor visibility in the captured image acquired by the image acquisition unit 27, the image improvement unit 51 is configured to improve the visibility related to the region with poor visibility. An image correction process is performed on the image. In the captured image in which the entire image is corrected by the image improving unit 51, the visibility related to the poorly visible area is improved, but the visibility may be adversely decreased for the highly visible area. is there. Note that the image improvement unit 51 performs image correction on the entire captured image using a known image correction technique (image improvement technique) such as a histogram smoothing process, for example. In addition, the image correction process for the entire captured image may be performed by an arbitrary process suggested in the first embodiment of the present invention described above.

  Further, the shadow area image cutout unit 52 has a function of identifying a shadow area in the captured image with reference to the shadow recognition result and cutting out only the image of the shadow area from the captured image corrected by the image improvement unit 51. Yes.

  Also, the shadow area image pasting unit 53 identifies a shadow area in the captured image with reference to the shadow recognition result, and the shadow area image clipping unit 52 extracts the captured image acquired by the image acquisition unit 27. A function of pasting the image of the shadow area.

  Next, an overview of image correction processing according to the image correction method in the second embodiment of the present invention will be described with reference to FIGS. 11 and 23. FIG. 11 is a flowchart showing an overview of image correction processing according to the image correction method of the second embodiment of the present invention. Note that the image correction processing illustrated in FIG. 11 is executed by the shadow correction device 20 illustrated in FIGS. 1 and 10.

  In the image correction process illustrated in FIG. 11, a captured image is acquired (step S2501), and the coordinates of the pixels in the shadow area are acquired from the shadow recognition result (step S2502). The image improvement unit 51 performs image correction processing on the entire captured image so that the visibility related to the image of the shadow region is improved (step S2503), and the shadow region image cutout unit 52 is corrected in step S2503. A process of cutting out an image of a shadow area from the captured image (captured image in which the entire image is corrected) is performed. Then, the shadow region image pasting unit 53 pastes the image of the shadow region cut out in step S2504 on the captured image acquired in step S2501 (step S2505), thereby improving the visibility relating to only the shadow region. The corrected captured image is generated. It should be noted that with respect to the corrected captured image, an averaging process for pixels in the vicinity of the shadow boundary between the shadow area and the non-shadow area (an averaging process for the pixels near the shadow boundary illustrated in FIG. 9) is performed. Also good.

  FIG. 23 is a conceptual diagram for explaining the outline of the image correction method according to the second embodiment of the present invention. According to the image correction process illustrated in FIG. 11, first, as illustrated in FIG. 23, in the image improvement unit 51, a captured image 82 in which the entire image is corrected is generated from the captured image 81. In the shadow area image cutout unit 52, only the shadow area image 83 is cut out from the captured image 82 in which the entire image is corrected. Then, the shadow region image pasting unit 53 pastes the shadow region image 83 cut out by the shadow region image cutout unit 52 to the shadow region of the captured image 81 and corrects only the shadow region. A captured image 84 is generated.

  As described above, according to the second embodiment of the present invention, as in the above-described first embodiment of the present invention, regarding a region with poor visibility in an image, the pixels in that region are assigned. On the other hand, by performing conversion that improves the visibility, it is possible to improve the visibility related to the region having poor visibility locally while maintaining the visibility of the entire image.

<Third Embodiment>
Next, the configuration of a shadow correction apparatus that executes the image correction method according to the third embodiment of the present invention will be described with reference to FIG. FIG. 12 shows an example of the configuration of a shadow correction apparatus that executes the image correction method according to the third embodiment of the present invention. Note that the shadow correction device 20 illustrated in FIG. 12 is a configuration example of the shadow correction device 20 illustrated in FIG.

The shadow correction apparatus 20 illustrated in FIG. 12 includes a shadow recognition result acquisition unit 26, an image acquisition unit 27, a corrected image output unit 28, an object detection / follow-up unit 61, an object presence region feature amount calculation unit 62, and an object presence region. A feature amount storage unit 63, a shadow intrusion determination unit 64, and a shadow correction unit 65 are provided. Note that the functions of the shadow correction device 20 represented by the object detection / follow-up unit 61, the object presence region feature amount calculation unit 62, the shadow intrusion determination unit 64, and the shadow correction unit 65 are executed by hardware and / or a computer. It can be realized by software.

  The shadow recognition result acquisition unit 26, the image acquisition unit 27, and the corrected image output unit 28 in FIG. 12 have the same functions as the shadow recognition result acquisition unit 26, the image acquisition unit 27, and the corrected image output unit 28 illustrated in FIG. have.

  Further, the object detection / following unit 61 detects an object (moving body) existing in the captured image, and follows the movement of the object (or the movement of the object in the captured image caused by the movement on the imaging side). It has a function to do. The object detection / follow-up unit 61 instructs the object presence region feature amount calculation unit 62 to appropriately calculate the feature amount of the image area (object presence region) around the object including the detected object. Further, the object detection / following unit 61 always follows the object, and supplies the existence area of the object to the shadow intrusion determining unit 64.

  In addition, the object presence region feature amount calculation unit 62 has a function of calculating the feature amount of the object presence region related to the object detected by the object detection / following unit 61 when receiving an instruction from the object detection / following unit 61. ing. The feature quantity of the object existence area calculated by the object existence area feature quantity calculation unit 62 is information corresponding to the feature quantity Ym of the non-shadow area in the first embodiment of the present invention, and is stored in the object existence area feature quantity. Stored in the unit 63.

  On the other hand, the shadow intrusion determination unit 64 has a function of determining whether or not an object has entered the shadow based on the presence area of the object supplied from the object detection / tracking unit 61 and the coordinates of the shadow region. Yes. In addition, it may be determined that the object has entered the shadow when a part of the object has entered the shadow, or it may be determined that the object has entered the shadow when the object has completely entered the shadow. . When the shadow intrusion determining unit 64 determines that the object has entered the shadow, the shadow intrusion determining unit 64 instructs the shadow correcting unit 65 to correct the shadow based on the object existing area related to the object.

When receiving a shadow correction instruction from the shadow intrusion determination unit 64, the shadow correction unit 65 performs image correction processing based on the image correction method according to the first embodiment of the present invention described above, and performs shadow correction in the captured image. Improve visibility of the area. At this time, as described above, by using the feature quantity of the object existence area stored in the object existence area feature quantity storage unit 63 as the feature quantity Ym of the non-shadow area, the object can be surely visually recognized. A corrected captured image is generated.

  Note that the shadow correction unit 65 may correct the shadow only when receiving an instruction from the shadow intrusion determining unit 64. If there is no instruction from the shadow intrusion determining unit 64, the shadow correcting unit 65 of the present invention described above. Shadow correction is performed using the image processing method according to the first or second embodiment, and when an instruction is received from the shadow intrusion determination unit 64, shadow correction specialized for improving the visibility of the object is performed. May be.

  Next, an overview of image correction processing according to the image correction method of the third embodiment of the present invention will be described with reference to FIGS. FIG. 13 is a flowchart showing an outline of image correction processing according to the image correction method of the third embodiment of the present invention. Note that the image correction processing illustrated in FIG. 13 is executed by the shadow correction device 20 illustrated in FIGS. 1 and 12.

  In the image correction process illustrated in FIG. 13, for example, a moving image is input, and a captured image and a shadow recognition result are always supplied (this process is not illustrated in FIG. 13). When the object detecting / following unit 61 detects an object in the captured image (step S2601), the object detecting / following unit 61 instructs the object existing region feature amount calculating unit 62 to calculate the feature amount of the object existing region (step S2602), and at the same time, Follow-up processing is started (step S2603).

  In the image correction process shown in FIG. 13, the operation is shown as if the feature amount of the object existing area is calculated before the object is tracked. The feature amount of the object presence area may be calculated and stored as appropriate during the period. In the third embodiment of the present invention, the feature amount of the object existence area is calculated at least once after the object is detected until the object enters the shadow, and the calculation result is the object existence. It needs to be stored in the region feature amount storage unit 63.

  When the tracking of the object is started, the shadow intrusion determination unit 64 continuously monitors the intrusion of the object into the shadow (step S2604). When it is detected that the object has entered the shadow, the feature amount of the object existing region when the object is present in the non-shadow region (that is, stored in the object existing region feature amount storage unit 63) The image correction processing according to the first embodiment of the present invention is performed using the feature amount of the existing object existence region) as the feature amount Ym of the non-shadow region (step S2605). This makes it possible to correct the shadow area so as to approach the reference based on the feature amount in a state where the object has good visibility, and specialized in improving the visibility of the object Shadow correction is performed.

  FIG. 24 is a conceptual diagram for explaining the outline of the image correction method according to the third embodiment of the present invention. According to the image correction process illustrated in FIG. 13, as illustrated in FIG. 24, the area around the object including the object 91 is the object existing area before the object 91 enters the shadow area 93. 92, the feature amount of the object presence area 92 is calculated and stored. If it is determined that the object has entered the shadow area 93 as a result of tracking the object, the shadow area is corrected using the feature quantity of the stored object existence area as the feature quantity Ym of the non-shadow area. Is called.

  When correcting the shadow region, a partial or entire feature amount of the shadow region may be calculated as the feature amount Xm of the shadow region, or within the object existence region 94 of the object 91 existing in the shadow region. The feature amount may be calculated. When the feature amount in the object existence region 94 of the object 91 existing in the shadow region is used as the feature amount Xm of the shadow region, image correction processing specialized for improving the visibility of the object is performed more. Become.

  As described above, according to the third embodiment of the present invention, as in the first and second embodiments of the present invention described above, an area with poor visibility in an image is displayed. By performing conversion that improves the visibility of the pixels, it is possible to improve the visibility related to the region having poor visibility locally while maintaining the visibility of the entire image. Further, according to the third embodiment of the present invention, in particular, when an object moving in the captured image enters an area with poor visibility, the visibility is specialized for improving the visibility of the object. It is possible to improve the visibility related to the poor region.

  In the above-described first to third embodiments of the present invention, description has been given mainly by taking an example of an image area that is locally dark due to shadow reflection as an area with poor visibility. In addition to the reflection, for example, there may be a case where an area with poor visibility occurs in the captured image because the light source is reflected and the light source and the surrounding image area are too bright. The present invention is also applied to correction of an image having a region with poor visibility caused by a cause other than such a shadow. When the cause of the deterioration in visibility is not a shadow but a light source, in the first to third embodiments of the present invention described above, the shadow area is read as an excessively bright area, It suffices to perform the darkening operation so as to match the area.

  In addition, in the present specification, the techniques according to the first to third embodiments of the present invention are described independently. However, the technique according to the first to third embodiments of the present invention is arbitrarily described. In combination, the present invention can be realized.

According to the present invention, there is also provided an image correction apparatus for correcting an image,
Image acquisition means for acquiring the image captured by the imaging device;
Visibility recognition result acquisition means for acquiring, as a visibility recognition result, a region with poor visibility present in the image detected by the visibility determination device;
Based on the visibility recognition result acquired by the visibility recognition result acquisition unit, the poor visibility region in the image acquired by the image acquisition unit is specified, and the image of the poor visibility region Image correction means for correcting
An image correction apparatus is also provided.

  The present invention also provides a program for causing a computer to execute the image correction method of the present invention.

  The present invention has the effect of making it possible to improve the visibility of a region having poor visibility locally while maintaining the visibility of the entire image, and to improve the image quality of the image. The present invention can be applied to image correction technology, vehicle driving support technology for providing a driver with a captured image captured by an imaging device installed in a vehicle, and the like.

It is a figure which shows an example of a structure of the image correction system using the image correction method in the 1st-3rd embodiment of this invention. It is a figure which shows an example of a structure of the shadow correction apparatus which performs the image correction method in the 1st Embodiment of this invention. It is a flowchart which shows the outline | summary of the image correction process which concerns on the image correction method in the 1st Embodiment of this invention. It is a flowchart which shows an example of the detailed operation | movement of the feature-value calculation process of the shadow area which concerns on the image correction method in the 1st Embodiment of this invention. It is a flowchart which shows an example of the detailed operation | movement of the feature-value calculation process of the non-shadow area | region which concerns on the image correction method in the 1st Embodiment of this invention. It is a flowchart which shows an example of the detailed operation | movement of the image quality improvement process of the shadow area which concerns on the image correction method in the 1st Embodiment of this invention. It is a flowchart which shows an example of the detailed operation | movement of the image quality improvement process of the shadow area | region except the shadow boundary vicinity which concerns on the image correction method in the 1st Embodiment of this invention. It is a flowchart which shows an example of a detailed operation | movement of the image quality improvement process of the shadow area | region near the shadow boundary which concerns on the image correction method in the 1st Embodiment of this invention. It is a flowchart which shows an example of the detailed operation | movement of the averaging process of the pixel near the shadow boundary which concerns on the image correction method in the 1st Embodiment of this invention. It is a figure which shows an example of a structure of the shadow correction apparatus which performs the image correction method in the 2nd Embodiment of this invention. It is a flowchart which shows the outline | summary of the image correction process which concerns on the image correction method in the 2nd Embodiment of this invention. It is a figure which shows an example of a structure of the shadow correction apparatus which performs the image correction method in the 3rd Embodiment of this invention. It is a flowchart which shows the outline | summary of the image correction process which concerns on the image correction method in the 3rd Embodiment of this invention. The figure which shows an example of the captured image before the correction | amendment input into the image correction system shown in FIG. The figure which represents typically the outline of the object in the captured image shown in FIG. The figure which shows an example of the captured image after the correction | amendment output from the image correction system shown in FIG. The figure which represents typically the outline of the object in the captured image shown in FIG. It is a figure which shows an example of the shadow area | region and non-shadow area | region which concern on the image correction method in the 1st Embodiment of this invention. It is the figure which represented typically the pixel of the shadow boundary vicinity of the captured image before correction | amendment which concerns on this invention. It is a graph which shows the value of the pixel value before correction | amendment on the line segment AB of FIG. 19, and after correction | amendment. It is a figure which shows the definition of the coordinate for demonstrating Formula (3) used with the image correction method in the 1st Embodiment of this invention. It is a graph for demonstrating the parameter (weight) set depending on the y direction of FIG. It is a conceptual diagram for demonstrating the outline | summary of the image correction method in the 2nd Embodiment of this invention. It is a conceptual diagram for demonstrating the outline | summary of the image correction method in the 3rd Embodiment of this invention.

Explanation of symbols

10 Camera (imaging device)
11 Shadow recognition device (visibility judgment device)
15 Solid object sensor 20 Shadow correction device (image correction device)
DESCRIPTION OF SYMBOLS 21 Shadow area feature-value calculation part 22 Non-shadow area | region feature-value calculation part 23 Shadow area image quality improvement part 26 Shadow recognition result acquisition part 27 Image acquisition part 28 Correction | amendment image output part 30 Display monitor 40 Captured image utilization apparatus 51 Image improvement part 52 Shadow Area image cutout section 53 Shadow area image pasting section 61 Object detection / follow-up section 62 Object presence area feature quantity calculation section 63 Object existence area feature quantity storage section 64 Shadow intrusion determination section 65 Shadow correction section 81 Captured image 82 Whole image is corrected Corrected captured image 83 Image of corrected shadow region 84 Corrected captured image 91 Object 92, 94 Object existing region 93 Shadow region

Claims (15)

An image correction method performed by a computer to perform image correction,
An image acquisition step of acquiring the image captured by the imaging device;
A visibility recognition result acquisition step of acquiring, as a visibility recognition result, a region with poor visibility present in the image detected by the visibility determination device;
Based on the visibility recognition result acquired in the visibility recognition result acquisition step, an area specifying step for specifying the poor visibility area in the image acquired in the image acquisition step ;
A first feature amount calculating step of calculating a feature amount of the region with poor visibility;
A second feature amount calculating step of calculating a feature amount of a region other than the region with poor visibility;
An image correction step of correcting an image of the region with poor visibility based on the feature amount calculated in the first and second feature amount calculation steps ;
Have
An object detection step of detecting an object moving in the plurality of images that are continuously imaged in time series;
When the object detected in the object detection step exists in a region other than the poor visibility region as a feature amount of the region other than the poor visibility region, an area around the object including the object is included. A feature amount storing step of calculating a feature amount and storing a feature amount of a region around the object including the calculated object;
An object tracking step of following the movement of the object in the plurality of images continuously imaged in time series and determining whether or not the object has entered the poor visibility region;
Have
When it is determined that the object has entered the region with poor visibility, in the image correction step, the feature amount of the region around the object including the object stored in the feature amount storage step is visually recognized. An image correction method for correcting an image in a region with poor visibility by setting the feature amount in a region other than the region with poor properties .   In the image correction step, a pixel value changing step of newly calculating a pixel value of a pixel belonging to the poor visibility region and changing a pixel value of the pixel belonging to the poor visibility region to the calculated pixel value The image correction method according to claim 1, further comprising:   The image correction method according to claim 2, wherein in the pixel value changing step, the pixel value is changed based on a coordinate position of a pixel belonging to the poor visibility region.   In the pixel value changing step, when the calculated pixel value exceeds a predetermined threshold value, the pixel value of the pixel belonging to the poor visibility region is changed to the original pixel value before the change or the predetermined pixel value. The image correction method according to claim 2 or 3, wherein the image correction method is set.   5. The pixel value correction step for correcting the pixel value so that a change in the pixel value is smooth at a boundary between the poor visibility region and a region other than the poor visibility region. The image correction method described in 1. In the first and second characteristic quantity calculating step performs statistical processing of the pixel value of each pixel, the image correction method according to one of claims 1 to 5 for the statistical processing results to the feature amount. The image correction method according to claim 6 , wherein, in the first and second feature amount calculating steps, an average value of pixel values of each pixel is calculated as the statistical processing result. In the image correction step, pixel values of pixels belonging to the poor visibility region are calculated so that the feature amount of the poor visibility region matches the feature amount of a region other than the poor visibility region. an image correction method according to any one of claims 1 7 for changing the pixel values of the pixels belonging to the poor areas of the visibility on the calculated pixel value. In the first feature quantity calculating step, according to the time of calculating the feature value of the visibility poor region, any one of claims 1 to 8, wherein the pixel value is ignore larger pixels than a predetermined threshold value Image correction method. In the second feature quantity calculation step, when calculating the feature amount of a region other than the visibility poor region, any one of claims 1 to 9 in which the pixel value is ignore smaller pixels than a predetermined threshold value The image correction method described in 1. Examples region other than visibility poor region, the image correction method according to one of claims 1 to 10 for setting the area around the poor the visibility region. The image correction step includes
An overall image correction step for correcting the entire image acquired in the image acquisition step so as to improve the visibility of the image in the poor visibility region;
An image cutout step of cutting out an image of the region with poor visibility from the image in which the entire image is corrected in the whole image correction step;
An image pasting step of pasting the image of the poor visibility region cut out in the image clipping step on the poor visibility region of the image obtained in the image obtaining step;
The image correction method according to claim 1. The image correction method according to claim 1, further comprising an image display step of displaying the corrected captured image corrected in the image correction step on a display device installed in the moving body. An image correction method performed by a computer to perform image correction,
An image acquisition step of acquiring the image captured by an imaging device installed in the mobile body so as to move with the mobile body ;
A visibility recognition result acquisition step of acquiring, as a visibility recognition result, a region with poor visibility present in the image detected by the visibility determination device;
Based on the visibility recognition result acquired in the visibility recognition result acquisition step, the poor visibility region in the image acquired in the image acquisition step is specified, and the image of the poor visibility region An image correction step for correcting
Have
The three-dimensional object detection means for detecting a three-dimensional object existing in a peripheral region including the imaging direction imaged by the imaging device is mounted on the moving body, and the three-dimensional object detected by the three-dimensional object detection means Is present in the poorly visible region in the image, the image correcting method includes correcting the image of the poorly visible region where the three-dimensional object is present in the image correcting step . An operation mode in which the user corrects an image of the poorly visible area where the solid object exists when the solid object exists in the poorly visible area in the image, and all the visibility The image correction method according to claim 14 , wherein an operation mode for correcting an image in a poor region can be switched.