JP4674179B2 - Shadow recognition method and shadow boundary extraction method - Google Patents

Description

  The present invention relates to a shadow recognition method for recognizing a shadow reflected in an image and a shadow boundary extraction method for extracting a boundary of a shadow reflected in an image (a boundary between a shadow region and a non-shadow region).

  Conventionally, for example, techniques disclosed in the following Non-Patent Documents 1 to 3 are known as a shadow recognition method for recognizing a shadow reflected in an image captured by an imaging device (camera).

In the technique disclosed in Non-Patent Document 1, a region with a small amount of irradiated light is detected as a shadow region in the distribution of light irradiated on the scene. In the technique disclosed in Non-Patent Document 2, shadow areas are detected by utilizing the fact that only the brightness is different and the color components are not significantly changed between the shadow area and the non-shadow area in the image. Yes. In the technique disclosed in Non-Patent Document 3, the shadow area detected together with the moving object is removed from the detection result of the moving object in the image.
“Automatic Image Shadow Identification using LPF in Homomorphic Processing System”, Hamideh Etemadnia and Prof. Mohammad Reza Alsharif, Proc. VIIth Digital Image Computing, 10-12, Dec. 2003 “Cast shadow segmentation using invariant color features”, E. Salvador et al., Computer Vision and Image Understanding 95, 238-259, 2004 “Improving Shadow Suppression in Moving Object Detection with HSV Color Information”, Rita Cucchiara et al., IEEE Intelligent Transportation Systems Conference Proceedings, pp.334-339, 2001

  However, with the technology disclosed in Non-Patent Document 1, shadow recognition in a simple image in which only a single object exists in the image is possible, but a complex image (a large number of objects in an actual environment) It is difficult to perform shadow recognition on images that exist.

  Further, regarding the technique disclosed in Non-Patent Document 2, when the influence of environmental light such as outdoors is large, a difference occurs in the color components in the shadow area and the non-shadow area in the image. Application of the technique disclosed in 2 is difficult.

  In the technique disclosed in Non-Patent Document 3, it is necessary to first detect the moving object using a background subtraction method using a background image. Therefore, for non-patented images taken with a camera installed on a moving body such as a car or robot whose background changes (that is, when the background of a continuous image in time series changes) It is difficult to apply the technique disclosed in Document 3.

  In order to solve the above-described problem, the present invention solves the above-described problem by using a shadow recognition method and a shadow boundary extraction method for recognizing a shadow in an image from a single image even when the background in the image is complicated. The purpose is to provide.

To achieve the above object, a shadow recognition method of the present invention is a shadow recognition method executed by a computer to recognize a shadow in an image in which a shadow of an arbitrary object is captured by an imaging device. ,
An image acquisition step of acquiring the image captured by the imaging device;
An area extraction step for extracting an area in which the shadow is assumed to be present in the image using an arbitrary shadow area extraction technique ;
A boundary extraction step of extracting a boundary between an area in which the shadow is assumed to be present in the image and an area in which the shadow is not assumed to be present using a predetermined shadow boundary extraction technique ;
Said region extraction region extraction result of the region extracted in step, and on the basis of the boundary extraction result of the relationship of the boundary extracted by the boundary extracting step, to calculate a decision value for each pixel forming the image determining A value calculation step;
A shadow recognition step of determining whether each of the pixels forming the image belongs to a shadow based on the determination value calculated in the determination value calculation step ;
Have.
In order to solve the above problem, even when the background in an image is complicated, it is possible to recognize a shadow in the image from one image.

Furthermore, the shadow recognition method of the present invention, in addition to the above-described shadow recognition method, in the shadow recognition step,
Based on the determination value, a pixel belonging to the region extracted in the region extraction step is determined as a pixel belonging to a shadow , while a pixel that does not belong to either the region or the boundary does not belong to a shadow. Determining a pixel;
Based on the determination value, if the pixels do not belong to the region extracted in the region extraction step and belong to the boundary and are adjacent to each other, the adjacent pixels are aggregated. Set setting step to be considered as
Based on the judgment value, any pixel belonging to the aggregate set by the set setting step, when adjacent to the pixel belonging to the area extracting the region extracted in step to the region Determining all pixels belonging to the aggregate together with pixels belonging to the pixel belonging to a shadow ;
On the basis of the decision value, all pixels belonging to the aggregate set by the set setting step, when not adjacent to the pixel belonging to the area extracting the region extracted in step the assembly Determining all pixels belonging to the pixel not belonging to the shadow ,
Have.
Thus, for example, a region obtained by the region extraction processing, it is possible to properly integrate the obtained boundary by the boundary extraction processing.

In order to achieve the above object, the shadow boundary extraction method of the present invention is a non-shadow that is different from the shadow area in the image and the shadow area in the image in which the shadow of an arbitrary object is captured by the imaging device. A shadow boundary extraction method executed by a computer to extract a shadow boundary with a region,
An image acquisition step of acquiring the image captured by the imaging device;
An edge extraction step of extracting an edge of the object to be imaged in the image;
An edge pixel specifying step for specifying an edge pixel belonging to the edge extracted in the edge extraction step;
A pattern calculation step of calculating a pattern indicating a relationship between pixel values of the edge pixel and a plurality of pixels existing in the vicinity of the edge pixel;
A shadow boundary pattern reference step for referring to a shadow boundary pattern in which a relationship between pixel values of shadow boundary pixels belonging to the shadow boundary and a plurality of pixels existing in the vicinity of the shadow boundary pixel is set;
A pattern matching step for matching the shadow boundary pattern referred to in the shadow boundary pattern reference step with the pattern calculated in the pattern calculation step;
A pattern matching shadow boundary determining step for determining that the edge pixel belonging to the pattern calculated in the pattern calculating step matched with the shadow boundary pattern in the pattern matching step is a pixel existing in the shadow boundary;
Have.
As a result, even when the background in the image is complicated, it is possible to recognize a shadow in the image from one image. It is also possible to realize shadow recognition processing with a high processing speed.

Furthermore, in the shadow boundary extraction method of the present invention, in addition to the above-described shadow boundary extraction method, the shadow boundary pattern includes an arbitrary shadow boundary pixel and a plurality of pixels existing in the vicinity of the arbitrary shadow boundary pixel. It is generated from the relationship between pixel values.
As a result, the shadow boundary pattern can be easily created by learning.

Furthermore, in the shadow boundary extraction method of the present invention, in addition to the above-described shadow boundary extraction method, the pixel values are RGB color system color component values.
This makes it possible to perform a shadow boundary extraction process based on a color image.

In order to achieve the above object, the shadow boundary extraction method of the present invention is a non-shadow that is different from the shadow area in the image and the shadow area in the image in which the shadow of an arbitrary object is captured by the imaging device. A shadow boundary extraction method executed by a computer to extract a shadow boundary with a region,
An image acquisition step of acquiring the image captured by the imaging device;
An edge extraction step of extracting an edge of the object to be imaged in the image;
A feature amount calculating step of performing binarization processing on the edge extracted in the edge extraction step, and calculating respective feature amounts of two regions separated by the binarized edge ;
A feature amount relationship determination step for determining whether or not the relationship between the feature amounts of the two regions matches the relationship between the feature amounts of the shadow region and the non-shadow region separated by the shadow boundary;
In the feature amount relationship determination step, when the relationship between the feature amounts of the two regions matches the relationship of the feature amounts of the shadow region and the non-shadow region separated by the shadow boundary, the edge is converted into the shadow region. A feature amount matching shadow boundary determination step for determining a boundary,
Have.
As a result, even when the background in the image is complicated, it is possible to recognize a shadow in the image from one image.

Furthermore, in addition to the above-described shadow boundary extraction method, the shadow boundary extraction method of the present invention irradiates the shadow region with the relationship between the feature amounts of the shadow region and the non-shadow region separated by the shadow boundary. It is determined based on the difference between the spectral characteristics of light and the spectral characteristics of light irradiated on the non-shadow area.
As a result, it is possible to realize shadow boundary extraction processing in consideration of the environmental influence during imaging.

Furthermore, in the shadow boundary extraction method of the present invention, in addition to the above-described shadow boundary extraction method, an edge gradient direction related to the pixel value of the edge pixel belonging to the binarized edge is calculated in the feature amount calculating step. Then, the calculation result relating to the pixel value of the pixel on the straight line substantially parallel to the edge gradient direction passing through the edge pixel is set as the feature amount.
This makes it possible to reliably calculate the feature amounts of the two regions that are separated from the edge.

Further, in the shadow boundary extraction method of the present invention, in addition to the above-described shadow boundary extraction method, the feature amount is a statistic of pixel values of some or all pixels belonging to each of the two regions.
Accordingly, it is possible to easily and quickly calculate the feature amounts of the two regions separated from each other by statistical processing.

Furthermore, in the shadow boundary extraction method of the present invention, in addition to the above-described shadow boundary extraction method, the pixel values are RGB color system color component values.
This makes it possible to perform a shadow boundary extraction process based on a color image.

In order to achieve the above object, the shadow boundary extraction method of the present invention is a non-shadow that is different from the shadow area in the image and the shadow area in the image in which the shadow of an arbitrary object is captured by the imaging device. A shadow boundary extraction method executed by a computer to extract a shadow boundary with a region,
An image acquisition step of acquiring the image captured by the imaging device;
An edge extraction step of extracting an edge of the object to be imaged in the image;
An edge pixel specifying step for specifying an edge pixel belonging to the edge extracted in the edge extraction step;
A pattern calculation step of calculating a pattern indicating a relationship between pixel values of the edge pixel and a plurality of pixels existing in the vicinity of the edge pixel;
A shadow boundary pattern reference step for referring to a shadow boundary pattern in which a relationship between pixel values of a shadow boundary pixel belonging to the shadow boundary and a plurality of pixels existing in the vicinity of the shadow boundary pixel is determined;
A pattern matching step for matching the shadow boundary pattern referred to in the shadow boundary pattern reference step with the pattern calculated in the pattern calculation step;
In the pattern matching step, a pattern matching shadow region determining step that identifies the edge pixel belonging to the pattern calculated in the pattern calculating step that matches the shadow boundary pattern;
An edge gradient direction related to a pixel value of the edge pixel specified in the pattern matching shadow region determination step is calculated, and a calculation result relating to the pixel value of a pixel on a straight line that is substantially parallel to the edge gradient direction passing through the edge pixel A feature amount calculating step for calculating each feature amount of two regions separated by the edge,
A feature amount relationship determination step for determining whether or not the relationship between the feature amounts of the two regions matches the relationship between the feature amounts of the shadow region and the non-shadow region separated by the shadow boundary;
In the feature amount relationship determination step, when the relationship between the feature amounts of the two regions matches the relationship of the feature amounts of the shadow region and the non-shadow region separated by the shadow boundary, the edge is converted into the shadow region. A feature amount matching shadow boundary determination step for determining a boundary,
Have.
In order to solve the above problem, even when the background in an image is complicated, it is possible to recognize a shadow in the image from one image.

  The present invention has an effect that, even when the background in an image is complicated, it is possible to extract a shadow boundary (a boundary between a shadow region and a non-shadow region) reflected in the image from one image. It has the effect that shadow recognition with high accuracy can be performed.

  Hereinafter, first to third embodiments of the present invention will be described with reference to the drawings.

  First, terms used in this specification will be defined with reference to FIG. FIG. 23 is a diagram illustrating an example of an object to be imaged and a captured image in the shadow recognition method and the shadow boundary extraction method according to the present invention.

  As shown in FIG. 23, when an object exists in space, a part of the light emitted from the light source (for example, the sun) is blocked by the object, and the shadow of the object is projected on the projection surface such as the ground. Projected. The shadow of an object is not directly exposed to light from a light source, but light (environmental light) reflected or scattered by, for example, atmospheric particles or other light reflectors (such as walls) is indirectly projected. The

  In this specification, the area where the shadow of the object exists is a shadow area, the area different from the shadow area (the area where the shadow of the object does not exist) is the non-shadow area, and the boundary between the shadow area and the non-shadow area (the shadow area Edge) is called a shadow boundary. In addition, shadow recognition (shadow recognition) according to the present invention refers to whether or not there is a shadow in a captured image, or whether a non-imaged object represented by a certain region (or pixel itself) in a captured image is a shadow. It represents recognizing whether or not.

  Next, the configuration of the image correction system common to the first to third embodiments of the present invention will be described with reference to FIG. FIG. 1 shows an example of the configuration of an image correction system common to the first to third embodiments of the present invention.

  The image correction system illustrated in FIG. 1 includes a camera (imaging apparatus) 10, a shadow recognition apparatus 20, a shadow correction apparatus (image correction apparatus) 30, a display monitor 40, and a captured image utilization apparatus 50.

  In FIG. 1, the camera 10 has a function of performing imaging in an arbitrary imaging direction. Note that the captured image captured by the camera 10 may be a moving image or a still image. Further, the installation mode of the camera 10 is arbitrary, and may be fixed at a predetermined position such as a surveillance camera, for example, so as to move together with a moving body such as a vehicle, a ship, an airplane, or a movable robot. It may be installed in the moving body. The camera 10 may be a camera that can be carried by a human.

  As shown in FIG. 1, the captured image captured by the camera 10 is supplied to the shadow recognition device 20 and the shadow correction device 30. 1 illustrates a configuration in which a captured image captured by the camera 10 is directly supplied to the shadow recognition device 20 and the shadow correction device 30, but the captured image captured by the camera 10 is temporarily recorded. It may be supplied to the shadow recognition device 20 and the shadow correction device 30 after being stored in a medium or subjected to image processing by another image processing device, or may be supplied to the shadow recognition device 20 and the shadow correction device through a network or the like. 30 may be supplied.

  In addition, the shadow recognition device 20 has a function of recognizing a shadow present in a captured image captured by the camera 10. The detailed configuration of the shadow recognition device 20 will be described later with reference to FIG. The shadow recognition result by the shadow recognition device 20 is supplied to the shadow correction device 30.

  Further, the shadow correction device 30 has a function of correcting (improving) the image quality of the shadow reflected in the captured image supplied from the camera 10 based on the shadow recognition result supplied from the shadow recognition device 20. Yes. The shadow correction device 30 can reduce the influence of the shadow in the captured image by correcting the shadow. Note that the present invention does not particularly limit the method of correcting a shadow existing in a captured image recognized by the shadow recognition device 20, that is, the shadow correction device 30 corrects a shadow existing in the captured image. In order to do this, any shadow correction technique can be used. The captured image (captured image after correction) whose shadow has been corrected by the shadow correction device 30 can be used for various purposes.

  For example, the corrected captured image generated by the shadow correction device 30 is sent to the display monitor 40 and displayed. The corrected captured image has improved the visibility deteriorated by the shadow, and the visibility of the image region that should originally be difficult to see by the shadow is improved.

  Further, for example, the corrected captured image generated by the shadow correction device 30 is sent to a captured image utilization device 50 such as a parking assistance device or an object detection device. In the parking support device, for example, support for parking the vehicle at a desired parking target position is performed based on an image captured by the camera 10 mounted on the vehicle. However, instead of the captured image from the camera 10 including a shadow, By using the corrected captured image from which the influence of the above is removed, safer and more reliable parking assistance is performed. Also in the object detection device, an object (obstacle) existing in the traveling direction of the moving body is detected based on the image captured by the camera 10 as in the parking assistance device. Therefore, in the object detection apparatus, the object detection can be performed more reliably by using the corrected captured image from which the influence of the shadow is removed instead of the captured image from the camera 10 including the shadow. .

  Next, the configuration of the shadow recognition apparatus common to the first to third embodiments of the present invention will be described with reference to FIG. FIG. 2 shows an example of the configuration of a shadow recognition apparatus common to the first to third embodiments of the present invention. Note that the shadow recognition device 20 illustrated in FIG. 2 is a configuration example of the shadow recognition device 20 illustrated in FIG.

  The shadow recognition device 20 illustrated in FIG. 2 includes a color image acquisition unit 21, a color space decomposition unit 22, an image storage unit 23, a shadow boundary extraction unit 24, a shadow region extraction unit 25, and a shadow boundary / shadow region integration unit 26. , A shadow recognition result output unit 27 is provided. Note that the function of the shadow recognition device 20 shown in the block diagram in FIG. 2 can be realized by hardware and / or software executed by a computer.

  In FIG. 2, the color image acquisition unit 21 has a function of acquiring a color captured image (color image) captured by the camera 10. The color image acquired by the color image acquisition unit 21 is supplied to the color space separation unit 22.

  The color space separation unit 22 has a function of separating the color image supplied from the color image acquisition unit 21 into each color component image in a predetermined color space. For example, when the RGB color system is used as the color space, the color space decomposition unit 22 decomposes the color image into three color component images of an R image, a G image, and a B image. The color component images generated by the color space separation unit 22 (in the case of the RGB color system, three images of R image, G image, and B image) are stored in the image storage unit 23.

  In the following, a case where the RGB color system is mainly used as a color space will be described as an example. However, in the present invention, any color space can be used. For example, an XYZ color system that can be converted to the RGB color system may be used. In addition, an HSV color system or HSI color system V image or I image (an image representing lightness) may be used. In this case, the present invention can be applied to a grayscale image. It becomes.

  The image storage unit 23 has a function of temporarily storing the color component image generated by the color space decomposition unit 22. The color component image stored in the image storage unit 23 is read by the shadow boundary extraction unit 24 or the shadow region extraction unit 25 as appropriate.

  The shadow boundary extraction unit 24 has a function of extracting a shadow boundary (boundary between a shadow region and a non-shadow region) existing in the image based on the color component image read from the image storage unit 23. ing. The shadow boundary extraction result extracted by the shadow boundary extraction unit 24 is supplied to the shadow boundary / shadow region integration unit 26. Note that the shadow boundary extraction unit 24, for example, as described later, determines the shadow boundary by pattern matching (configuration illustrated in FIG. 3), and statistics of color component values based on the spectral characteristics of the image. A configuration for determining a shadow boundary according to a condition that an amount (for example, an average value) satisfies (configuration shown in FIG. 5), and a configuration that combines the above two configurations (configuration shown in FIG. 7) Etc.

  Further, the shadow area extraction unit 25 has a function of extracting a shadow area existing in the image based on the color component image read from the image storage unit 23. In FIG. 2, the shadow region extraction unit 25 is configured to use the color component image stored in the image storage unit 23 as basic data for extracting the shadow region. It is also possible to extract a shadow area using the acquired color image as base data. The shadow region extraction result extracted by the shadow region extraction unit 25 is supplied to the shadow boundary / shadow region integration unit 26.

  Note that the present invention does not particularly limit the method for extracting a shadow area existing in the captured image, that is, the shadow area extraction unit 25 extracts, for example, a shadow area present in the captured image. Any shadow region extraction technique including the techniques disclosed in Non-Patent Documents 1 and 2 can be used.

  The shadow boundary / shadow region integration unit 26 recognizes shadows based on both the shadow boundary extraction result extracted by the shadow boundary extraction unit 24 and the shadow region extraction result extracted by the shadow region extraction unit 25. And has a function of generating a shadow recognition result. The shadow boundary / shadow region integration unit 26 generates, for example, a shadow recognition result in which each extraction result is complemented by combining the shadow boundary extraction result and the shadow region extraction result (see FIG. 8).

  The shadow recognition result output unit 27 has a function of outputting the shadow recognition result supplied from the shadow boundary / shadow region integrating unit 26 to the outside (for example, the shadow correction device 30 or an arbitrary recording medium).

<First Embodiment>
Next, a first embodiment of the present invention will be described with reference to FIG. FIG. 3 shows an example of the configuration of the shadow boundary extraction unit 24 of the shadow recognition device 20 according to the first embodiment of the present invention. Note that the shadow boundary extraction unit 24 illustrated in FIG. 3 is a configuration example of the shadow boundary extraction unit 24 in the shadow recognition device 20 illustrated in FIG.

  The shadow boundary extraction unit 24 illustrated in FIG. 3 includes a color component image acquisition unit 2411, an edge extraction unit 2412, a binarization processing unit 2413, a neighborhood pattern calculation unit 2414, a pattern matching / shadow boundary determination unit 2415, a shadow boundary. A pattern storage unit 2416, a shadow boundary extraction result storage unit 2417, and a shadow recognition result output unit 2418 are provided. Note that the function of the shadow boundary extraction unit 24 shown in a block diagram in FIG. 3 can be realized by hardware and / or software executed by a computer.

  3, the color component image acquisition unit 2411 has a function of acquiring the color component image stored in the image storage unit 22 of FIG. 2 as necessary. The color component image acquired by the color component image acquisition unit 2411 is supplied to the edge extraction unit 2412 and the neighborhood pattern calculation unit 2414.

  The edge extraction unit 2412 has a function of extracting an edge (an outline of an object existing in the image) in the color component image supplied from the color component image acquisition unit 2411. The edge extraction unit 2412 can perform edge extraction using a known edge extraction technique such as an edge extraction technique using a Sobel filter. The edge extraction result (edge image) obtained by the edge extraction unit 2412 is supplied to the binarization processing unit 2413. Note that an edge image is generated for each color component image.

  The binarization processing unit 2413 has a function of binarizing the edge image supplied from the edge extraction unit 2412. Note that the binarization processing unit 2413 can binarize the edge image using a known technique such as Otsu's method capable of automatically setting a binarization threshold. The edge image binarized by the binarization processing unit 2413 is a monochrome image in which only the edge of the object in the color component image is drawn. Hereinafter, a pixel representing an edge in the binarized edge image is referred to as an edge pixel. Note that it is difficult to determine at this stage whether the edge pixel obtained by the binarization processing unit 2413 is a shadow boundary pixel or a simple object edge pixel. The binarized edge image (edge pixel) obtained by the binarization processing unit 2413 is supplied to the neighborhood pattern calculation unit 2414.

  The neighborhood pattern calculation unit 2414 identifies an edge pixel in the image based on the edge image supplied from the binarization processing unit 2413, and coordinates of each edge pixel in the color component image acquired from the color component image acquisition unit 2411. With respect to the position, it has a function of calculating the relationship (neighboring pattern) between the color component value of the pixel at the coordinate position and the color component value of the pixel existing in the vicinity of the pixel at the coordinate position. As pixels existing in the vicinity of a certain pixel, for example, as shown in FIG. 10 described later, it is possible to set pixels in the vicinity of 8 directions (near 8). The neighborhood pattern calculated by the neighborhood pattern calculation unit 2414 is supplied to the pattern matching / shadow boundary determination unit 2415.

  In addition, the pattern matching / shadow boundary determining unit 2415 applies the neighborhood of each pixel supplied from the neighborhood pattern calculating unit 2414 to any one of the plurality of shadow boundary patterns stored in the shadow boundary pattern storage unit 2416. It has a function of collating whether or not the patterns match, and determining a pixel in the neighboring pattern that is confirmed to be a match as a pixel at the shadow boundary. Information (such as coordinate position information) related to pixels determined to be shadow boundaries by the pattern matching / shadow boundary determination unit 2415 is stored in the shadow boundary extraction result storage unit 2417. In the following, a case where the color component value (RGB value) of each pixel is used for the calculation of the pattern is taken as an example, but any type of pixel value (a value unique to each pixel) is used. Is possible.

  The shadow boundary pattern storage unit 2416 stores in advance a plurality of shadow boundary patterns that are used as a criterion for determining a shadow boundary by the pattern matching / shadow boundary determining unit 2415. The plurality of shadow boundary patterns stored in advance in the shadow boundary pattern storage unit 2416 are all neighboring patterns specific to the pixels on the shadow boundary. In the first embodiment of the present invention, a pixel having the same neighborhood pattern as the shadow boundary pattern stored in the shadow boundary pattern storage unit 2416 is determined as a shadow boundary pixel. For example, it is possible to easily generate a shadow boundary pattern to be stored in the shadow boundary pattern storage unit 2416 by calculating and accumulating a pixel neighborhood pattern that is known in advance as a shadow boundary.

  Further, the shadow boundary extraction result storage unit 2417 stores information (such as coordinate position information) related to pixels determined to be shadow boundaries by the pattern matching / shadow boundary determination unit 2415. In addition, it is desirable to manage information (such as coordinate position information) related to pixels determined to be shadow boundaries for each image.

  Also, the shadow boundary extraction result output unit 2418 outputs information related to the pixel determined to be a shadow boundary stored in the shadow boundary extraction result storage unit 2417 to the outside as a shadow boundary extraction result, for example, in units of images. It has a function to do.

  Next, the operation of the shadow boundary extraction unit 24 of the shadow recognition device 20 according to the first embodiment of the present invention will be described with reference to FIG. FIG. 4 is a flowchart showing an example of the operation of the shadow boundary extraction unit 24 of the shadow recognition device 20 according to the first embodiment of the present invention. Note that the operation according to the flowchart illustrated in FIG. 4 is executed by the shadow boundary extraction unit 24 of the shadow recognition device 20 illustrated in FIGS. 1 and 2.

  In the flowchart shown in FIG. 4, first, the color component image acquisition unit 2411 acquires a color component image (step S1001), and the edge extraction unit 2412 performs edge extraction processing for each of the color component images. An edge image is generated (step S1002), and the binarization processing unit 2413 performs binarization processing on each edge image (step S1003).

  Then, the neighborhood pattern calculation unit 2414 selects one edge pixel specified by the binarized edge image (step S1004), and the color component value of the color component image corresponding to the coordinate position of the selected edge pixel. Based on the above, the neighborhood pattern of the edge pixel is calculated (step S1005).

  Here, the calculation process of the neighborhood pattern in step S1005 will be described with a specific example. In step S1005, as shown in FIG. 10, for example, pixels in the vicinity of eight directions around the certain pixel (near eight) are set as pixels existing in the vicinity of the certain pixel, and step S1004 is set. By comparing the color component value of the edge pixel selected in Step 1 with the color component value of each pixel in the vicinity of the edge pixel, the neighborhood pattern of the edge pixel is determined.

  For example, in step S1005, first, in each color component image (R image, G image, B image), the color component value (R0, G0, B0) of the edge pixel selected in step S1004 and the edge pixel are the center. Comparison is made with the color component values (Ri, Gi, Bi) (where i = 1 to an integer of 1 to 8) of each pixel in the vicinity of 8.

For a relationship between pixels that satisfy a predetermined condition (for example, a condition of R0 <Ri, G0 <Gi, B0 <Bi), a determination flag = 1 is set, and a relationship between pixels that does not satisfy the above condition is By setting the determination flag = 0, the determination flag in each of the R image, the G image, and the B image of the edge pixel selected in step S1004 is determined. In FIG. 10, R0 <R1, R0 <R2, R0 <R3, R0 <R4 for the R image, and G0 <G2, G0 <G3, G0 <G4, G0 <G6 for the G image. For the B image, determination flags determined when B0 <B1, B0 <B2, and B0 <B3 are schematically illustrated. When 8 neighborhoods are set in the RGB color system (three color components), numerically, (2 ⁸ ) ^three determination flag patterns can exist.

Further, when the value of the determination flag pattern described above satisfies a predetermined condition (for example, the determination flag values of the R, G, and B images in the same pixel are all 1), the value of the 8-neighbor pattern is set to 1. Thus, an 8-neighbor pattern of the edge pixel selected in step S1004 is generated. In the example of FIG. 10, the value of the determination flag of the pixels present on the upper center side and the upper right side with respect to the edge pixel is 1 in all of the R image, the G image, and the B image. An 8-neighboring pattern in which the 8-neighboring pattern value of pixels existing on the upper center side and upper right side is 1 (others are 0) is provided. 8 near the pattern is aggregated pattern determination flag that existed ^{in triplicate} (2 ^8), numerically, it may be present are two ^8.

  In addition, when the HSV color system is used in the color space, the color component image may use only the V image representing the brightness, and the V value (V0) of the edge pixel and each pixel in the vicinity of 8 may be used. By determining whether or not the relationship with the V value (Vi) (where i is an integer of 1 to 8) satisfies a predetermined condition, it is selected in step S1004 as in the case of the RGB color system described above. It is possible to generate an 8-neighbor pattern of the edge pixels. As described above, when 8 neighborhoods are set in the HSV color system (only one value V indicating brightness), the value of the determination flag and the value of the 8 neighborhood pattern are the same.

  The 8-neighbor pattern calculated in step S1005 is supplied to the pattern matching / shadow boundary determining unit 2415. At this time, the pattern matching / shadow boundary determination unit 2415 reads one shadow boundary pattern stored in the shadow boundary pattern storage unit 2416 (step S1006), and whether or not the pattern matches / matches the eight neighboring patterns calculated in step S1005. These are collated (step S1007). If they do not match, the presence of a new shadow boundary pattern that has not been read out from the shadow boundary pattern storage unit 2416 is confirmed and a new shadow boundary pattern is read out (step S1008). It is sequentially checked whether or not there is a shadow boundary pattern that matches the calculated 8-neighbor pattern.

  Here, the shadow boundary pattern read in step S1006 will be specifically described. The shadow boundary pattern is prepared in advance in the shadow boundary pattern storage unit 2416. As described above, for example, by calculating a neighborhood pattern of a pixel (shadow boundary pixel) that is known in advance as a shadow boundary, The calculation result can be stored as a shadow boundary pattern.

  For example, as shown in FIG. 11, edge pixels that are known to be shadow boundaries and pixels in the vicinity thereof are prepared in advance, and a neighborhood pattern calculation algorithm (in FIG. 10) used in the neighborhood pattern calculation unit 2414 is prepared. It is possible to generate a shadow boundary pattern of shadow boundary pixels by applying the above-described 8-neighboring pattern calculation algorithm schematically illustrated. In order to improve the extraction accuracy of the shadow boundary, it is necessary to prepare a large number of shadow boundary patterns. Therefore, it is desirable to prepare various shadow boundary images, calculate a large number of shadow boundary patterns, and store them in the shadow boundary pattern storage unit 2416. Note that the shadow boundary pattern generation method illustrated in FIG. 11 is an example, and the shadow boundary pattern generation method according to the present invention is not limited to this.

  In step S1007, if the eight neighboring patterns calculated in step S1005 match the shadow boundary pattern, the pattern matching / shadow boundary determination unit 2415 determines that the edge pixel is a shadow boundary pixel ( In step S1009), the shadow boundary extraction result storage unit 2417 stores information indicating that the edge pixel is a shadow boundary pixel. On the other hand, if the eight neighboring patterns calculated in step S1005 do not match all the shadow boundary patterns, it is not determined that the edge pixel is a shadow boundary pixel.

  The processing in steps S1004 to S1009 is performed for all edge pixels. That is, when the determination of whether or not the selected edge pixel is a shadow boundary pixel is completed, it is confirmed whether or not all the edge pixels have been selected (step S1010). If there is an unselected edge pixel, the process returns to step S1004, and the processes of steps S1004 to S1009 are performed on the unselected edge pixel. On the other hand, when all the edge pixels are selected (that is, when a determination is made regarding all the edge pixels), the process ends.

  As described above, according to the shadow boundary extraction method according to the first embodiment of the present invention, the pattern representing the relationship between the pixels is a pattern characteristic of the shadow boundary pixels (shadow boundary pattern). If so, the pixel extracts the shadow boundary by determining that the pixel is a shadow boundary pixel that forms the boundary of the shadow in the image (the boundary between the shadow region and the non-shadow region). It becomes possible.

<Second Embodiment>
Next, a second embodiment of the present invention will be described with reference to FIG. FIG. 5 shows an example of the configuration of the shadow boundary extraction unit 24 of the shadow recognition device 20 according to the second embodiment of the present invention. The shadow boundary extraction unit 24 illustrated in FIG. 5 is a configuration example of the shadow boundary extraction unit 24 in the shadow recognition device 20 illustrated in FIG.

  The shadow boundary extraction unit 24 illustrated in FIG. 5 includes a color component image acquisition unit 2421, an edge extraction unit 2422, a binarization processing unit 2423, an edge gradient direction calculation unit 2424, a shadow boundary determination condition calculation unit 2425, a shadow boundary. A determination unit 2426, a shadow boundary extraction result storage unit 2427, and a shadow extraction result output unit 2428 are included. Note that the function of the shadow boundary extraction unit 24 shown in a block diagram in FIG. 5 can be realized by hardware and / or software executed by a computer.

  In FIG. 5, the functions of the color component image acquisition unit 2421, the edge extraction unit 2422, the binarization processing unit 2423, the shadow boundary extraction result storage unit 2427, and the shadow boundary extraction result storage unit 2428 are illustrated in FIG. The functions are the same as those of the color component image acquisition unit 2411, the edge extraction unit 2412, the binarization processing unit 2413, the shadow boundary extraction result storage unit 2417, and the shadow boundary extraction result storage unit 2418. However, the color component image acquired by the color component image acquisition unit 2421 is also supplied to the shadow boundary determination condition calculation unit 2426 in addition to the edge extraction unit 2422. The edge extraction result obtained by the edge extraction unit 2422 is also supplied to the edge gradient direction calculation unit in addition to the binarization processing unit 2423. The binarized edge image obtained by the binarization processing unit 2423 is supplied to the edge gradient direction calculation unit 2424 and the shadow boundary determination condition calculation unit 2425.

  Further, in the second embodiment of the present invention, for each edge pixel of the edge image binarized by the binarization processing unit 2423, an edge gradient direction calculation process and a color component value statistical process (for example, an average value) However, if the edges are thick, these processes are performed a plurality of times at the same location, which is inefficient. Therefore, in order to reduce processing, it is desirable to perform further thinning processing on the edges included in the binarized edge image obtained by the binarization processing unit 2423.

  Further, the edge gradient direction calculation unit 2424 has a function of obtaining the edge gradient direction from the calculation result of the Sobel filter, for example, for each edge pixel. The calculated edge gradient direction indicates a direction substantially perpendicular to the edge in the image plane, but in the second embodiment of the present invention, the strict edge gradient direction for each edge pixel. There is no need to calculate. The edge gradient direction will be described later with reference to FIG. 12, for example.

  Further, the shadow boundary determination condition calculation unit 2425 faces the edge gradient direction calculated by the edge gradient direction calculation unit 2424, and the edge is determined on a straight line passing through the edge pixel that determines whether the pixel is a shadow boundary. It has a function of calculating the statistic of the color component value of a pixel belonging to each of two regions (regions existing on both sides of the target pixel) sandwiching the pixel. Then, the shadow boundary determination condition calculation unit 2425 grasps the spectral characteristic in the vicinity of the edge pixel based on the calculated statistic of the color component value, and determines whether the edge pixel is a shadow boundary pixel or not by this spectral characteristic. Has the function to judge. Note that, here, the shadow boundary determination condition calculation unit 2425 is described as an example of a configuration that calculates the statistic of each color component value of two regions separated by an edge. These pixel values (values unique to each pixel) may be used. In addition, any statistical processing algorithm can be used for calculation of the statistic, but in the following, a case where an average value is used as the statistic will be described as an example.

  Also, the shadow boundary determination unit 2426 determines whether or not the target pixel exists at the shadow boundary based on the condition for changing the average value of the pixels on the straight line in the gradient direction passing through the target pixel calculated by the shadow boundary determination condition calculation unit 2425. It has the function to judge. Information (such as coordinate position information) relating to the pixel determined to be a shadow boundary by the shadow boundary determination unit 2426 is stored in the shadow boundary extraction result storage unit 2427.

  Next, the operation of the shadow boundary extraction unit 24 of the shadow recognition device 20 according to the second embodiment of the present invention will be described with reference to FIG. FIG. 6 is a flowchart showing an example of the operation of the shadow boundary extraction unit 24 of the shadow recognition device 20 according to the second embodiment of the present invention. Note that the operation according to the flowchart illustrated in FIG. 6 is executed by the shadow boundary extraction unit 24 of the shadow recognition device 20 illustrated in FIGS. 1 and 2.

  In the flowchart shown in FIG. 6, first, the color component image acquisition unit 2421 acquires a color component image (step S2001), and the edge extraction unit 2422 performs edge extraction processing for each of the color component images. An edge image is generated (step S2002), and the binarization processing unit 2423 performs binarization processing on each edge image (step S2003). Here, the thinning process may be performed on the binarized edge image.

  Then, the edge gradient direction calculation unit 2424 selects one edge pixel specified by the binarized edge image (step S2004), and the color component of the color component image corresponding to the coordinate position of the selected edge pixel. Based on the value, the edge gradient direction is calculated (step S2005).

  Here, the calculation process of the edge gradient direction in step S2005 will be described with a specific example. The edge pixel existing at a certain coordinate position is located on the boundary between two regions separated by the edge. At this time, when the direction of the line segment passing through the edge pixel moves perpendicular to the line segment and exceeds the line segment passing through the edge pixel, the direction in which the change in the color component value of the pixel is the largest is the edge. Gradient direction. The edge gradient direction may be a perpendicular to the tangent to the boundary near a certain edge pixel, for example.

  The calculation of the edge gradient direction can be performed using any known technique. For example, when the Sobel filter is used in the edge extraction process in Step S2002, it is possible to obtain the vertical edge extraction Sobel filter result and the horizontal edge extraction Sobel filter result of the edge pixel simultaneously with the edge extraction process in Step S2002. These Sobel filter results include information indicating the gradient of the change in color component value between adjacent pixels in two vertical and horizontal directions, and for an edge pixel at a certain coordinate position (x, y). By applying the following equation (1), it is possible to obtain the edge gradient direction g (x, y) as shown in FIG.

In Equation (1), R _V (x, y), G _V (x, y), and B _V (x, y) are vertical edge extraction Sobel filters for each color component value at the coordinate position (x, y). R _H (x, y), G _H (x, y), and B _H (x, y) are horizontal edge extraction Sobel filter results of each color component value at the coordinate position (x, y).

  In equation (1), a more accurate edge gradient direction can be obtained by replacing the sum of the Sobel filter results of each color component value with a norm having the Sobel filter result of each color component value as a vector element. However, the edge gradient direction to be obtained in step S2005 is a direction substantially perpendicular to the edge in the image plane, and it is not always necessary to obtain an accurate value.

  For example, in Equation (1), the edge gradient direction considering three color component values is obtained. For example, the edge gradient direction related to only one color component value is obtained, and the edge gradient direction is determined as the coordinate position (x , Y), or the edge gradient direction of the edge pixel existing in the vicinity of the coordinate position (x, y) can be used as the edge pixel of the coordinate position (x, y). . Alternatively, the edge gradient direction may be determined by using a vertical edge extraction result or a horizontal edge extraction result obtained from a gray scale image or an edge image of a color component value of another color system.

  The edge gradient direction calculation process in step S2005 is performed for all edge pixels. That is, when the calculation process of the edge gradient direction related to the selected edge pixel is completed, it is confirmed whether or not there is an edge pixel whose edge gradient direction is not calculated (unselected edge pixel) (step S2006). If there is an unselected edge pixel, a new edge pixel is selected and the edge gradient direction is sequentially calculated.

  On the other hand, when the calculation of the edge gradient direction is completed for all edge pixels, the calculation based on the calculated edge gradient direction is subsequently performed for each edge pixel. The shadow boundary determination condition calculation unit 2425 selects one edge pixel specified by the binarized edge image (step S2007), and performs a calculation based on the edge gradient direction of the edge pixel (step S2008). Specifically, in step S2008, a line segment that passes through the selected edge pixel and is parallel to the edge gradient direction of the edge pixel is set, and the two regions that are separated by sandwiching the edge pixel are set. It is possible to calculate the average value of each color component value.

  Hereinafter, the calculation based on the edge gradient direction of the edge pixel in step S2008 will be described with a specific example. In step S2008, one and the other region (two regions separated by a tangent line perpendicular to the edge gradient direction) across the edge pixel are considered. Then, an average value of each color component value of a plurality of pixels existing on the line segment of one region is calculated, and an average value of each color component value of a plurality of pixels present on the line segment of the other region is calculated. . In addition, as shown in FIG. 12, by setting an area including pixels for calculating an average value at a point slightly away from the edge pixels, it is possible to avoid the influence of overshoot that occurs near the edges. Is possible.

FIG. 13 schematically shows the average value on the line segment AB in the edge gradient direction calculated in step S2008 for the edge pixel at the coordinate position (x, y) shown in FIG. Yes. Incidentally, each average value F _R ^N of the RGB value of the A side of the calculated line segment AB in step S2008, F _G ^N, is denoted by F _B ^N, the B side of the line segment AB of the RGB values of The average value is expressed as F _R ^S , F _G ^S , F _B ^S.

  The average value calculation process in step S2008 is also performed for all edge pixels. That is, when the calculation processing of the average value related to the selected edge pixel is completed, it is confirmed whether or not there is an edge pixel whose average value is not calculated (unselected edge pixel) (step S2009). If there is an edge pixel, a new edge pixel is selected and the average value is calculated sequentially.

  Then, based on the average value calculated for the edge pixels as described above, the shadow boundary determination unit 2426 determines whether each edge pixel is a shadow boundary pixel, and determines the edge pixels belonging to the shadow boundary. (Step S2010).

  When determining whether or not the edge pixel is a shadow boundary pixel in step S2010, for example, an edge pixel that satisfies both of the following two conditions (first and second conditions) is determined to be a shadow boundary pixel.

(First condition)
Considering that the RGB values are all smaller in the shadow area than in the non-shadow area, an edge pixel that satisfies the following relational expression is determined as a pixel candidate for the shadow area.

F _R ^S <F _R ^N
F _G ^S <F _G ^N
F _B ^S <F _B ^N

(Second condition)
In the case of the outdoors, sunlight has a white to slightly yellowish color, and ambient light has a strong blue component of scattered light from the sky. Considering this condition, an edge pixel satisfying the following expression (2) is set as a shadow region pixel candidate. If the spectral characteristics of the light from the light source and the spectral characteristics of the ambient light are known in advance, the conditions of the following formula (2) are appropriately changed in accordance with the spectral characteristics so that the light is not affected by the environment other than outdoors. It is possible to extract the boundary.

r _R <r _G <r _B (2)
However,
r _R = F _R ^S / (F _R ^N −F _R ^S )
r _G = F _G ^S / (F _G ^N −F _G ^S )
r _B = F _B ^S / (F _B ^N −F _B ^S )

Since the above formula is substituted into formula (2), it can be seen that the following formula also satisfies the condition of formula (2). Therefore, the following formula can also be used.
r _R = F _R ^S / F _R ^N
r _G = F _G ^S / F _G ^N
r _B = F _B ^S / F _B ^N

  Further, when it is known that the spectral reflection characteristics of the material do not change greatly with the RGB color component values, the following tendency is expected when the blue component of the ambient light is strong, such as outdoors.

F _R ^S <F _G ^S <F _B ^S

  This condition satisfies the condition of the above expression (2), and instead of the above expression (2), an edge pixel satisfying the following expression (3) may be used as a pixel candidate for the shadow region.

_{^{F R N -F R S> F}} G N -F G S> F B N -F B S ··· (3)

  Similarly to the condition of the expression (2), the condition of the expression (3) is the above-described expression (3) according to the spectral characteristic when the spectral characteristic of the light of the light source and the spectral characteristic of the ambient light are known in advance. ), It is possible to extract a shadow boundary even in an environment other than the outdoors.

  As described above, the information related to the shadow boundary pixel determined in step S2010 is stored in the shadow boundary extraction result storage unit 2427 as the shadow boundary extraction result.

  As described above, according to the shadow boundary extraction method according to the second embodiment of the present invention, the shadow boundary is extracted by using the fact that the shadow area is lighter than the non-shadow area. Is possible. Further, in the case of the RGB color system, it is possible to extract a shadow boundary by utilizing the fact that all the RGB values of the shadow area are smaller than the RGB values of the non-shadow area. Furthermore, the spectral characteristics of the light irradiated to the shadow area (the shadow area is not irradiated with light from the light source, but the light reflected from the wall (environment light) is irradiated, for example), and the non-shadow area The shadow boundary can be extracted by utilizing the difference from the spectral characteristic of the light irradiated on the.

<Third Embodiment>
Next, a third embodiment of the present invention will be described with reference to FIG. FIG. 7 illustrates an example of the configuration of the shadow boundary extraction unit 24 of the shadow recognition device 20 according to the third embodiment of the present invention. The shadow boundary extraction unit 24 illustrated in FIG. 7 is a configuration example of the shadow boundary extraction unit 24 in the shadow recognition device 20 illustrated in FIG.

  The shadow boundary extraction unit 24 illustrated in FIG. 7 includes a color component image acquisition unit 2431, an edge extraction unit 2432, a binarization processing unit 2433, a neighborhood pattern calculation unit 2434, a pattern matching / shadow boundary determination unit 2435, a shadow boundary. A pattern storage unit 2436, an edge gradient direction calculation unit 2437, a shadow boundary determination condition calculation unit 2438, a shadow boundary determination unit 2439, a shadow boundary extraction result storage unit 2440, and a shadow extraction result output unit 2441 are provided.

  The color component image acquisition unit 2431, the edge extraction unit 2432, the binarization processing unit 2433, the neighborhood pattern calculation unit 2434, the pattern matching / shadow boundary determination unit 2435, and the shadow boundary pattern storage unit 2436 shown in FIG. The same functions as the color component image acquisition unit 2411, edge extraction unit 2412, binarization processing unit 2413, neighborhood pattern calculation unit 2414, pattern matching / shadow boundary determination unit 2415, and shadow boundary pattern storage unit 2416 illustrated in FIG. The edge gradient direction calculation unit 2437, the shadow boundary determination condition calculation unit 2438, the shadow boundary determination unit 2439, the shadow boundary extraction result storage unit 2440, and the shadow extraction result output unit 2441 illustrated in FIG. Edge gradient direction calculation unit 2424, shadow boundary determination condition calculation unit 2425, shadow boundary determination unit 2426, shadow boundary extraction illustrated in FIG. Result storing unit 2427 has the same functions and shadow extraction result output section 2428.

  Hereinafter, for convenience, a mechanism having a color component image acquisition unit 2431, an edge extraction unit 2432, a binarization processing unit 2433, a neighborhood pattern calculation unit 2434, a pattern matching / shadow boundary determination unit 2435, and a shadow boundary pattern storage unit 2436. Is called the first half processing mechanism, and the mechanism having the edge gradient direction calculation unit 2437, the shadow boundary determination condition calculation unit 2438, the shadow boundary determination unit 2439, the shadow boundary extraction result storage unit 2440, and the shadow extraction result output unit 2441 is called the second half processing mechanism. I will call it.

  The third embodiment of the present invention is basically a combination of the first and second embodiments of the present invention described above. That is, the shadow boundary extraction method illustrated in FIG. 4 is executed by the first half processing mechanism illustrated in FIG. 7, and the shadow boundary illustrated in FIG. 6 is performed by the second half processing mechanism illustrated in FIG. An extraction method is executed.

  The shadow boundary extraction method disclosed in the first embodiment of the present invention, for example, immediately detects a match between a neighborhood pattern calculated for a pixel and the shadow boundary pattern. However, the shadow boundary extraction accuracy is disclosed in the second embodiment of the present invention. It may be lower than the shadow boundary extraction method.

  On the other hand, the shadow boundary extraction method disclosed in the second embodiment of the present invention can extract a shadow boundary with high accuracy by referring to the spectral characteristics of an image. There are many calculation processes such as value calculation, and the process may take a relatively long time compared to the shadow boundary extraction method disclosed in the first embodiment of the present invention.

  Therefore, in the third embodiment of the present invention, as shown in FIG. 7, the shadow extraction processing result generated in the first half processing mechanism by combining the configurations shown in FIG. 3 and FIG. Only with respect to (in the configuration shown in FIG. 5, it is necessary to perform processing on all edge pixels), the shadow processing is further performed in the latter half processing mechanism. This realizes higher extraction accuracy than the technique disclosed in the first embodiment of the present invention and a shorter processing time than the technique disclosed in the second embodiment of the present invention. It becomes possible to do.

  Next, referring to FIG. 8, the shadow boundary of the shadow recognition device 20 that performs shadow recognition based on the shadow boundary extraction result obtained by the shadow boundary extraction method according to the first to third embodiments of the present invention. The configuration of the shadow area integration unit 26 will be described. FIG. 8 illustrates an example of the configuration of the shadow boundary / shadow region integration unit 26 of the shadow recognition device 20 that uses the shadow boundary extraction results obtained in the first to third embodiments of the present invention. The shadow boundary / shadow region integration unit 26 illustrated in FIG. 8 is a configuration example of the shadow boundary / shadow region integration unit 26 of the shadow recognition device 20 illustrated in FIG.

  The shadow boundary / shadow region integration unit 26 illustrated in FIG. 8 includes a shadow boundary extraction result acquisition unit 261, a shadow region extraction result acquisition unit 262, a shadow boundary / shadow region relationship calculation unit 263, a shadow determination unit 264, and shadow recognition. A result storage unit 265 is included. Note that the function of the shadow boundary / shadow region integration unit 26 shown in the block diagram in FIG. 8 can be realized by hardware and / or software executed by a computer.

  In FIG. 8, the shadow boundary extraction result acquisition unit 261 has a function of acquiring the shadow boundary extraction result generated by the shadow boundary extraction unit 24 illustrated in FIG. The shadow area extraction result acquisition unit 262 has a function of acquiring the shadow area extraction result generated by the shadow area extraction unit 25 illustrated in FIG. The shadow boundary extraction result acquired by the shadow boundary extraction result acquisition unit 261 and the shadow region extraction result acquired by the shadow region extraction result acquisition unit 262 are supplied to the shadow boundary / shadow region relationship calculation unit 263.

  Further, the shadow boundary / shadow region relationship calculating unit 263 is based on the relationship between the shadow boundary extraction result acquired by the shadow boundary extraction result acquisition unit 261 and the shadow region extraction result acquired by the shadow region extraction result acquisition unit 262. Thus, each pixel has a function of calculating a determination value obtained by integrating the shadow boundary extraction result and the shadow region extraction result. The determination value of each pixel calculated by the shadow boundary / shadow region relationship calculating unit 263 is supplied to the shadow determining unit 264. Note that the determination value calculated by the shadow boundary / shadow region relationship calculating unit 263 will be specifically described later.

  The shadow determination unit 264 has a function of determining a shadow present in the image based on the determination value calculated by the shadow boundary / shadow region relationship calculation unit 263. The determination result in the shadow determination unit 264 is a final recognition result of the shadow present in the image. The shadow recognition result of each pixel obtained by the shadow determination unit 264 is stored in the shadow recognition result storage unit 265. The shadow determination method in the shadow determination unit 264 will be specifically described later.

  The shadow recognition result storage unit 265 stores the shadow recognition result obtained by the shadow determination unit 264 (for example, information indicating which pixel belongs to the shadow). The shadow recognition result stored in the shadow recognition result storage unit 265 is output from the shadow recognition result output unit 27 of the shadow recognition apparatus 20 to the outside, for example, in units of images.

  Next, an outline of a shadow recognition apparatus for executing shadow recognition based on the shadow boundary extraction results obtained in the first to third embodiments of the present invention will be described with reference to FIG. FIG. 9 is a flowchart showing an example of the operation of the shadow recognition device for executing shadow recognition based on the shadow boundary extraction results obtained in the first to third embodiments of the present invention. The operation according to the flowchart shown in FIG. 9 is executed by the shadow boundary / shadow region integration unit 26 of the shadow recognition device 20 shown in FIGS.

  In the flowchart shown in FIG. 9, first, the shadow boundary extraction result acquisition unit 261 acquires the shadow boundary extraction result (step S2601), and the shadow region extraction result acquisition unit 262 acquires the shadow region extraction result. (Step S2602).

  Then, the shadow boundary / shadow region relationship calculating unit 263 associates the shadow boundary extraction result and the shadow region extraction result for the same pixel in each pixel, and calculates one value (determination value) (step S2603). ).

  Here, the association between the shadow boundary extraction result and the shadow region extraction result by the shadow boundary / shadow region relationship calculating unit 263 will be described. The shadow boundary extraction result includes information indicating that each pixel in the image is a shadow boundary (shadow boundary extraction result = 1) or not (shadow boundary extraction result = 0). The shadow region extraction result includes information indicating that each pixel in the image is a shadow region (shadow region extraction result = 1) or not a shadow region (shadow boundary extraction result = 0). Yes. Accordingly, a certain pixel can be classified into four types with respect to the combination of the shadow boundary extraction result and the shadow region extraction result, and the shadow boundary / shadow region relationship calculating unit 263 performs, for example, the following for the four combinations: The determination value is calculated as follows.

Shadow boundary extraction result = 1 and shadow region extraction result = 1 → judgment value = 2
Shadow boundary extraction result = 0 and shadow area extraction result = 1 → judgment value = 2
Shadow boundary extraction result = 1 and shadow region extraction result = 0 → judgment value = 1
Shadow boundary extraction result = 0 and shadow region extraction result = 0 → judgment value = 0

  Here, the shadow region extraction result is more important than the shadow boundary extraction result, and the same determination value is assigned when the shadow region extraction result = 1, regardless of the value of the shadow boundary extraction result. ing. Moreover, the setting of the determination value is an example, and any appropriate setting can be performed.

  Further, based on the determination value calculated by the shadow boundary / shadow region relationship calculating unit 263, the shadow determining unit 264 takes into account the relationship between the pixels in the image and the balance of the entire image, so that each pixel becomes a shadow. A determination is made as to whether it belongs (step S2604).

  Hereinafter, a shadow determination method performed by the shadow determination unit 264, which is performed based on the determination value calculated by the shadow boundary / shadow region relationship calculation unit 263, will be described. For example, as described below, the shadow determination unit 264 determines that the pixel with the determination value = 2 is a shadow, and determines that the pixel with the determination value = 0 is not a shadow. On the other hand, with respect to a pixel with determination value = 1, a determination value of a pixel adjacent to the pixel is taken into consideration, and a set (cluster) of pixels with determination value = 1 is derived. It is then determined whether or not it is in contact with the pixel having the determination value = 2. A pixel belonging to the set of determination value = 1 that is in contact with a pixel of determination value = 2 at an arbitrary position is determined to be a shadow, and is not in contact with a pixel of determination value = 2 at any position. A pixel belonging to the set of determination value = 1 is determined not to be a shadow.

  The above-described shadow determination method will be described using the image shown in FIG. 14 as an example. FIG. 14 shows a part of an image, in which each pixel and a determination value assigned to each pixel are illustrated. In FIG. 14, there are two sets A and B with determination value = 1.

  In FIG. 14, in the set A with the determination value = 1, some pixels belonging to the set A are adjacent to the pixels with the determination value = 2. Therefore, based on the above-described conditions, all the pixels belonging to the set A with the determination value = 1 are determined to be shadows. On the other hand, in the set B with the determination value = 1, none of the pixels belonging to the set B is adjacent to the pixel with the determination value = 2. Therefore, based on the above-described conditions, it is determined that all the pixels belonging to the set B with the determination value = 1 are not shadows.

  After determining whether or not each pixel belongs to the shadow based on the determination value as described above, the shadow determination unit 264 corrects, for example, the break of the shadow boundary in consideration of the balance of the entire image. Then, the final shadow recognition result is created by filling the closed region surrounded by the shadow boundary (considering the pixels inside the closed region as pixels belonging to the shadow).

  FIG. 15 is a conceptual diagram for explaining the outline of the shadow recognition method based on the shadow boundary extraction result obtained in the first to third embodiments of the present invention. According to the shadow recognition method illustrated in FIG. 9, first, as illustrated in FIG. 15, shadow boundary extraction processing and shadow region extraction processing are performed from the original image 500 in which the shadow is reflected, A shadow boundary extraction result 501 and a shadow region extraction result 502 are obtained. As shown in FIG. 15, the shadow boundary extraction result 501 includes a shadow boundary discontinuity because a shadow boundary edge cannot be sufficiently extracted. , It is assumed that there is a missing shadow region different from the actual shadow.

  The process in the shadow boundary / shadow area relationship calculating unit 263 integrates the shadow boundary extraction results and the shadow area extraction results and expresses them by multi-valued determination values. As a result, an integrated image 503 including a multi-valued determination value is obtained (corresponding to the process in step S2603 in FIG. 9). Then, processing such as clustering of pixels having the determination value = 1 is performed to create an image 504 from which extra shadow boundaries are deleted, and further, correction of shadow boundary interruptions and closed regions by shadow boundaries and shadow regions are created. And the shadow recognition result 505 is generated.

  As described above, according to the shadow recognition method according to the present invention, the results extracted by different approaches of shadow region extraction and shadow boundary extraction are integrated into the shadow reflected in the image. Therefore, it is possible to greatly improve the shadow recognition accuracy as compared with the conventional technique.

  The shadow recognition method and the shadow boundary extraction method according to the present invention can be used for various purposes. For example, by correcting the shadow in the image recognized by the present invention, it is possible to present an image that is easy to see for human eyes (an image with excellent visibility).

  Specifically, an image with high visibility for human eyes can be obtained by increasing the shade of the shadow area in the image or matching the shade of the shadow area with the shade of the non-shadow area adjacent to the shadow area. Can be presented.

  In addition, when there is a sudden change in brightness such as when a vehicle enters a tunnel, it takes time for the human eye to adapt to the darkness. Therefore, the visibility of the driver can be improved by presenting an image in which the shadow portion (the shadow on the road surface and the inside of the tunnel) is brightened.

  Further, for example, by correcting a shadow in an image recognized by the present invention, it is possible to improve processing accuracy of image processing and image recognition.

  For example, when a road area is to be extracted from an image captured by a camera set on a vehicle, a shadow area of another object on the road may not be extracted as a road area. Therefore, it is possible to extract only the road region by matching the shading of the shadow region with the shading of the adjacent non-shadow region.

  In addition, for example, by performing shadow recognition according to the present invention, it is possible to acquire information regarding a three-dimensional object. For example, since a shadow is generated when light from a light source is blocked by a three-dimensional object, it can be used to acquire information (shape, position, etc.) of the three-dimensional object from the recognized shadow.

  In addition, according to the present invention, for example, it is possible to perform shadow recognition more accurately than an image captured in a complicated environment where various objects exist such as a general road, compared to the conventional technique. Become. For example, a product that captures an image with a camera (rear camera) attached to the rear of the vehicle and displays the captured image on the image display unit (the screen of the navigation device) is already on the market for the purpose of assisting the driver in parking. Although it appears, it is difficult to see the shadow part included in the image and to check it. Therefore, there is a possibility that children and pets existing in the shadow area may be missed. On the other hand, according to the present invention, it is possible to correct a specific shadow so that it can be easily seen by the driver by recognizing the shadow reflected together in the rear camera image. By presenting to the driver, it becomes possible to prevent an accident.

  Next, a first application example of the shadow recognition method and the shadow boundary extraction method according to the present invention will be described. In the first application example, the camera is installed in the vehicle, and the captured image captured by the camera is displayed on the image display unit installed in the vehicle so that the driver can confirm the safety in the moving direction of the vehicle. The present invention is applied to a vehicle driving support system for display.

  FIG. 16 is a diagram showing a system configuration of a first application example of the shadow recognition method and the shadow boundary extraction method according to the present invention. In FIG. 16, a color image acquisition unit 601, a color image storage unit 602, a shadow recognition unit 603, a shadow presence / absence determination unit 604, a vehicle shadow specific area information storage unit 605, a vehicle shadow extraction unit 606, a vehicle shadow presence / absence determination A unit 607, a vehicle shadow correction unit 608, and an image display unit 609 are illustrated.

  The color image acquisition unit 601 is a camera that is installed in a vehicle and captures the moving direction of the vehicle, and has a function of capturing a color image. The color image storage unit 602 has a function of holding the color image acquired (captured) by the color image acquisition unit 602.

  The shadow recognizing unit 603 has a function of recognizing a shadow captured in a color image stored in the color image storage unit 603. Note that the shadow recognizing unit 603 can generate a shadow recognition result in the image using the shadow recognition method or the shadow boundary extraction method defined by the present invention.

  The shadow presence / absence determination unit 604 refers to the shadow recognition result supplied from the shadow recognition unit 603 to determine whether there is a shadow in the image, and when there is no shadow in the image. Instructing the color image stored in the color image storage unit 602 to be sent to the image display unit 609, and if there is a shadow in the image, the shadow recognition result is acquired as the vehicle shadow. It has a function of supplying to the unit 605.

  Further, in the own vehicle background image storage unit 605, for example, when a shadow of the own vehicle (own vehicle shadow) is reflected in the color image, an area in the image in which the own vehicle shadow is reflected (own vehicle shadow region: For example, information for specifying the vicinity of a part of the host vehicle on the lower center side of the image is stored. Then, the own vehicle shadow acquisition unit 606 acquires a shadow reflected in this area, and supplies the acquisition result to the own vehicle shadow presence determination unit 607.

  The own vehicle shadow presence / absence determination unit 607 determines whether or not a shadow is reflected in this area, and if a shadow is reflected in this area, determines whether the shadow is a shadow of the own vehicle or not. It has a function of determining whether or not the subject's own car shadow is reflected. Also, the own vehicle shadow presence / absence determination unit 607 instructs the image display unit 609 to send the color image stored in the color image storage unit 602 when the own vehicle shadow is not reflected in the image. On the other hand, when the own vehicle shadow is reflected in the image, the shadow recognition result of the identified own vehicle shadow is supplied to the own vehicle shadow correction unit 608.

  The own vehicle shadow correction unit 608 is stored in the color image storage unit 602 when the own vehicle shadow is reflected in the image and receives the shadow recognition result of the own vehicle shadow from the own vehicle shadow presence / absence determination unit 607. A function to generate a color image in which the influence of the vehicle shadow is reduced (the vehicle shadow is removed) by, for example, processing for improving the brightness of only the image region of the vehicle shadow. ing. The color image corrected by the vehicle shadow correction unit 608 is displayed on the image display unit 609 installed at a position where the driver in the vehicle can visually recognize, for example.

  Next, the operation of the first application example of the shadow recognition method and the shadow boundary extraction method according to the present invention will be described. FIG. 17 is a flowchart showing the operation of the first application example of the shadow recognition method and the shadow boundary extraction method according to the present invention.

  In the flowchart shown in FIG. 17, first, a color image is acquired from a camera installed at an arbitrary location of the vehicle (step S651), and the color image is stored in the color image storage unit 602 (step S652). . Then, shadow recognition processing is performed in the shadow recognizing unit 603, and a shadow in the color image is specified (step S653).

  Next, the shadow presence / absence determination unit 604 determines whether or not there is a shadow in the image by the shadow recognition processing by the shadow recognition unit 603 (step S654). Here, if there is a shadow, the vehicle shadow acquisition unit 606 acquires the vehicle shadow region (step S655), and whether or not there is a shadow (vehicle shadow) in the vehicle shadow region. A determination is made. If the own vehicle shadow exists, the own vehicle shadow correction unit 608 performs correction for removing the own vehicle shadow from the color image (step S656), and then the corrected color image is displayed in the image display unit 609. (Step S657).

  If it is determined in step S654 that there is no shadow, or if it is determined in step S656 that there is no vehicle shadow, the image display unit 609 displays the video acquired by the color image acquisition unit as it is. Is done.

  In the first application example, for example, as illustrated in FIG. 18, a vehicle 660 in which a rear camera 650 is installed at the rear of the vehicle is assumed. Here, the sun 670 is present at a position where the own vehicle shadow 680 is formed in the imaging direction of the rear camera 650, and the object 690 present in the traveling direction of the vehicle 660 (during back travel) is included in the own vehicle shadow. I will enter. As a result, the object 690 existing in the traveling direction of the vehicle 650 becomes dark, and when the image captured by the rear camera 650 is displayed on the image display unit 609, the driver himself / herself is shown in FIG. It may be difficult to visually recognize the object 690 due to the influence of the vehicle shadow 680. However, in the first application example according to the present invention, the own vehicle shadow recognition result is obtained by the own vehicle shadow recognition process, and as a result, the own vehicle shadow 690 can be removed from the image. .

  Next, a second application example of the shadow recognition method and the shadow boundary extraction method according to the present invention will be described. In the second application example, the present invention is applied to image region division used when detecting a region (traveling path) in which a moving body such as a vehicle or a robot can travel autonomously.

  FIG. 20 is a diagram showing a system configuration of a second application example of the shadow recognition method and the shadow boundary extraction method according to the present invention. In FIG. 20, a color image acquisition unit 701, a color image storage unit 702, an image region division unit 703, a region boundary extraction unit 704, a shadow boundary determination unit 705, a region boundary correction unit 706, a travelable region extraction unit 707, and travel are possible. An availability determination unit 708, an alarm unit 709, and a braking control unit 710 are illustrated.

  The color image acquisition unit 701 is a camera that is mounted on a moving body such as a vehicle or a robot, and that captures a direction in which the moving body tries to determine whether or not it can travel, and has a function of capturing a color image. The color image storage unit 702 has a function of holding the color image acquired (captured) by the color image acquisition unit 602.

  The image area dividing unit 703 reads a color image stored in the color image storage unit 702 and uses an existing area dividing technique such as a level set method or a watershed method, for example. It has a function of dividing. A region-divided image is generated by region division by the image region dividing unit 703. In addition, the region boundary extraction unit 704 has a function of extracting the boundary of each region included in the region divided image obtained by the image region division unit 703.

  Further, the shadow boundary determination unit 705 determines whether the boundary of each region obtained by the region boundary extraction unit 704 is an edge of an object or a shadow boundary, thereby determining each region. It has the function of judging only the shadow boundary from the boundaries. Note that the shadow boundary extraction method defined by the present invention can be used in the shadow boundary determination processing performed by the shadow boundary determination unit 705.

  Further, the region boundary correction unit 706 has a function of deleting the shadow boundary determined as the shadow boundary determination unit 705 from the boundaries included in the region divided image from which the boundary is extracted by the region boundary extraction unit 704. is doing.

  In addition, the travelable area extracting unit 707 has a function of extracting the travel path of the moving body based on the area divided image from which the shadow area is deleted. Further, the travelability determination unit 708 has a function of determining the travel path of the mobile body based on the travel path extracted by the travelable area extraction unit 707. For example, when there is no travelable area Instructing the alarm unit 709 to notify the alarm, or instructing the braking control unit 710 that performs the braking control of the moving body to perform a forced stop or move to a travelable area. Is possible.

  Next, the operation of the second application example of the shadow recognition method and the shadow boundary extraction method according to the present invention will be described. FIG. 21 is a flowchart showing the operation of the second application example of the shadow recognition method and the shadow boundary extraction method according to the present invention.

  In the flowchart shown in FIG. 21, first, a color image is acquired from a camera installed at an arbitrary place on the moving body (step S751), and the color image is stored in the color image storage unit 702 (step S752). ). Then, the image area dividing unit 703 performs image area dividing processing to generate an area divided image (step S753).

  Next, in the region boundary extracting unit 704, the boundary of each region is extracted from the region divided image generated by the image region dividing unit (step S754). Then, the shadow boundary determination unit 705 performs, for example, pattern matching performed on the boundary of each region extracted by the region boundary extraction unit 704 using the shadow boundary extraction method according to the above-described first embodiment of the present invention. Alternatively, the shadow boundary is determined and extracted from each region by referring to the spectral characteristics performed by the shadow boundary extraction method according to the second embodiment of the present invention described above (step S755).

  The determination result in the shadow boundary determination unit 705 is supplied to the region boundary correction unit 706, and the region boundary correction unit 706 deletes the shadow boundary from the region boundary included in the region divided image (step S756). At this time, it is desirable that the areas that are in contact with each other through the shadow boundary are integrated.

  Based on the region-divided image from which the influence of the shadow is eliminated as described above, the travelable region extraction unit 707 performs a travelable region extraction process (step S757), and the travelability determination unit 708 extracts it. Whether or not the mobile body can travel is determined based on the travelable area (step S758). While the moving body is in operation, the above-described processing is continuously performed. For example, when travel is impossible, such as when there is no travelable area, a warning is issued from the alarm unit 709 (step S110). S759).

  In the second application example, for example, as illustrated in FIG. 22, when the original image acquired by the color image acquisition unit 701 includes a shadow (shadow 782 of the building 781), It is difficult to perform proper region segmentation. For example, although the region “5” is the same material as the region “1”, the irradiated light is different, so that the region “5” is divided into different regions. Similarly, the region “6” is the same material as the region “7”, but is irradiated with different light, and thus is divided into different regions.

  Therefore, even if the area “7” can be recognized as a traveling path in the traveling path 780 of the moving object, whether the area “6” is another object (for example, an obstacle) or paint on the road, It is difficult to accurately determine whether it is a shadow or a traveling road, and as a result, the region “6” that should originally be able to travel is determined as a region where it cannot travel.

  On the other hand, when the shadow boundary extraction method according to the present invention is used, it is possible to determine that the boundary forming the region “5” or the region “6” is a shadow boundary. As a result, it is possible to clearly distinguish the shadow boundary from the edge of the object and delete only the shadow boundary, and it is possible to prevent erroneous determination due to the influence of the shadow when detecting the road area.

According to the present invention, a shadow recognition device that recognizes the shadow in an image in which a shadow of an arbitrary object is captured by the imaging device,
Image acquisition means for acquiring the image captured by the imaging device;
A shadow area extracting means for extracting a shadow area in the image;
A shadow boundary extracting means for extracting a shadow boundary between the shadow area in the image and a non-shadow area different from the shadow area;
Shadow recognition means for recognizing the shadow based on both the shadow area extracted by the shadow area extraction means and the shadow boundary extracted by the shadow boundary extraction means;
A shadow recognition device is also provided.

In addition, according to the present invention, a shadow boundary extraction that extracts a shadow boundary between a shadow area in the image and a non-shadow area different from the shadow area in an image in which a shadow of an arbitrary object is captured by the imaging device. A device,
Image acquisition means for acquiring the image captured by the imaging device;
Edge extraction means for extracting the edge of the object to be imaged in the image;
Edge pixel specifying means for specifying edge pixels belonging to the edge extracted by the edge extracting means;
Pattern calculating means for calculating a pattern indicating a relationship between pixel values of the edge pixel and a plurality of pixels existing in the vicinity of the edge pixel;
A shadow boundary pattern reference means for referencing a shadow boundary pattern in which a relationship between pixel values of a shadow boundary pixel belonging to the shadow boundary and a plurality of pixels existing in the vicinity of the shadow boundary pixel is set;
Pattern matching means for matching the shadow boundary pattern referred to by the shadow boundary pattern reference means and the pattern calculated by the pattern calculation means;
In pattern matching by the pattern matching unit, pattern matching shadow boundary determination unit that determines the edge pixel belonging to the pattern calculated by the pattern calculation unit that matches the shadow boundary pattern as a pixel existing in the shadow boundary; The
A shadow boundary extraction apparatus is also provided.

In addition, according to the present invention, a shadow boundary extraction that extracts a shadow boundary between a shadow area in the image and a non-shadow area different from the shadow area in an image in which a shadow of an arbitrary object is captured by the imaging device. A device,
Image acquisition means for acquiring the image captured by the imaging device;
Edge extraction means for extracting the edge of the object to be imaged in the image;
A feature amount calculating means for calculating each feature amount of two regions separated by the edge extracted by the edge extracting means;
A feature amount relationship determining means for determining whether or not the relationship between the feature amounts of the two regions matches the relationship between the feature amounts of the shadow region and the non-shadow region separated by the shadow boundary;
In the determination by the feature amount relationship determining means, when the relationship between the feature amounts of the two regions matches the relationship between the feature amounts of the shadow region and the non-shadow region separated by the shadow boundary, the edge is determined. A feature amount matching shadow boundary determining means for determining the shadow boundary;
A shadow boundary extraction apparatus is also provided.

In addition, according to the present invention, a shadow boundary extraction that extracts a shadow boundary between a shadow area in the image and a non-shadow area different from the shadow area in an image in which a shadow of an arbitrary object is captured by the imaging device. A device,
Image acquisition means for acquiring the image captured by the imaging device;
Edge extraction means for extracting the edge of the object to be imaged in the image;
Edge pixel specifying means for specifying edge pixels belonging to the edge extracted by the edge extracting means;
Pattern calculating means for calculating a pattern indicating a relationship between pixel values of the edge pixel and a plurality of pixels existing in the vicinity of the edge pixel;
A shadow boundary pattern reference means for referencing a shadow boundary pattern in which a relationship between pixel values of a shadow boundary pixel belonging to the shadow boundary and a plurality of pixels existing in the vicinity of the shadow boundary pixel is determined;
Pattern matching means for matching the shadow boundary pattern referred to by the shadow boundary pattern reference means and the pattern calculated by the pattern calculation means;
In pattern matching by the pattern matching means, pattern matching shadow boundary determination means for specifying the edge pixel belonging to the pattern calculated by the pattern calculation means that matches the shadow boundary pattern;
An edge gradient direction related to the pixel value of the edge pixel specified by the pattern matching shadow boundary determination unit is calculated, and a calculation result related to the pixel value of a pixel on a straight line that is substantially parallel to the edge gradient direction passing through the edge pixel As a feature amount, feature amount calculation means for calculating each feature amount of two regions separated by the edge, and
A feature amount relationship determining means for determining whether or not the relationship between the feature amounts of the two regions matches the relationship between the feature amounts of the shadow region and the non-shadow region separated by the shadow boundary;
In the determination by the feature amount relationship determining means, when the relationship between the feature amounts of the two regions matches the relationship between the feature amounts of the shadow region and the non-shadow region separated by the shadow boundary, the edge is determined. A feature amount matching shadow boundary determining means for determining the shadow boundary;
A shadow boundary extraction apparatus is also provided.

  The present invention also provides a program for causing a computer to execute the image correction method of the present invention.

It is a figure which shows an example of a structure of the image correction system common to the 1st-3rd embodiment of this invention. It is a figure which shows an example of a structure of the shadow recognition apparatus common to the 1st-3rd embodiment of this invention. It is a figure which shows an example of a structure of the shadow boundary extraction part of the shadow recognition apparatus in the 1st Embodiment of this invention. It is a flowchart which shows an example of operation | movement of the shadow boundary extraction part of the shadow recognition apparatus in the 1st Embodiment of this invention. It is a figure which shows an example of a structure of the shadow boundary extraction part of the shadow recognition apparatus in the 2nd Embodiment of this invention. It is a flowchart which shows an example of operation | movement of the shadow boundary extraction part of the shadow recognition apparatus in the 2nd Embodiment of this invention. It is a figure which shows an example of a structure of the shadow boundary extraction part of the shadow recognition apparatus in the 3rd Embodiment of this invention. An example of the configuration of the shadow boundary / shadow region integration unit of the shadow recognition apparatus that uses the shadow boundary extraction results obtained in the first to third embodiments of the present invention is illustrated. It is a flowchart which shows an example of operation | movement of the shadow recognition apparatus for performing shadow recognition based on the shadow boundary extraction result obtained in the 1st-3rd embodiment of this invention. It is a figure for demonstrating the neighborhood pattern utilized in the 1st Embodiment of this invention. It is a figure for demonstrating the shadow boundary pattern utilized in the 1st Embodiment of this invention. It is a figure for demonstrating the edge gradient direction calculated in the 2nd Embodiment of this invention. It is a figure which shows typically the average value calculated on line segment AB of an edge gradient direction regarding the edge pixel of the coordinate position (x, y) illustrated in FIG. It is a figure for demonstrating the method in which the shadow recognition apparatus using the shadow boundary extraction result obtained in the 1st-3rd embodiment of this invention judges a shadow pixel based on a judgment value. It is a conceptual diagram for demonstrating the outline | summary of the shadow recognition method based on the shadow boundary extraction result obtained in the 1st-3rd embodiment of this invention. It is a figure which shows the system configuration | structure of the 1st application example of the shadow recognition method and shadow boundary extraction method which concern on this invention. It is a flowchart which shows operation | movement of the 1st application example of the shadow recognition method and shadow boundary extraction method which concern on this invention. It is a figure which shows typically the influence of the own vehicle shadow assumed in the 1st application example of the shadow recognition method and shadow boundary extraction method which concern on this invention. It is a figure which shows typically a mode that the own vehicle shadow is removed from the image imaged with the rear camera in the 1st application example of the shadow recognition method and shadow boundary extraction method which concern on this invention. It is a figure which shows the system configuration | structure of the 2nd application example of the shadow recognition method and shadow boundary extraction method which concern on this invention. It is a flowchart which shows operation | movement of the 2nd application example of the shadow recognition method and shadow boundary extraction method which concern on this invention. It is a figure which shows typically a mode that the area | region of a shadow boundary is removed from the area | region divided | segmented by area | region division | segmentation in the 2nd application example of the shadow recognition method and shadow boundary extraction method which concern on this invention. It is a figure which shows an example of the to-be-photographed object and the captured image in the shadow recognition method and shadow boundary extraction method which concern on this invention.

Explanation of symbols

10 Camera (imaging device)
DESCRIPTION OF SYMBOLS 20 Shadow recognition apparatus 21,601,701 Color image acquisition part 22 Color space decomposition part 23 Image storage part 24 Shadow boundary extraction part 25 Shadow area extraction part 26 Shadow boundary and shadow area integration part 27 Shadow recognition result output part 30 Shadow correction apparatus (Image correction device)
40 Display Monitor 50 Captured Image Utilization Device 261 Shadow Boundary Extraction Result Acquisition Unit 262 Shadow Region Extraction Result Acquisition Unit 263 Shadow Boundary / Shadow Region Relationship Calculation Unit 264 Shadow Judgment Unit 265 Shadow Recognition Result Storage Unit 500 Original Image 501 Shadow Boundary Extraction Result 502 Shadow region extraction result 503 Integrated image including determination value 504 Image from which excess shadow boundary is deleted 505 Shadow recognition result 602, 702 Color image storage unit 603 Shadow recognition unit 604 Shadow presence / absence determination unit 605 Own vehicle shadow specific region information storage unit 606 Own vehicle shadow acquisition unit 607 Own vehicle shadow presence determination unit 608 Own vehicle shadow correction unit 609 Image display unit 650 Rear camera 660 Vehicle 670 Sun 680 Own vehicle shadow 690 Object 703 Image region division unit 704 Region boundary extraction unit 705 Shadow boundary determination Section 706 Area boundary correction section 707 Travelable area extraction section 708 Whether travel is possible Fixed part 709 Alarm part 710 Braking control part 780 Traveling road 781 Building 782 Shadow 2411, 2421, 2431 Color component image acquisition part 2412, 2422, 2432 Edge extraction part 2413, 2423, 2433 Binarization processing part 2414, 2434 Neighborhood pattern Calculation unit 2415, 2435 Pattern matching / shadow boundary determination unit 2416, 2436 Shadow boundary pattern storage unit 2417, 2427, 2440 Shadow boundary extraction result storage unit 2418, 2428, 2441 Shadow boundary extraction result output unit 2424, 2437 Edge gradient direction calculation unit 2425, 2438 Shadow boundary determination condition calculation unit 2426, 2439 Shadow boundary determination unit

Claims (11)

A shadow recognition method executed by a computer to recognize a shadow in an image in which a shadow of an arbitrary object is captured by an imaging device,
An image acquisition step of acquiring the image captured by the imaging device;
An area extraction step for extracting an area in which the shadow is assumed to be present in the image using an arbitrary shadow area extraction technique ;
A boundary extraction step of extracting a boundary between an area in which the shadow is assumed to be present in the image and an area in which the shadow is not assumed to be present using a predetermined shadow boundary extraction technique ;
Said region extraction region extraction result of the region extracted in step, and on the basis of the boundary extraction result of the relationship of the boundary extracted by the boundary extracting step, to calculate a decision value for each pixel forming the image determining A value calculation step;
A shadow recognition step of determining whether each of the pixels forming the image belongs to a shadow based on the determination value calculated in the determination value calculation step ;
A shadow recognition method. In the shadow recognition step,
Based on the determination value, a pixel belonging to the region extracted in the region extraction step is determined as a pixel belonging to a shadow , while a pixel that does not belong to either the region or the boundary does not belong to a shadow. Determining a pixel;
Based on the determination value, if the pixels do not belong to the region extracted in the region extraction step and belong to the boundary and are adjacent to each other, the adjacent pixels are aggregated. Set setting step to be considered as
Based on the judgment value, any pixel belonging to the aggregate set by the set setting step, when adjacent to the pixel belonging to the area extracting the region extracted in step to the region Determining all pixels belonging to the aggregate together with pixels belonging to the pixel belonging to a shadow ;
On the basis of the decision value, all pixels belonging to the aggregate set by the set setting step, when not adjacent to the pixel belonging to the area extracting the region extracted in step the assembly Determining all pixels belonging to the pixel not belonging to the shadow ,
The shadow recognition method according to claim 1. A shadow boundary extraction method executed by a computer to extract a shadow boundary between a shadow area in the image and a non-shadow area different from the shadow area in an image in which a shadow of an arbitrary object is captured by an imaging device Because
An image acquisition step of acquiring the image captured by the imaging device;
An edge extraction step of extracting an edge of the object to be imaged in the image;
An edge pixel specifying step for specifying an edge pixel belonging to the edge extracted in the edge extraction step;
A pattern calculation step of calculating a pattern indicating a relationship between pixel values of the edge pixel and a plurality of pixels existing in the vicinity of the edge pixel;
A shadow boundary pattern reference step for referring to a shadow boundary pattern in which a relationship between pixel values of shadow boundary pixels belonging to the shadow boundary and a plurality of pixels existing in the vicinity of the shadow boundary pixel is set;
A pattern matching step for matching the shadow boundary pattern referred to in the shadow boundary pattern reference step with the pattern calculated in the pattern calculation step;
A pattern matching shadow boundary determining step for determining that the edge pixel belonging to the pattern calculated in the pattern calculating step matched with the shadow boundary pattern in the pattern matching step is a pixel existing in the shadow boundary;
A shadow boundary extraction method.   The shadow boundary extraction method according to claim 3, wherein the shadow boundary pattern is generated from a relationship between pixel values of an arbitrary shadow boundary pixel and a plurality of pixels existing in the vicinity of the arbitrary shadow boundary pixel. .   The shadow boundary extraction method according to claim 3, wherein the pixel value is a color component value of an RGB color system. A shadow boundary extraction method executed by a computer to extract a shadow boundary between a shadow area in the image and a non-shadow area different from the shadow area in an image in which a shadow of an arbitrary object is captured by an imaging device Because
An image acquisition step of acquiring the image captured by the imaging device;
An edge extraction step of extracting an edge of the object to be imaged in the image;
A feature amount calculating step of performing binarization processing on the edge extracted in the edge extraction step, and calculating respective feature amounts of two regions separated by the binarized edge ;
A feature amount relationship determination step for determining whether or not the relationship between the feature amounts of the two regions matches the relationship between the feature amounts of the shadow region and the non-shadow region separated by the shadow boundary;
In the feature amount relationship determination step, when the relationship between the feature amounts of the two regions matches the relationship of the feature amounts of the shadow region and the non-shadow region separated by the shadow boundary, the edge is converted into the shadow region. A feature amount matching shadow boundary determination step for determining a boundary,
A shadow boundary extraction method.   The relationship between the characteristic amounts of the shadow area and the non-shadow area separated by the shadow boundary is due to the difference between the spectral characteristics of the light irradiated to the shadow area and the spectral characteristics of the light irradiated to the non-shadow area. The shadow boundary extraction method according to claim 6, wherein the shadow boundary extraction method is determined based on the above. In the feature amount calculating step, an edge gradient direction related to a pixel value of an edge pixel belonging to the binarized edge is calculated, and the pixels on a straight line that is substantially parallel to the edge gradient direction passing through the edge pixel are calculated. The shadow boundary extraction method according to claim 6 or 7, wherein a calculation result relating to a pixel value is used as the feature amount.   The shadow boundary extraction method according to claim 6, wherein the feature amount is a statistic of pixel values of some or all of the pixels belonging to each of the two regions.   The shadow boundary extraction method according to claim 6, wherein the pixel value is a color component value of an RGB color system. A shadow boundary extraction method executed by a computer to extract a shadow boundary between a shadow area in the image and a non-shadow area different from the shadow area in an image in which a shadow of an arbitrary object is captured by an imaging device Because
An image acquisition step of acquiring the image captured by the imaging device;
An edge extraction step of extracting an edge of the object to be imaged in the image;
An edge pixel specifying step for specifying an edge pixel belonging to the edge extracted in the edge extraction step;
A pattern calculation step of calculating a pattern indicating a relationship between pixel values of the edge pixel and a plurality of pixels existing in the vicinity of the edge pixel;
A shadow boundary pattern reference step for referring to a shadow boundary pattern in which a relationship between pixel values of a shadow boundary pixel belonging to the shadow boundary and a plurality of pixels existing in the vicinity of the shadow boundary pixel is determined;
A pattern matching step for matching the shadow boundary pattern referred to in the shadow boundary pattern reference step with the pattern calculated in the pattern calculation step;
A pattern matching shadow boundary determining step for identifying the edge pixel belonging to the pattern calculated in the pattern calculating step that matches the shadow boundary pattern in the pattern matching step;
An edge gradient direction related to a pixel value of the edge pixel specified in the pattern matching shadow boundary determination step is calculated, and a calculation result related to the pixel value of a pixel on a straight line that is substantially parallel to the edge gradient direction passing through the edge pixel A feature amount calculating step for calculating each feature amount of two regions separated by the edge,
A feature amount relationship determination step for determining whether or not the relationship between the feature amounts of the two regions matches the relationship between the feature amounts of the shadow region and the non-shadow region separated by the shadow boundary;
In the feature amount relationship determination step, when the relationship between the feature amounts of the two regions matches the relationship of the feature amounts of the shadow region and the non-shadow region separated by the shadow boundary, the edge is converted into the shadow region. A feature amount matching shadow boundary determination step for determining a boundary,
A shadow boundary extraction method.

Images