KR102156024B1 - Shadow removal method for image recognition and apparatus using the same - Google Patents

Abstract

The present invention relates to a shadow removal method for image recognition, and the shadow removal method for image recognition according to an embodiment of the present invention includes the steps of separating each continuously input image frame into a foreground region and a background region, Extracting shadow candidate pixels using an algorithm for removing one or more shadows from the image frame, the foreground region and the background region separated from the image frame, weighting each of the extracted shadow candidate pixels according to the used algorithm And determining a final shadow pixel by applying and mixing shadow candidate pixels according to the applied weight, and removing the determined shadow pixel from the foreground region.

Description

A shadow removal method for image recognition and a shadow removal device therefor {SHADOW REMOVAL METHOD FOR IMAGE RECOGNITION AND APPARATUS USING THE SAME}

The present invention relates to a shadow removal method for image recognition, and more particularly, a shadow candidate pixel is extracted using an algorithm for removing one or more shadows for consecutive image frames, and then extracted according to the used algorithm. A shadow removal method for image recognition capable of removing the determined shadow pixel after determining a final shadow pixel according to the applied weight by applying a weight according to the analysis technique used to each shadow candidate pixel, and a shadow removal apparatus therefor. .

The content described in this section merely provides background information on the present embodiment and does not constitute the prior art.

In general, video analysis and recognition functions such as video surveillance systems used in video security/surveillance systems and military equipment are basically divided into a foreground region and a background region within an image input from a camera, Objects are detected and tracked within the foreground area. In this case, the accuracy of image recognition depends on how similar the shape of the separated foreground area is to the shape of an actual object. However, in an outdoor environment with severe sunlight changes and an indoor environment with severe lighting changes, it becomes difficult to separate the foreground and background due to the ever-changing shadow, and the separated foreground area is distorted by the shadow and even unnecessary areas are detected to improve analysis accuracy. It drops significantly.

Accordingly, various technologies have been introduced to remove shadows in images. However, the shadow removal techniques up to now simply remove the shadow based on the brightness value, or apply a threshold value in the foreground area separated from the image and remove the part outside the threshold value as a shadow. There is a problem that it is difficult to remove shadows consistently in various environments.

Korean Patent Publication No. 2013-0076049, published on July 8, 2013 (Name: Shadow removal method based on relative correlation)

The present invention has been proposed to solve the above-described problems in the related art, and in particular, an object of the present invention is to provide a shadow removal method for image recognition capable of effectively removing shadows even in various environments, and a shadow removal apparatus therefor.

To this end, the present invention extracts shadow candidate pixels by using at least one analysis technique among color and illuminance analysis, vector dot product analysis, boundary distribution analysis, and frame pixel analysis for consecutive image frames, and then extracts shadow candidate pixels. To provide a shadow removal method for image recognition capable of removing the determined shadow pixels after determining a final shadow pixel according to the applied weight by applying a weight according to the analysis technique used for each, and a shadow removal device for the same. There is this.

However, the object of the present invention is not limited to the above object, and other objects that are not mentioned will be clearly understood from the following description.

A shadow removal method for image recognition according to an embodiment of the present invention for achieving the above object includes the steps of separating each continuously input image frame into a foreground area and a background area; Extracting shadow candidate pixels using an algorithm for removing one or more shadows from the image frame, a foreground region and a background region separated from the image frame; Applying a weight to each of the extracted shadow candidate pixels according to the used algorithm and mixing the shadow candidate pixels according to the applied weight to determine a final shadow pixel; And removing the determined shadow pixel from the foreground area.

In this case, the algorithm for removing the shadow may be color and illuminance analysis, vector dot product analysis, boundary distribution analysis, and frame pixel analysis.

Here, the step of extracting the shadow candidate pixels using the color and illuminance analysis algorithm may include converting the RGB color coordinate system of the foreground area and the background area into an HSV color coordinate system; Detecting a dark area based on a brightness value in the converted foreground area and background area; Firstly extracting a pixel having a predetermined threshold value or higher as a shadow candidate pixel through a saturation comparison of the dark area of the foreground area and the dark area of the background area; Analyzing an amount of change in brightness for the shadow candidate pixels first extracted from the background area and the foreground area; And extracting a pixel whose brightness change amount is less than the preset threshold as a final shadow candidate pixel.

In addition, the step of extracting the shadow candidate pixel using the vector dot product analysis algorithm may include setting a window having a predetermined size for the background region and the foreground region; Calculating an inner product of a vector value of a pixel in the window in the background area and a pixel in the window in the foreground area; And comparing the dot product in the background area with the dot product in the foreground area, and extracting a pixel with a variation of a vector value equal to or greater than a preset threshold as a shadow candidate pixel.

In addition, the extracting of the shadow candidate pixel using the boundary distribution analysis algorithm may include generating a vertical histogram for the foreground region; Setting a range of the x-axis having the highest value as a central axis using the vertical histogram; Predicting a direction of a shadow by analyzing a pixel brightness value within a predetermined radius based on the central axis; And when the shadow direction is predicted, extracting a pixel of a predetermined distance or more from the central axis in the shadow direction as a shadow candidate pixel.

In this case, the step of setting the central axis may include dividing the vertical histogram into a predetermined number of blocks and determining a block area having a high average histogram value for each block as the central axis.

In addition, the extracting of the shadow candidate pixel using the frame pixel analysis algorithm may include extracting an illumination component of the image frame; Generating a reflection component using the lighting component; Correcting a dark area using the value of the reflection component; And extracting a pixel corresponding to the corrected dark area as a shadow candidate pixel.

In this case, the determining of the final shadow pixel may include analyzing information on surrounding landscapes of the current image frame; By comparing the surrounding environment information of the current image frame and the surrounding environment information of the previous image frame, analyzing the amount of change in the surrounding environment information, and weighting each of the extracted shadow candidate pixels according to the algorithm used based on the change amount of the surrounding environment information Applying; And determining a final shadow pixel by mixing the shadow candidate pixels according to the applied weight.

Here, the amount of change of the surrounding environment information may be a shadow pixel distribution map, and an amount of change in intensity and amount of light analyzed based on a histogram.

In addition, in the determining of the final shadow pixel, when no previous image frame exists, it is determined whether or not the shadow candidate pixels extracted according to each algorithm overlap, and a shadow candidate pixel having a large overlapping degree may be determined as the final shadow pixel. have.

In addition, the present invention can provide a computer-readable recording medium recording the shadow removal method for image recognition as described above.

A shadow removal apparatus according to an embodiment of the present invention for achieving the above-described object includes: a foreground background separator for separating each continuously input image frame into a foreground region and a background region; And extracting shadow candidate pixels using an algorithm for removing one or more shadows from the image frame, the foreground region and the background region separated from the image frame, and weighting each of the extracted shadow candidate pixels according to the used algorithm. And a shadow removal unit configured to remove the determined shadow pixel from the foreground area by applying and mixing shadow candidate pixels according to the applied weight to determine a final shadow pixel.

According to the shadow removal method for image recognition of the present invention and the shadow removal device therefor, it is possible to adaptively operate even in complex image environments such as changes in lighting brightness, complexity of foreground and background, and diversity of colors in an image, and thus more efficient Shadow removal becomes possible.

In addition, various effects other than the above-described effects may be directly or implicitly disclosed in the detailed description according to an embodiment of the present invention to be described later.

1 is a block diagram showing a main configuration of an image recognition apparatus using a shadow removal apparatus according to an embodiment of the present invention.
2 is a block diagram showing the main configuration of a shadow removal apparatus according to an embodiment of the present invention.
3 is a flowchart illustrating a shadow removal method for image recognition according to an embodiment of the present invention.
4 is a flowchart illustrating a shadow removal method for image recognition according to an embodiment of the present invention in more detail.
5 to 8 are flowcharts illustrating a processing process of an algorithm for removing shadows according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, when it is determined that a detailed description of a related known function or configuration may unnecessarily obscure the subject matter of the present invention in describing the operating principle of the preferred embodiment of the present invention in detail, the detailed description thereof will be omitted. This is to more clearly convey the core of the present invention by omitting unnecessary description. In addition, the present invention is intended to illustrate specific embodiments in the drawings and describe in detail in the detailed description, as various changes may be made and various embodiments may be applied, but this is not intended to limit the present invention to specific embodiments, It is to be understood as including all changes, equivalents, and substitutes included in the spirit and scope of the present invention.

In addition, terms including ordinal numbers such as first and second are used to describe various elements, and are only used for the purpose of distinguishing one element from other elements, and to limit the elements. Not used. For example, without departing from the scope of the present invention, a second component may be referred to as a first component, and similarly, a first component may be referred to as a second component.

In addition, when a component is referred to as being "connected" or "connected" to another component, it means that it is logically or physically connected, or can be connected. In other words, it should be understood that a component may be directly connected or connected to another component, but another component may exist in the middle, or may be indirectly connected or connected.

In addition, terms used in the present specification are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In addition, terms such as "comprises" or "have" described in the present specification are intended to designate the existence of features, numbers, steps, actions, components, parts, or a combination thereof described in the specification. It is to be understood that the above other features, or the possibility of the presence or addition of numbers, steps, actions, components, parts, or combinations thereof, are not preliminarily excluded.

Now, a shadow removal method for image recognition and a shadow removal apparatus for the same according to an embodiment of the present invention will be described in detail with reference to the drawings. In this case, the same reference numerals are used for portions having similar functions and functions throughout the drawings, and redundant descriptions thereof will be omitted.

First, an image recognition apparatus using a shadow removal apparatus according to an embodiment of the present invention will be briefly described.

1 is a block diagram showing a main configuration of an image recognition apparatus using a shadow removal apparatus according to an embodiment of the present invention.

Referring to FIG. 1, the shadow removal apparatus 100 according to an embodiment of the present invention may exist as a component of the image recognition apparatus 1000, and the image recognition apparatus 1000 includes an image receiving apparatus 200 and a display. The device 300 and the storage device 400 may be further included.

The image receiving device 200 serves to receive continuous image frames in real time. When the image recognition device 1000 exists as a component of a camera device, the image receiving device 200 receives an image input through a sensor such as a charge coupled device (CCD)/Complementary Metal-Oxide Semiconductor (CMOS). You can play a role. Also, the image receiving device 200 may receive and receive an image frame transmitted from an external device in real time through a communication network (not shown). At this time, the image receiving device 200 may be a wireless communication method such as WLAN (Wireless LAN), Wi-Fi, Wibro, Wimax, High Speed Downlink Packet Access (HSDPA), or Ethernet ( Ethernet), xDSL (ADSL, VDSL), HFC (Hybrid Fiber Coaxial Cable), FTTC (Fiber to The Curb), FTTH (Fiber To The Home), etc. through wired communication network (not shown) can do. In addition, it is also possible to receive an image frame through a communication network (not shown) according to any other well-known or future developed communication method.

The shadow removal apparatus 100 is a device that removes a shadow from an input image frame according to a shadow removal method for image recognition according to an embodiment of the present invention. At this time, the shadow removal device 100 separates each continuously input image frame transmitted through the image receiving device 200 into a foreground area and a background area, and includes an image frame, a foreground area and a background area separated from the image frame. After extracting shadow candidate pixels using at least one analysis method among color and illuminance analysis, vector dot product analysis, boundary distribution analysis, and frame pixel analysis for at least one of the targets, each of the shadow candidate pixels according to the used analysis technique A weight is applied to and a shadow candidate pixel is mixed according to the applied weight to determine a final shadow pixel, and the determined shadow pixel is removed from the foreground area. A more specific configuration and operation method of the shadow removal apparatus 100 will be described later.

The display device 300 may play a role of displaying an image from which a shadow has been removed through the shadow removal device 100. In addition, the display device 300 may play a role of displaying various result screens other than image recognition. The display device 300 of the present invention includes a Liquid Crystal Display (LCD), a Thin Film Transistor LCD (TFT-LCD), an Organic Light Emitting Diodes (OLED), a light emitting diode (LED), an Active Matrix Organic LED (AMOLED), and a flexible display device. It may be composed of a display (flexible display) and a three-dimensional display (3 Dimension). In addition, some of these displays may be configured as a transparent type or a light-transmitting type so that the outside can be seen through them. This may be configured in the form of a transparent display including TOLED (Transparant OLED).

The storage device 400 serves to temporarily or permanently store an input image frame, and may store and manage various pieces of information for removing a shadow from an image frame through the shadow removal device 100. In particular, the storage device 400 according to an embodiment of the present invention may serve to store and record information on a current frame and information on a shadow removal as information on a previous frame. The storage device 400 according to the exemplary embodiment of the present invention includes a flash memory, a hard disk, a multimedia card micro type memory (eg, SD or XD memory, etc.), and RAM. It may be configured to include a storage medium such as (RAM) and ROM.

A processor mounted in the shadow removal apparatus 100 or the image recognition apparatus 1000 may process a program command for executing the method according to the present invention. In one implementation example, the processor may be a single-threaded processor, and in another implementation example, the processor may be a multithreaded processor. Furthermore, the processor may process commands stored on the storage device 400.

In addition, for convenience of description, the image recognition apparatus 1000 of the present invention is described as an example that includes only the image receiving apparatus 200, the shadow removing apparatus 100, the display apparatus 300, and the storage apparatus 400. However, the present invention is not limited thereto, and may be configured as a set of various devices or modules for image recognition. Since various devices or modules for image recognition can use various known methods, detailed descriptions will be omitted.

Hereinafter, the main configuration and operation method of the shadow removal apparatus 100 according to an embodiment of the present invention will be described.

2 is a block diagram showing the main configuration of a shadow removal apparatus according to an embodiment of the present invention.

Referring to FIG. 2, a shadow removal apparatus 100 according to an embodiment of the present invention includes a pre-processing unit 10, a foreground and background separation unit 20, a shadow removing unit 30, and a post-processing unit 40. Can be.

The preprocessor 10 performs a preprocessing process for continuously input image frames. For example, the preprocessor 10 may perform various processes to improve shadow removal performance and image recognition efficiency through noise removal, color correction, gamma correction, etc. existing in the image.

The foreground background separating unit 20 separates the image frame on which the preprocessing process has been performed through the preprocessor 10 into a foreground area and a background area. For example, the foreground background separation unit 20 designates a plurality of nodes having a weight greater than a predetermined criterion among pixels including a predetermined number of Gaussian nodes as background pixels, A foreground pixel within an image frame is determined based on a specified background pixel. In addition, the foreground background separating unit 20 separates the foreground region using the determined foreground pixels. In addition, the foreground background separating unit 20 may generate a background model according to a specific algorithm such as a statistical model or a Gaussian Mixture Model (GMM), and may separate the foreground region and the background region based on the background model.

The separated foreground area includes objects to be recognized and tracked, such as people and cars. However, the foreground area is separated by recognizing shadows as well as objects other than the object to be recognized and tracked. This is because, as described above, the technique of separating the foreground and background areas is based on the degree of pixel change between the current frame and the previous frame. Because of the separation, shadows that move like objects in the image are also generated by recognizing them as part of the object.

Therefore, the process of removing the shadow included in the foreground area must be performed.

As described above, the shadow removal unit 30 removes the shadow included in the foreground area. In particular, the shadow removal unit 30 according to an embodiment of the present invention is a predetermined method for at least one of an original image frame before the foreground and background regions are separated, a foreground region and a background region separated from the image frame. The shadow candidate pixels are extracted by using an algorithm for removing the shadow of. Here, the algorithm for removing the shadow of a predetermined method may be color and illuminance analysis, vector dot product analysis, boundary distribution analysis, and frame pixel analysis algorithm.

The color and illuminance analysis algorithm is an algorithm that can extract shadow candidate pixels according to the color and brightness of the foreground and background areas.In the RGB color system, the color and saturation change between the shadow and the background area is small, but the brightness change is large. Is a method of determining a pixel that is a candidate for a shadow in the foreground region, which is a feature that generally represents a dark color.

The vector dot product analysis algorithm is an algorithm capable of extracting a shadow candidate pixel according to a motion vector of a pixel existing on a window of a predetermined size. The color and illuminance analysis algorithm measures the degree of change of individual pixels and extracts shadow candidate pixels accordingly, while the vector dot product analysis algorithm sets a window of a certain size first, so that each region is a set of individual pixels. It is an algorithm that can complement color and illuminance analysis algorithms by measuring the degree of change and extracting shadow candidate pixels accordingly.

The boundary distribution analysis algorithm is a method that obtains the center of the edge of the foreground region, not the degree of change of the pixel, obtains the shadow direction, and then extracts the candidate pixel of the shadow by assuming a pixel larger than the reference distance as a shadow. The boundary distribution analysis algorithm is an algorithm method that uses the principle that shadows are hidden by an object and occur at a certain distance or more based on the size of the object.

The frame change analysis algorithm corrects pixels of an image according to a pixel change rate of the entire image of the current frame and the previous frame, not the foreground and background regions, and extracts the corrected pixels as shadow candidate pixels.

A detailed description of a method of extracting shadow candidate pixels using an algorithm for shadow removal according to an embodiment of the present invention will be described later, and the shadow removal unit 30 of the present invention performs one or more shadow removal as described above. After extracting the shadow candidate pixels using the algorithm for, a self weight value is applied to each of the extracted shadow candidate pixels according to the used algorithm, and the final shadow pixels are determined by mixing the shadow candidate pixels according to the applied weight, and It will remove the shadow pixels.

More specifically, the shadow removal unit 30 first compares the surrounding environment information of the currently input image frame and the surrounding environment information of the previous image frame, and the shadow extracted according to the algorithm used through the amount of change in the surrounding environment information. A weight is applied to each of the candidate pixels. Here, the amount of change in the surrounding environment information is information that can be analyzed based on a shadow pixel distribution map and a histogram, and may be, for example, a change amount of the intensity and light amount of the shadow. For a more specific example, the size of the current image frame is 4x4, the shadow candidate pixels extracted using the color and illuminance analysis algorithm are (2,1), (2,2), and the vector dot product analysis algorithm is used. The extracted shadow candidate pixels are (2,2), (2,3), the extracted shadow candidate pixels using the boundary distribution analysis algorithm are (3,2), (3,3), and the frame pixel analysis algorithm It is assumed that the shadow candidate pixels extracted by using are (3,2), (3,3), (4,2), (4,3). And as a result of analyzing the amount of change in the surrounding environment information, if the shadow intensity is 20, for example, by checking the threshold of the shadow intensity preset for each algorithm, the threshold of the shadow intensity preset in the frame pixel analysis algorithm is 10 days. In this case, since the shadow intensity is equal to or greater than the threshold value, a weight is given to the frame pixel analysis algorithm. In addition, as a result of analyzing the amount of change in the surrounding environment information, when the amount of change in the amount of light is 100, and the threshold value of the amount of change in the amount of light preset for each algorithm is checked, and the threshold value of the amount of change in the amount of light preset in the vector dot product analysis algorithm is 90, the threshold Since the amount of light change is greater than or equal to the value, a weight is given to the vector dot product analysis algorithm. In addition, a weight is applied to each of the extracted shadow candidate pixels according to the weighted algorithm, and the shadow candidate pixels (2,2), (2,3), (3,) having a higher weight or higher weight than a predetermined criterion. 2), (3,3), (4,2), (4,3) can be determined as final shadow pixels.

The weight for each of these analysis algorithms can be assigned a high weight for the color and illuminance analysis and vector dot product analysis algorithm in a strong shadow environment, depending on the amount of change in surrounding environment information, and in a strong light environment, and frame pixel analysis in a weak shadow environment and weak light environment. The algorithm may have a high weight, and in an ambiguous environment, the boundary distribution analysis algorithm may have a high weight.

On the other hand, in the case of the first image frame in which no previous image frame exists, the accuracy of the analysis of the amount of change in the surrounding environment information may be degraded. In this case, the degree of overlap between shadow candidate pixels extracted according to each analysis algorithm is analyzed, and the number of overlaps The shadow candidate pixel having the highest value may be determined as the final shadow pixel. In the above example, (2,2), (3,2), (3,3) may be determined as the final shadow pixel.

The shadow removal unit 30 determines the final shadow pixel through this process, and removes the determined shadow pixel from the foreground area.

The post-processing unit 40 serves to compensate for loss in the foreground area from which the shadow has been removed through the shadow removal unit 30. In other words, if a shadow is removed in an environment where an object and a shadow are mixed, loss of pixels corresponding to the object may inevitably occur. In the post-processing unit 40, a fill technique for the area due to an error during shadow removal is performed. It is applied to correct the image.

Through this process, the shadow from the image frame can be removed, and the object in the foreground area from which the shadow has been removed can be monitored and tracked more easily.

In the above, the main configuration and operation method of the shadow removal apparatus 100 according to an embodiment of the present invention have been described. However, not all of the components shown through FIG. 2 are essential components, and the shadow removal device 100 may be implemented by more components than the shown components, and the shadow removal device by fewer components. 100 may be implemented. For example, in an embodiment of the present invention, information on each frame and information on a frame from which a shadow has been removed have been described as an example that is stored and managed in the storage device 400 illustrated in FIG. 1. It is not possible and may exist as a component of the shadow removal device 100.

In addition, although the elements constituting the shadow removal apparatus 100 of the present invention are arranged according to the flow of the shadow removal sequence, it should be clarified that the present invention is not limited thereto.

Although the present specification and drawings describe exemplary device configurations, the functional operations and implementations of the subject described in this specification may be implemented with other types of digital electronic circuits, or the structures disclosed herein and structural equivalents thereof. It may be implemented with computer software, firmware, or hardware, or may be implemented by a combination of one or more of them. Implementations of the subject matter described in this specification are one or more computer program products, that is, one relating to computer program instructions encoded on a tangible program storage medium for execution by or for controlling the operation of a device according to the invention. It can be implemented as the above module. The computer-readable medium may be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of materials that affect a machine-readable radio wave signal, or a combination of one or more of them.

Hereinafter, a method of removing a shadow for image recognition according to an embodiment of the present invention will be described.

3 is a flowchart illustrating a shadow removal method for image recognition according to an embodiment of the present invention.

Referring to FIG. 3, when a continuous image frame is input, the shadow removal apparatus 100 of the present invention separates a current image frame into a foreground region and a background region (S110). At this time, the shadow removal apparatus 100 may analyze the characteristic change of the pixel according to the motion in each image frame to separate the image frame into a foreground region and a background region, and separate image frames into a foreground region and a background region. Various known configurations can be applied.

At this time, since the separated foreground area may include a shadow as well as an object to be recognized, color and illumination analysis, vector dot product analysis, targeting the image frame, foreground area, and background area, in order to remove the shadow from the foreground area. Analysis is performed using an algorithm for removing one or more shadows of boundary distribution analysis and frame pixel analysis (S120). A shadow candidate pixel is first extracted according to the algorithm for shadow removal (S130), a weight is applied for each shadow candidate pixel according to the used analysis technique, that is, the algorithm (S140), and a final shadow pixel is determined (S150). , It is possible to remove the determined final shadow pixel from the foreground area (S160).

Hereinafter, a shadow removal method for image recognition according to an embodiment of the present invention will be described in more detail with reference to FIGS. 4 to 8.

4 is a flowchart illustrating a shadow removal method for image recognition according to an embodiment of the present invention in more detail, and FIGS. 5 to 8 illustrate a processing process of an algorithm for shadow removal according to an embodiment of the present invention. This is a flow chart for

First, referring to FIG. 4, when a continuous image frame is received through the image receiving device 100 shown in FIG. 1 (S200), the shadow removal device 100 further enhances the efficiency of shadow removal and image recognition. First, a pre-processing process is performed (S300). For example, the shadow removal apparatus 100 may perform noise removal, color correction, gamma correction, and the like existing in an image.

In addition, the shadow removal apparatus 100 separates the image frame into a foreground region and a background region by analyzing a change in characteristics of each pixel according to movement in the image frame on which the preprocessing process has been performed (S400). For example, the shadow removal apparatus 100 designates a plurality of nodes having a weight greater than a predetermined criterion among pixels including a predetermined number of Gaussian nodes as background pixels, and After determining the foreground pixel in the image frame based on the background pixel, the foreground region is separated using the determined foreground pixel. In addition, the shadow removal apparatus 100 may generate a background model according to a specific algorithm such as a statistical model or a Gaussian Mixture Model (GMM), and may separate the foreground region and the background region based on the background model.

The separated foreground area includes objects to be recognized and tracked, such as people and cars. However, the foreground area is separated by recognizing shadows as well as objects other than the object to be recognized and tracked. This is because, as described above, the technique of separating the foreground and background areas is based on the degree of pixel change between the current frame and the previous frame. Because of the separation, shadows that move like objects in the image are also generated by recognizing them as part of the object.

Therefore, the process of removing the shadow included in the foreground area must be performed. To this end, first, a pre-processing process for the foreground region such as binarization or noise removal for the foreground region may be performed (S500). And, the foreground area in which the pre-processing process including binarization has been performed, the background area separated from the image frame, and the image frame after the pre-processing process corresponding to the original are analyzed using an algorithm for shadow removal in a predetermined manner. Then, shadow candidate pixels are extracted accordingly (S600 to S700). Here, the algorithm for removing the shadow of a predetermined method may be color and illuminance analysis, vector dot product analysis, boundary distribution analysis, and frame pixel analysis algorithm.

First, the color and illuminance analysis algorithm is an algorithm that can extract shadow candidate pixels according to the color and brightness of the foreground and background areas.In the RGB color system, the amount of change in color and saturation between the shadow and the background area is small, but the amount of change in brightness is large. Shadows are generally dark colors and are a method of determining pixels that are candidates for shadows in the foreground area.

This will be described with reference to FIG. 5, in order to extract shadow candidate pixels according to the color and illuminance analysis algorithm, first, the shadow removal device 100 uses the RGB color coordinate system as the HSV color coordinate system for the foreground area and the background area. Convert (S610). In addition, the shadow removal apparatus 100 detects a dark area based on a brightness value in the converted foreground area and background area (S611).

In addition, the shadow removal apparatus 100 first extracts a pixel having a predetermined threshold value or higher as a shadow candidate pixel through a saturation comparison for a dark area in the foreground area and a dark area in the background area (S612). In this case, the shadow removal apparatus 100 may first extract a shadow candidate pixel by comparing hue and saturation values according to the histogram comparison analysis.

Thereafter, the shadow removal apparatus 100 analyzes the amount of change in brightness targeting the shadow candidate pixels first extracted from the background area and the foreground area (S613), and selects a pixel whose brightness change amount is within the preset value as a final shadow candidate pixel. It is extracted with (S614).

On the other hand, the vector dot product analysis algorithm is an algorithm capable of extracting a shadow candidate pixel according to a motion vector of a pixel existing on a window of a predetermined size. The color and illuminance analysis algorithm measures the degree of change of individual pixels and extracts shadow candidate pixels accordingly, while the vector dot product analysis algorithm sets a window of a certain size first, so that each region is a set of individual pixels. It is an algorithm that can complement color and illuminance analysis algorithms by measuring the degree of change and extracting shadow candidate pixels accordingly.

A process of extracting a shadow candidate pixel using a vector dot product analysis algorithm will be described with reference to FIG. 6. First, the shadow removal apparatus 100 sets a window of a predetermined size for the background area and the foreground area. Do (S620). Here, the predetermined size may be changed according to system performance such as the resolution of the display device. It can also be set directly by the user. Then, the inner product of the vector values of the pixel in the window in the background area and the pixel in the window in the foreground area is calculated (S621). For example, a vector value of a pixel is calculated from the amount of change, such as an average, a maximum value, and a minimum value, of a pixel in a set window. The shadow removal apparatus 100 compares the dot product in the background area with the dot product in the foreground area (S622), and extracts a pixel with a change amount of a vector value equal to or greater than a preset threshold as a shadow candidate pixel (S623).

On the other hand, the boundary distribution analysis algorithm is a method that obtains the center of the edge of the foreground region rather than the degree of change of the pixel, obtains the shadow direction, and then extracts the candidate pixel of the shadow by assuming a pixel equal to or greater than the reference distance as a shadow. The boundary distribution analysis algorithm is an algorithm method that uses the principle that shadows are hidden by an object and occur at a certain distance or more based on the size of the object.

A process of extracting a shadow candidate pixel using a boundary distribution analysis algorithm will be described with reference to FIG. 7. First, the shadow removal apparatus 100 generates a vertical histogram for a foreground area (S630). Then, the range of the x-axis having the highest value is set as the central axis by using the generated vertical histogram (S631). Here, the central axis is used as a criterion for determining the direction of the shadow. In general, the head position can be the central axis in the case of a person, so the range of the x-axis having the highest value in the vertical histogram is the central axis. Is set to.

Also, since the highest value of the vertical histogram may change depending on the situation, the shadow removal apparatus 100 may divide the vertical histogram into a predetermined number of blocks and determine a block area having a high average histogram value for each block as a central axis.

Subsequently, the shadow removal apparatus 100 predicts the direction of the shadow by analyzing the pixel brightness value within a certain radius based on the central axis (S632), and when the shadow direction is predicted, a pixel of a predetermined distance or more from the central axis to the shadow direction. It is extracted as a shadow candidate pixel (S633). Here, a certain distance in the shadow direction may be set in consideration of the size of a person. In this case, the user may directly set it or set automatically in consideration of the size of the person.

On the other hand, the frame change analysis algorithm corrects the pixels of an image according to the pixel change rate of the entire image of the current frame and the previous frame, not the foreground area and the background area, and extracts the corrected pixels as shadow candidate pixels.

Referring to FIG. 8, a shadow candidate pixel extraction process according to a frame change analysis algorithm according to an embodiment of the present invention will be described with reference to FIG. 8, in which the shadow removal apparatus 100 is firstly used as an original image frame before the foreground and background regions are separated. The lighting component is extracted from (S640). In this case, the shadow removal apparatus 100 may extract the lighting component using a low-pass filter. In addition, the shadow removal apparatus 100 may generate a reflection component by using the lighting component (S641). In this case, the shadow removal apparatus 100 generates a reflection component by calculating a logarithmic operation on each of the image frame and the extracted lighting component and then calculating the difference therebetween. Then, the shadow removal apparatus 100 corrects the dark area using the value of the reflection component (S642), and extracts a pixel whose dark circle area is corrected as a shadow candidate pixel (S643).

Referring back to FIG. 4, when a shadow candidate pixel is extracted through the above-described process (S700), the shadow removal apparatus 100 determines a final shadow pixel (S800). At this time, if there is one algorithm applied to the shadow candidate pixel extraction, all of the extracted shadow candidate pixels can be determined as the final shadow pixels, and if there are two or more algorithms applied to the shadow candidate pixel extraction, the shadow extracted according to each algorithm A weight may be applied to each of the candidate pixels, and only a specific shadow candidate pixel may be determined as a final shadow pixel according to the weight.

More specifically, the shadow removal apparatus 100 first compares the surrounding environment information of the currently input image frame and the surrounding environment information of the previous image frame, and the shadow extracted according to the algorithm used through the amount of change in the surrounding environment information. A weight is applied to each of the candidate pixels. Here, the amount of change in the surrounding environment information is information that can be analyzed based on a shadow pixel distribution map and a histogram, and may be, for example, a change amount of the intensity and light amount of the shadow. For a more specific example, the size of the current image frame is 4x4, the shadow candidate pixels extracted using the color and illuminance analysis algorithm are (2,1), (2,2), and the vector dot product analysis algorithm is used. The extracted shadow candidate pixels are (2,2), (2,3), the extracted shadow candidate pixels using the boundary distribution analysis algorithm are (3,2), (3,3), and the frame pixel analysis algorithm It is assumed that the shadow candidate pixels extracted by using are (3,2), (3,3), (4,2), (4,3). And as a result of analyzing the amount of change in the surrounding environment information, if the shadow intensity is 20, for example, by checking the threshold of the shadow intensity preset for each algorithm, the threshold of the shadow intensity preset in the frame pixel analysis algorithm is 10 days. In this case, since the shadow intensity is equal to or greater than the threshold value, a weight is given to the frame pixel analysis algorithm. In addition, as a result of analyzing the amount of change in the surrounding environment information, when the amount of change in the amount of light is 100, and the threshold value of the amount of change in the amount of light preset for each algorithm is checked, Since the amount of light change is greater than or equal to the value, a weight is given to the vector dot product analysis algorithm. Then, a weight is applied to each of the extracted shadow candidate pixels according to the weighted algorithm, and the shadow candidate pixels with high weight (2,2), (2,3), (3,2), (3,3) , (4,2), (4,3) can be determined as final shadow pixels.

The weight for each of these analysis algorithms can be assigned a high weight for the color and illuminance analysis and vector dot product analysis algorithm in a strong shadow environment, depending on the amount of change in surrounding environment information, and in a strong light environment, and frame pixel analysis in a weak shadow environment and weak light environment. The algorithm may have a high weight, and in an ambiguous environment, the boundary distribution analysis algorithm may have a high weight.

On the other hand, in the case of the first image frame in which no previous image frame exists, the accuracy of the analysis of the amount of change in the surrounding environment information may be degraded. In this case, the degree of overlap between shadow candidate pixels extracted according to each analysis algorithm is analyzed, and the number of overlaps The shadow candidate pixel having the highest value may be determined as the final shadow pixel. In the above example, (2,2), (3,2), (3,3) may be determined as the final shadow pixel.

The shadow removal apparatus 100 determines a final shadow pixel through such a process, and removes the determined shadow pixel from the foreground area (S900). The foreground area, background area, and the original image frame from which the shadow pixels have been removed are stored as information on the previous frame (S1000), so that they can be used in the final shadow pixel extraction process in the next frame.

Thereafter, the shadow removal apparatus 100 serves to compensate for the loss in the foreground area from which the shadow has been removed. In other words, if the shadow is removed in an environment where an object and a shadow are mixed, loss of pixels corresponding to the object may inevitably occur. The shadow removal device 100 uses a method of filling the area due to an error during shadow removal. The image is corrected by applying (S1100). Then, the result of the foreground area from which the shadow is removed is transmitted to the display device 300 illustrated in FIG. 1 and outputted (S1200).

In the above, a shadow removal method for image recognition according to an embodiment of the present invention has been described.

The shadow removal method for image recognition of the present invention as described above may be provided in the form of a computer-readable medium suitable for storing computer program commands and data. A program recorded on a recording medium for implementing a shadow removal method for image recognition according to an embodiment of the present invention comprises the steps of separating each continuously input image frame into a foreground region and a background region, the image frame, and the image Extracting shadow candidate pixels using an algorithm for removing one or more shadows for the foreground and background areas separated from the frame, applying a weight to each of the extracted shadow candidate pixels according to the used algorithm, and applying a weight According to the method, a final shadow pixel may be determined by mixing shadow candidate pixels, and a step of removing the determined shadow pixel from the foreground area may be performed.

At this time, the program recorded on the recording medium can be read, installed, and executed by a computer to execute the above-described functions.

Here, in order for the computer to read the program recorded on the recording medium and execute the functions implemented as the program, the above-described program is C, C++, which can be read by the computer's processor (CPU) through the computer's device interface. It may include code coded in a computer language such as JAVA and machine language.

This code may include a functional code related to a function defining the above-described functions, and may include a control code related to an execution procedure necessary for the processor of the computer to execute the above-described functions according to a predetermined procedure. In addition, these codes may further include additional information necessary for the processor of the computer to execute the above-described functions, or code related to a memory reference to which location (address address) of the internal or external memory of the computer should be referenced. . In addition, when the computer's processor needs to communicate with any other computer or server in a remote in order to execute the above-described functions, the code is used by the computer's processor using the computer's communication module. It may further include a communication-related code for how to communicate with other computers or servers, and what information or media should be transmitted and received during communication.

Such computer-readable media suitable for storing computer program commands and data include, for example, magnetic media such as hard disks, floppy disks, and magnetic tapes, and compact disk read only memory (CD-ROM). , Optical Media such as DVD (Digital Video Disk), Magneto-Optical Media such as Floptical Disk, and ROM (Read Only Memory), RAM (RAM) , Random Access Memory), flash memory, EPROM (Erasable Programmable ROM), and EEPROM (Electrically Erasable Programmable ROM). The processor and memory can be supplemented by special purpose logic circuits or incorporated into it.

In addition, the computer-readable recording medium is distributed over a computer system connected through a network, so that computer-readable codes can be stored and executed in a distributed manner. In addition, a functional program for implementing the present invention, and related codes and code segments, etc., are programmers in the technical field to which the present invention pertains in consideration of the system environment of a computer that reads the recording medium and executes the program. May be easily inferred or modified by

While this specification includes details of a number of specific implementations, these should not be construed as limiting to the scope of any invention or claim, but rather as a description of features that may be peculiar to a particular embodiment of a particular invention. It must be understood. Certain features described herein in the context of separate embodiments may be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination. Furthermore, although features operate in a particular combination and may be initially described as so claimed, one or more features from a claimed combination may in some cases be excluded from the combination, and the claimed combination may be a subcombination. Or sub-combination variations.

Likewise, although operations are depicted in the drawings in a specific order, it should not be understood that such operations must be performed in that particular order or sequential order shown or that all illustrated operations must be performed in order to obtain a desired result. In certain cases, multitasking and parallel processing can be advantageous. In addition, separation of the various system components of the above-described embodiments should not be understood as requiring such separation in all embodiments, and the program components and systems described are generally integrated together into a single software product or packaged in multiple software products. You should understand that you can.

The present invention relates to a shadow removal method for image recognition, and more particularly, a shadow candidate pixel is extracted using an algorithm for removing one or more shadows for consecutive image frames, and then extracted according to the used algorithm. A shadow removal method for image recognition capable of removing the determined shadow pixel after determining a final shadow pixel according to the applied weight by applying a weight according to the analysis technique used to each shadow candidate pixel, and a shadow removal apparatus therefor. .

According to the present invention, it is possible to adaptively remove shadows in various video environments, and thus can be used in video security/surveillance systems, traffic analysis and control systems, and video analysis/recognition devices of military systems.

In addition, the present invention is not only sufficiently commercially available or commercially possible, but also has industrial applicability because it can be implemented clearly in reality.

10: preprocessing unit 20: foreground and background separation unit
30: shadow removal unit 40: post-processing unit
100: shadow removal device 200: image receiving device
300: display device 400: storage device
1000: image recognition device

Claims (12)

Separating each continuously inputted image frame into a foreground area and a background area;
Extracting shadow candidate pixels using an algorithm for removing one or more shadows from the image frame, a foreground region and a background region separated from the image frame;
Applying a weight to each of the extracted shadow candidate pixels according to the used algorithm and mixing the shadow candidate pixels according to the applied weight to determine a final shadow pixel; And
Including; removing the determined shadow pixel from the foreground area,
Determining the final shadow pixel,
Analyzing information about the surrounding environment of the current image frame;
By comparing the surrounding environment information of the current image frame with the surrounding environment information of the previous image frame, analyzing the amount of change in the surrounding environment information, and weighting each of the extracted shadow candidate pixels according to the algorithm used based on the change amount of the surrounding environment information Applying; And
Mixing the shadow candidate pixels according to the applied weight to determine a final shadow pixel;
Shadow removal method for image recognition comprising a. ◈ Claim 2 was abandoned upon payment of the set registration fee. The method of claim 1,
The algorithm for removing the shadow is
A shadow removal method for image recognition, comprising color and illuminance analysis, vector dot product analysis, boundary distribution analysis, and frame pixel analysis. ◈ Claim 3 was abandoned upon payment of the set registration fee. The method of claim 2,
Extracting a shadow candidate pixel using the color and illuminance analysis algorithm
Converting the RGB color coordinate system of the foreground area and the background area into an HSV color coordinate system;
Detecting a dark area based on a brightness value in the converted foreground area and background area;
Firstly extracting a pixel having a predetermined threshold value or higher as a shadow candidate pixel through a saturation comparison of the dark area of the foreground area and the dark area of the background area;
Analyzing an amount of change in brightness for the shadow candidate pixels first extracted from the background area and the foreground area; And
Extracting a pixel whose brightness change amount is less than the preset threshold as a final shadow candidate pixel;
Shadow removal method for image recognition comprising a. ◈ Claim 4 was abandoned upon payment of the set registration fee. The method of claim 2,
Extracting the shadow candidate pixel using the vector dot product analysis algorithm
Setting a window of a predetermined size for the background area and the foreground area;
Calculating an inner product of a vector value of a pixel in the window in the background area and a pixel in the window in the foreground area; And
Comparing the dot product in the background area with the dot product in the foreground area, and extracting a pixel with a change amount equal to or greater than a preset threshold as a shadow candidate pixel;
Shadow removal method for image recognition comprising a. ◈ Claim 5 was abandoned upon payment of the set registration fee. The method of claim 2,
Extracting the shadow candidate pixel using the boundary distribution analysis algorithm
Generating a vertical histogram for the foreground area;
Setting a range of the x-axis having the highest value as a central axis using the vertical histogram;
Predicting a direction of a shadow by analyzing a pixel brightness value within a predetermined radius based on the central axis; And
If a shadow direction is predicted, extracting a pixel of a predetermined distance or more from the central axis in the shadow direction as a shadow candidate pixel;
Shadow removal method for image recognition comprising a. ◈ Claim 6 was abandoned upon payment of the set registration fee. The method of claim 5,
The step of setting as the central axis
The method for removing shadows for image recognition, comprising dividing the vertical histogram into a predetermined number of blocks and determining a block area having the highest average histogram value for each block as a central axis. ◈ Claim 7 was abandoned upon payment of the set registration fee. The method of claim 2,
Extracting the shadow candidate pixel using the frame pixel analysis algorithm
Extracting a lighting component of the image frame;
Generating a reflection component using the lighting component;
Correcting a dark area using the value of the reflection component; And
Extracting a pixel corresponding to the corrected dark area as a shadow candidate pixel;
Shadow removal method for image recognition comprising a. delete ◈ Claim 9 was abandoned upon payment of the set registration fee. The method of claim 1,
The amount of change in the surrounding environment information is
A shadow removal method for image recognition, comprising a shadow pixel distribution diagram and an amount of change in intensity and amount of light analyzed based on a histogram. ◈ Claim 10 was abandoned upon payment of the set registration fee. The method of claim 1,
Determining the final shadow pixel
If there is no previous video frame,
A shadow removal method for image recognition, comprising: determining whether or not the shadow candidate pixels extracted according to each algorithm overlap, and determining the shadow candidate pixel having the largest overlapping degree as a final shadow pixel. ◈ Claim 11 was abandoned upon payment of the set registration fee. A computer-readable recording medium on which a program for executing the shadow removal method for image recognition according to any one of claims 1 to 7, 9 and 10 is recorded. A foreground background separating unit for separating each continuously input image frame into a foreground area and a background area; And
For the image frame, a foreground region and a background region separated from the image frame, a shadow candidate pixel is extracted using an algorithm for removing one or more shadows, and a weight is applied to each of the extracted shadow candidate pixels according to the used algorithm. A shadow removal unit configured to determine a final shadow pixel by applying and mixing shadow candidate pixels according to the applied weight, and removing the determined shadow pixel from the foreground area; Including,
The shadow removal unit,
By comparing the surrounding environment information of the current image frame and the surrounding environment information of the previous image frame, the amount of change in the surrounding environment information is analyzed, and a weight is applied to each of the shadow candidate pixels extracted according to the algorithm used based on the change amount of the surrounding environment information. And determine a final shadow pixel by mixing the shadow candidate pixels according to the applied weight.

Images