TWI451342B - Shadow Removal Method in Mobile Light Source Environment - Google Patents

Description

Shadow removal method in moving light source environment

The invention relates to a shadow removing method in a moving light source environment, in particular to removing a shadow generated by a moving light source on a foreground object in a nighttime environment, so as to improve the accuracy of object detection.

The "smart living environment" has always been the dream of mankind. It has simulated various human perceptions to construct a smart environment. It has also become a highly forward-looking technical direction in recent years. The research goal of smart technology is to increase the perception of the real world by the system or equipment, and to use the wisdom of experts in various fields to sense the environmental conditions and then actively respond or change the environment to serve, inform or remind humans. Achieve the establishment of a smart environment. "Computer Vision" plays a very important role in the development of intelligent environments. It is an important key to environmental intelligence to capture surrounding environmental images through image capture devices and develop related software systems to simulate the perception and response of human vision. The technical requirements of visual security monitoring can be divided into three categories: "moving object detection and tracking", "action analysis" and "behavior understanding". If the foreground object in the screen image can be reliably separated from the background, it will be of great help for subsequent tracking, motion analysis and behavioral understanding. Although there are many research results for foreground object detection, most studies focus on different methods to detect foreground objects in various environments, and occasionally propose object removal methods for object shadows caused by ambient light.

In the discussion of object shadow detection and removal technology, it can be roughly divided into the following types of methods:

1. Pixel projection method: Project all foreground image points vertically and horizontally to find the intersection of the bottom of the object and the shadow. The disadvantage of this method is that you need to know the direction of the light source and only be able to handle upright objects, and the shadows must be projected on the ground.

2. Area brightness analysis method: divide the image into several blocks, calculate the brightness average value for each block and sort according to the size, take the median as the threshold value to distinguish the shadow, to remove the shadow block; The disadvantage of the method is that the light source and the background must have stable brightness respectively.

3. Histogram statistics method: Using the brightness distribution statistics, the possible foreground pixels are further distinguished into shadow pixels, foreground object pixels or changed to background pixels by probability. The disadvantage of this method is that it must be assumed that the shadow is projected on a flat road surface and that the relative position of the light source to the camera needs to be known.

4. Color invariant method: Consider the brightness and chroma of the foreground bright color, and the degree of distortion in the HSV or RGB color space, define several threshold values, and subdivide the possible foreground image into foreground, background, glare and shadow. Four categories. This method requires that the color of the light source be white, and that shadows and non-shadows have similar chromaticities; for shadows in dark nights, it is easy to make false positives.

5. According to the physics theory method: according to the physical model of illuminance and reflection, use the space-time albedo discrimination method to distinguish the detected sub-regions from shadows or objects themselves. This method is suitable for stable ambient light source brightness and requires additional processing of glare and mirror surface reflection.

The most important difference between the nighttime environment and the daytime environment is that the brightness or color difference of the foreground image and the background environment is less obvious, the influence of the shadow is more serious, and it is susceptible to the drastic changes of the light source. The outdoor environment at night, because of the illumination of the vehicle in the vehicle, the rapid change of the light quickly causes the shadow to change rapidly, which increases the difficulty of the shadow removal, and also shows the technical advantage of the present invention.

For the prior art of shadow removal in a moving light source environment, please refer to patents TW I298857, TW I323434, TW I250466, TW I323434, TW I298857, TW I220969, TW201002073, TW201025189, TW201025193, TW201025198, TW201019268, TW201021574, TW201026081, TW201001338. Patent cases such as TW200912772, US5402118, US5548659, US6259802, US6950123, US7199821, US2003/0123703, US2006/0285723, US 2007/0127774.

The inventor of the case also published "Brightness Measurement and Object Shadow Removal in Multi-Light Source Environment" in July 2009, and then at the 2009 National Computer Conference in November 2009, "Shadow Model Establishment and Prospect Objects in Multi-Light Source Environment" Shadow removal is used as the basis for constructing the "original background model" in the training phase of the present invention, so that the present invention is based on the original background model, dynamically simulating the background model of the scene according to the illumination characteristics of the current picture, and is free from foreground objects. Affects and improves the detection of foreground pixels, as well as the judgment of shadow pixels.

The main object of the present invention is to provide a method for determining and removing the shadow of a foreground object when the video screen has a variable and irregular illumination area for the nighttime monitoring environment under the illumination of the moving light source, so as to reduce the influence of the shadow on the object detection. Improve the application and effectiveness of video surveillance systems in the night environment.

The second object of the present invention is to provide a method for detecting the foreground object by using a camera, and to determine the accuracy of the object position detection, and to analyze the current picture condition and dynamically adjust the original background model to become the current background model.

Another object of the present invention is to provide a method for constructing a shadow model by analyzing shadow color characteristics of different light-emitting objects by analyzing different color light sources in a nighttime environment.

A further object of the present invention is to provide a method for automatically detecting and analyzing the number, location and color distribution characteristics of illumination regions in a surveillance picture for a nighttime environment illuminated by a moving light source.

A further object of the present invention is to provide a method for simulating the background of the current situation. Based on the original background model trained in advance, the illumination region is automatically simulated in the original background model by using the illumination region characteristic of the current scene. Current background model.

Another object of the present invention is to provide a pixel-based shadow removal method, which uses the simulated current background model to first detect foreground pixels in a video image, and then matches the relative shadow according to the characteristics of the illumination region. The model can determine the shadow pixels and remove them.

The method for removing shadows in a moving light source environment that achieves the above object can include both a conventional training phase and a test phase of the present invention. In the training phase, firstly, from a pre-recorded video, draw enough image images, perform color distribution statistics of pixels at the same position, and record the color average and variation of each position to construct the original background model; And light sources of different light transmittances are respectively illuminated by light sources of common colors (such as white and yellow), the shaded areas are extracted, and the distribution of shadow pixel colors is counted and recorded as a shadow model.

In the test phase, first analyze the brightness of the pixels in the video image, find several bright extreme points, and then use the diffusion search method to find the number and position of the illumination area, and record the color distribution of each area; then, use the illumination area characteristics ( Quantity, position and color distribution), based on the original background model, automatically simulates the same number, the same area position, and similar color distribution of the illumination area through the gradient rendering method to generate the current background model, and then the current background The model performs foreground pixel detection; the obtained foreground pixel is further transmitted through the shadow model, and the qualified foreground pixels are regarded as shadows and removed; the remaining foreground pixels can be combined into foreground objects to obtain a more accurate foreground object detection result.

Referring to FIG. 2, the method for removing shadows in a moving light source environment provided by the present invention includes the following steps: a screen image extreme point detecting step, a lighting region range determining step, a lighting region characteristic analyzing step, a simulated lighting region step, and a revision background. The model step, the foreground pixel detection step, the shadow pixel determination and removal step, and the foreground object combination and correction steps are formed.

The main purpose of the invention is to remove the shadow generated by the moving light source on the foreground object in the nighttime environment, so as to improve the accuracy of the object detection. The photographic device used is fixed, and does not consider the influence of illumination of complex light sources such as other neon lights; therefore, in the object detection method, the background subtraction method is adopted, which is fast in processing, simple and easy to use, and the obtained foreground object The effect is also good.

The background subtraction method must construct the original background model and the shadow model in advance. The construction method was published at the 2009 National Computer Conference. "Shadow model creation and shadow removal of foreground objects in a multi-light environment", as shown in the figure. As shown in Annexes I (A) to (B), in order to capture an unselected and certain number of images (approximately tens to hundreds of images) from a pre-recorded video, and calculate The pixel color distribution of the same image position is recorded, and the color average and the variance of each position are recorded to establish an initial background model. In addition, the objects of different light transmittances are respectively illuminated by light sources of common colors (such as white and yellow), the shaded areas are extracted, and the distribution of the color of the shadow pixels is counted and recorded as a shadow model.

Because the nighttime environment is dim, each light source causes obvious and multiple shadows on the foreground object, which seriously affects the accuracy of object detection. Since the number, orientation and color of the light source can produce different degrees of shadow features on the object; the present invention targets the white and yellow light sources indoors and outdoors at night, proposes a shadow model in a multi-light source environment, and applies the moving object in the night environment. Detection. The step of establishing a shadow model is to establish a shadow model by analyzing the shadow features of each light source on different light-transmitting objects in a multi-light source environment, so as to apply to the moving object detection in the night environment, the color of different light sources (the number and orientation of the light sources) And the color value) will produce a different degree of shadow characteristics to the object, in order to establish different shadow features, a pure white correction paper will be used for different translucent objects (opaque objects, semi-transparent objects, all-transparent objects) Placed in the center of the light area to capture the image of the white light environment (such as Annex II (A) ~ (C)) and the yellow light environment (such as Annex III (A) ~ (C)), and analyze the shadow characteristics of the object To create a shadow model; white light for opaque objects (such as Annex II (A)) as an example, object pixels (such as Annexes (A) ~ (C)) and shadow pixels (such as Annex V (A) ~(C)) In the distribution of RGB color models, there are obvious differences, and the appropriate threshold can be set as the basis for distinguishing two types of pixels; however, opaque objects under yellow light (such as Annex III ( A)), in the RGB color model, object pixels (such as attachments six (A) ~ (C)) and shadows Hormone (such as Annex VII (A) ~ (C)) is not easy distribution segment, must be converted to a YC _b C _r color model, and to provide for multiple threshold embodiment, the object pixel segment (e.g., Annex VIII (A) ~ (C)) and shadow pixels (such as Annexes IX (A) ~ (C)), by analyzing the shadow characteristics of each light source on different translucent objects, you can create (Type _light , Type _perv , ClrInfo _shadow ) Among them, Type _light is the color mode of the light source, Type _perv is the light transmissive type of the object, and ClrInfo _shadow is the color characteristic of the shadow pixel.

As shown in FIG. 2, the present invention firstly detects the video image by using the original background model and the shadow model established in advance, and then analyzes the brightness of the pixels in the video image, finds several bright extreme points, and then finds the diffusion search method. The number and location of the illuminated areas, and record the color distribution of each area. Then, using the characteristics of the illumination area (number, position and color distribution), based on the original background model, through the gradient rendering method, automatically simulate the same number, the same area position, and similar color distribution of the illumination area, resulting in the current background The model is then used to detect foreground pixels using this current background model. The resulting foreground pixels are then passed through the shadow model, and the eligible foreground pixels are treated as shadows and removed. The remaining foreground pixels can be combined into a foreground object to obtain a more accurate foreground object detection result.

As shown in Figure 2 and Annex X (A) ~ (B), the image image extreme point detection step is to calculate the brightness distribution of all the pixels in the picture image, and find the brightest trough as the threshold value to find the relative The bright pixels are extreme points; the system automatically detects the video image and analyzes the extreme points in the image. Because the illumination area in the image has brightness concentration and gradual persistence, all the image images are calculated first. The difference in brightness distribution of the pixel is the threshold value of the brightest trough in the distribution map, and the brightest trough in the picture image is used as the threshold value, thereby finding the relative bright pixels as extreme points, and setting each extreme point as The center of the light area.

As shown in Annexes (C)~(D), the illumination area range determination step is to use the extreme points of the image image as the diffusion seed, and the detection and detection aspect is used to perform the automatic judgment and recognition of the illumination area by the diffusion search method to calculate the The range of the illumination area in the image is the diffusion point of the center of the illumination area, so that the outer neighbor points are searched out layer by layer, and the brightness difference from the center of the extreme point is determined. If the difference is too large, it belongs to the edge of the illumination area. Point location. Finally, the characteristic information of the extreme center point, the edge point position in each direction, and the color value are recorded separately.

The simulated illumination area step is to automatically simulate the illumination area in a gradient rendering manner to modify the background model by using the background model. Based on the background model, the same number, the same area position, and the like are automatically simulated by the gradient layer rendering method. The color distribution of the illumination area, and then automatically generate the current background model; in order to simulate the occurrence of each illumination area in the background model to conform to the background model of the current situation of the monitoring environment, the present invention adopts a gradient rendering method to simulate the illumination area. Join the background model. Firstly, according to the distance between the center of the illumination region and the position of the edge point and the color difference value, the respective variation ratios of the three primary colors of RGB (DiffRatR, DiffRatG, DiffRatB) are respectively calculated; then, in the background model image, starting from the center position of the illumination region, The layer is gradually increased by the RGB value according to the distance from the center of the illumination area, and the pixel value of the position is brightened (as shown in Annex XI (A) ~ (C)). The background model of the simulated illumination is used to detect the foreground object of the current picture; as shown in Annexes 12(A)-(C), the foreground object detection step is a current background model using simulated illumination, and the foreground pixels are subtracted by the background subtraction method. Detection; the present invention is directed to each video picture, first through the characteristics of the illumination area, through the layer-by-layer rendering simulation mode, the relative position of the illumination area is added to the original background model, generating a current background model, and for foreground pixel detection The measurement can greatly reduce the interference caused by the illumination area.

As shown in Annexes 13 (A) to (D), the shadow pixel determination and removal step is a current background model using simulated illumination, and the foreground pixels obtained by the background subtraction method are generated by previously establishing different light source colors. The shadow model allows for the determination and removal of shadow pixels for all foreground pixels. First, each foreground pixel is compared with the color of the background pixel at the same position. If the two conditions satisfying that the color is darker than the background and the darkening range of each color component is close, the foreground image of the position is determined as a shadow pixel. Removing; the foreground component combination and correction step, for the remaining foreground pixels after removing the shadow pixels, firstly connecting the foreground pixels to the foreground components through the connected component detection method, and then according to the size and spacing of the components The feature, the correction of the noise component removal and the similar component combination, has completed the detection result of the final foreground object.

The method for removing shadows in the mobile light source environment provided by the present invention has the following advantages when compared with the foregoing cited documents and other conventional techniques:

1. The invention is based on the background model, dynamically adds the illumination region to the background model according to the characteristics of the illumination region of the current picture image, so as to obtain the background model, which is not affected by the foreground image and is improved. The detection effect of the foreground image and the judgment result of the shadow pixel. For the nighttime video surveillance system, the present invention can provide more accurate foreground image detection effects to improve the performance of subsequent moving object tracking and behavior analysis.

2. The present invention captures the nighttime environment through a general photographic device, and even if there is a moving light source, the background model of the background environment can be dynamically simulated by analyzing the characteristics of the illumination region in the video image, and the shadow model of different light source colors is matched. Shadow pixels can be removed in a large amount to improve the accuracy of object detection. Applying this invention will increase the market share and product competitive advantage for the security video surveillance industry.

In summary, this case is not only innovative in research methods and practical applications, but also can enhance the above-mentioned multiple functions compared with the existing methods. It should fully comply with the statutory invention patent requirements of novelty and progress, and apply for it according to law. You have approved this invention patent application, in order to invent invention, to the sense of virtue.

Figure 1 shows the process of establishing a shadow model training;

Figure 2 shows the process of detecting the foreground object in a changing light source environment;

Annexes (A) to (B) are the original background models established by a certain number of screen images, (A) for a certain number of screen images, and (B) for the original background model that is completed;

Annex 2 (A) ~ (C) is to establish a shadow model by simulating the illumination of different color light sources, with different light transmissive objects ((A) opaque objects, (B) semi-transparent objects, (C) full penetration The light object is placed in the center of pure white paper to capture the image of the white light environment;

Annex III (A) ~ (C) is to establish a shadow model by simulating the illumination of different color light sources, with different light transmissive objects ((A) opaque objects, (B) semi-transmissive objects, (C) full penetration The light object is placed in the center of pure white paper to capture the image of the yellow light environment;

Annexes (A) to (C) are the distribution of object pixels in the RGB color model with white light illuminating the opaque object;

Annexes (A) to (C) are the distribution of shadowed pixels in the RGB color model with white light illuminating opaque objects;

Annexes (A) to (C) are the distribution of object pixels in the RGB color model by illuminating the opaque object with yellow light;

Annex VII (A) ~ (C) is the distribution of shadow pixels in the RGB color model with yellow light illuminating the opaque object;

Annexes VIII (A) ~ (C) are the distribution of object pixels in the YC _b C _r color model by illuminating the opaque object with yellow light;

Annex IX (A) ~ (C) is the distribution of shadow pixels in the YC _b C _r color model with yellow light illuminating the opaque object;

Annex 10 (A) ~ (D) is the illumination area determination method (A) is the video screen image, (B) is the brightness distribution of all the pixels in the picture image, and the brightest trough can be used as the threshold value to find the brightness extreme Point (C) is to use the brightness extreme point of the picture image as the diffusion seed, and (D) to calculate the range of the illumination area in the picture image by the diffusion search method;

Annexes XI (A) ~ (C) are used to simulate the illumination area using the original background model to be revised to the current background model, (A) is the illumination area detected in the current video image, and (B) is the original background model, (C) ) is the current background model after simulating illumination;

Annexes 12(A)~(C) are foreground pixel detection results, (A) is the current video screen image, (B) uses the original background model, and (C) uses the current background model of simulated illumination;

Annex XIII (A) ~ (D) is the shadow pixel determination and removal steps, (A) is the current video screen image, (B) is the current state of the simulated lighting background model (C) the foreground pixel (D) is removed before the shadow is removed The foreground pixels after the shadow can be combined into a more accurate foreground object.

Claims (10)

A method for removing shadows in a moving light source environment, including recording the characteristic information of each illumination area in the video image, in addition to the bright position of the center of the extreme point and the position of the surrounding edge points, the system also records the color of the center and each edge of the illumination area. Value; in order to simulate the occurrence of each illumination area in the background model, to match the background model of the current situation of the monitoring environment, the illumination area is added to the background model by simulation using the gradient rendering method; firstly, according to the position of the center and edge points of the illumination area The distance and color difference values are used to calculate the respective variation ratios of the three primary colors of RGB (DiffRatR, DiffRatG, DiffRatB); then, in the background model image, starting from the center of the illumination region, the original pixels are separated by the distance from the center of the illumination region. The values are gradually increased by RGB values in proportion, and the pixel values of the position are highlighted, and the background image of the simulated simulated illumination is used to detect the foreground object of the current picture. The method for removing shadows in a moving light source environment as described in claim 1 , wherein the extreme point is to automatically detect and analyze the illumination region and the brightness characteristic in the image of the image, and calculate the difference in brightness distribution of all pixels in the image of the image. And find the brightest trough in the picture image as the threshold value, and then find the relative bright pixels as extreme points. The method for removing shadows in a moving light source environment as described in claim 1 , wherein the background model of the simulated illumination further comprises a color image determination and a shadow model for performing shadow pixel determination and removal, that is, with the same position background pixel. Compared with the color, the difference amplitude between the three color components should be close. If the two conditions satisfying that the color is darker than the background and the darkening amplitude of each color component is close to each other, it is determined that the foreground image at this position is a shadow pixel and is removed. A method for removing a shadow in a moving light source environment includes: detecting a video image of a video; detecting an extreme point of the image, calculating a brightness distribution of all pixels in the image, and finding a brightest valley as a threshold to find a relative The bright pixels are extreme points; the illumination area range is determined by using the extreme points of the picture image as the diffusion seed, and the detection and the aspect of the illumination area are automatically judged and characterized by the diffusion search method to calculate the illumination in the picture image. Area range; analysis of the characteristics of the illumination area, to establish a shadow model by analyzing the shadow features of each light source on different light-transmitting objects in a multi-light source environment; simulating the illumination area, automatically simulating the illumination area in a gradient rendering manner to obtain the situation The background model is used to revise the background model; the background model is revised, and the distance between the center of the illumination region and the position of the edge point and the color difference value are calculated, and the respective changes of the RGB values are calculated as a scale reference for rendering brightening, which is revised to form a background of the simulated illumination. Model; foreground object detection, a current background model using simulated lighting, The background subtraction method is used for foreground pixel detection; the shadow pixel is determined and removed, and the foreground pixel model using simulated illumination is used, and the foreground pixel obtained by the background subtraction method is used, and then the shadow model generated by the object of different light sources is created in advance, for all foregrounds. Pixels are used to determine and remove shadow pixels; foreground components are combined and modified to detect foreground objects of the current picture with a background image of the revised simulated illumination; and foreground objects are completed to accurately detect foreground objects. The method for removing a shadow in a moving light source environment as described in claim 4 , wherein the image image extreme point detection system automatically detects and analyzes an illumination area and a light characteristic in the image image, because the illumination area has brightness. Aggregation and gradation characteristics, calculate the difference in brightness distribution of all pixels in the image, and find the brightest trough in the image as the threshold, and then find the relative bright pixels as extreme points. The method for removing shadows in a moving light source environment as described in claim 4 , wherein the illumination region is determined to be a diffusion seed at the center of the illumination region, so as to search for outer neighbor points layer by layer, and determine and extreme The difference in brightness at the center of the point, if the difference is too large, belongs to the edge point position of the illumination area, and records the characteristic information of the extreme center point, the edge point position in each direction, and the color value as the illumination area. The method for removing shadows in a moving light source environment as described in claim 4 , wherein the simulated illumination region is based on a background model, and the illumination region is automatically simulated into a background model by a gradient rendering method, thereby automatically simulating The background model appears, and the foreground image is detected by the current background model. The method for removing shadows in a moving light source environment as described in claim 4 , wherein the revised background model is to modify the background model with a current background model. In the background model, starting from the center of the illumination region, the distance is layer by layer. In the distance of the center of the illumination area, the original pixel values are gradually increased by RGB values, and the pixel values of the position are brightened, and the background model of the simulated illumination is formed by revision. The method for removing shadows in a moving light source environment as described in claim 4 , wherein the foreground object detection system first determines the characteristics of the illumination region for each video picture, and performs a layer-by-layer rendering simulation mode. The illumination region of the location is added to the original background model to produce a current background model for detecting foreground pixels. The method for removing shadows in a moving light source environment according to claim 4 , wherein the shadow pixel determining and removing system compares each foreground pixel with the color of the phase pixel of the phase position, if the color is more than the background If two conditions of darkness and darkness of each color component are close, it is determined that the foreground image at this position is a shadow pixel and is removed.