JP2007180933A - Image sensor - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image sensor capable of detecting an abnormal background image including an image area by a mobile object. <P>SOLUTION: The image sensor is provided with a background image setting part 24 which includes a background image generating part generating a background image on the basis of a monitored image acquired previously to a prescribed time, a change area extracting part 14 which includes a background difference extracting part for extracting a background difference area by difference processing between a plurality of monitored images acquired after the prescribed time and the background image, and a background abnormal area detecting part 20 for comparing the image of the obtained background difference area with the plurality of monitored images to determine whether an abnormal area caused by the mobile object is included in the background image. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an image sensor intended to detect an intruder using an image obtained by imaging a monitoring space.

  There has been proposed an image sensor that performs background difference processing or the like on image information obtained by imaging using an imaging device such as a camera that images a monitoring space, and detects the intrusion of a moving object into the monitoring space. The background difference process stores, as a background image, an image obtained by capturing a monitoring space without an intruder obtained immediately after monitoring is started, and corresponding pixels in the input image and the background image newly capturing the monitoring space. A set of pixels whose difference is equal to or greater than a predetermined value is extracted as a change region caused by a change caused by an intruder or other object.

  Patent Document 1 discloses a technique for detecting a moving object such as an intruder who has entered a monitoring space by background difference processing. By initializing the background image at the start of monitoring and updating the background image with the input image when the change area is hardly extracted or when the majority of the input image at the time of monitoring is extracted as the change area Corresponds to changes in the lighting conditions in the surveillance space. In addition, the input image captured sequentially is subjected to background difference processing, and the change areas extracted thereby are set as candidates for moving objects, and the change extracted by background difference processing for those change areas and the input image captured further before. If the position of the corresponding change area is changed, it is determined that the object is a moving object.

JP 2001-008189 A

  However, when the background image is initialized or updated with the input image as in the above-described prior art, an image captured when a moving object is present in the monitoring space may become a background image. If background difference processing is performed using such a background image, there is a possibility that a moving object will not be detected.

  For example, as shown in FIG. 9A, an input image captured when monitoring is started at time t0 is set as a background image P10. At this time, an intruder already exists in the monitoring space, and the input image The case where the image area A0 by the intruder is included will be described. After that, if the intruder has moved away from the camera at time t1, as shown in FIG. 9B, the input image P11 that newly captures the monitoring space at time t1 is intruded in the background image P10. An image area B1 by an intruder is included so as to be included in an area A1 corresponding to the person's image area A0. In such a case, when background difference processing is performed between the input image P11 and the background image P10, an image area A0 including the image area B1 is extracted as a change area C1 as shown in FIG. 9C. Furthermore, when the intruder is moving further away from the camera at time t2, as shown in FIG. 9D, the input image P12 that newly captures the monitoring space at time t2 is displayed in the background image P10. The image area B2 by the intruder is included so as to be included in the area A1 corresponding to the image area A0 of the intruder. In such a case, when background difference processing is performed between the input image P12 and the background image P10, as shown in FIG. 9E, an image area A0 that includes the image area B2 is extracted as the change area C2.

  At this time, if the change detection of the position or shape is made gentle, it is determined that there is no change in the position or shape in the image of the change area C1 at time t1 and the change area C2 at time t2, and the change due to the moving object In some cases, an intruder cannot be detected because it is not considered as a region.

  In addition, when a large moving object such as a track enters the surveillance space and most of the input image is extracted as a change area, if the background image is updated by this input image, an abnormal background image is obtained and the same problem occurs. there is a possibility.

  On the other hand, when the background image is not updated, the influence due to the change in the illumination environment of the monitoring space cannot be avoided, and the sensitivity and accuracy of intruder detection may be reduced.

  Further, in the above conventional technique, the background image is updated when the changed region is hardly extracted. However, when an abnormal background image is obtained as described above, the background image is not updated because the changed area is continuously extracted by the background difference process, and it is artificially initial to restore the background image to normal. Need to be done.

  An object of the present invention is to provide an image sensor that can solve at least one of the above-described problems in view of the above-described problems of the related art.

  The present invention is an image sensor that sequentially acquires a monitoring image obtained by imaging a monitoring space and detects a moving object existing in the monitoring space based on a change area extracted by a difference process between monitoring images having different acquisition times. A background image generation unit that generates a background image based on a monitoring image acquired before a predetermined time, and a background difference region is extracted by a difference process between the monitoring image acquired multiple times after the predetermined time and the background image. A background difference region extraction unit; and a background abnormal region detection unit that compares background difference regions in the plurality of monitoring images and determines whether or not the background image includes an abnormal region due to a moving object. It is characterized by.

  Here, it is preferable that the change area is extracted by a difference process between monitoring images captured at different times in the abnormal area, and the background difference area is extracted as the change area outside the abnormal area. is there. By performing such processing and extracting the change region, it is possible to detect a moving object even when an abnormal region due to the moving object exists in the background image.

  Further, the background abnormal area detection unit compares the position and shape of the background difference area in the plurality of monitoring images, and pixel values of pixels corresponding to each other in the plurality of monitoring images in the background difference area, and It is preferable to determine that the background image includes an abnormal region due to a moving object when the position and the shape substantially match and the pixel value fluctuates to a predetermined value or more. By performing such a determination process, it is possible to accurately extract an abnormal region due to a moving object included in the background image.

  The image processing apparatus may further include a background image update unit configured to update a difference area between the plurality of monitoring images from an abnormal area included in the background image with a newly acquired monitoring image to generate a new background image. Is preferred. By updating the background image in this way, the image in the abnormal region included in the background image can be updated to a normal image that is not an image of a moving object. This can increase the probability that a moving object can be detected accurately.

  According to the present invention, it is possible to detect an abnormal background image including an image area formed by a moving object. Thereby, it is possible to automatically normalize an abnormal background image including an image region of a moving object and detect the moving object. Therefore, the sensitivity and accuracy of detecting a moving object such as an intruder can be improved.

  As shown in FIG. 1, the image sensor 100 according to the embodiment of the present invention includes an imaging unit 10, an image input interface (I / F) 12, a change area extraction unit 14, a labeling processing unit 16, a tracking processing unit 18, and a background. The abnormal region detection unit 20, the background abnormal region setting unit 22, the background image setting unit 24, the feature amount calculation unit 26, the abnormality determination unit 28, the storage unit 30, and the output unit 32 are configured.

  Image input interface (I / F) 12, change region extraction unit 14, labeling processing unit 16, tracking processing unit 18, background abnormal region detection unit 20, background abnormal region setting unit 22, background image setting unit 24, feature amount calculation unit 26, the apparatus main body including the abnormality determination unit 28 and the storage unit 30 can be realized by a basic computer configuration including a CPU, a memory, an input / output device, and the like. In this embodiment, the detection target is described as an intruder, but the same processing can be performed even if the detection target is replaced with another detection target.

  The image sensor 100 starts intruder detection at an operation for starting image monitoring or at a set timing. The imaging unit 10 includes a camera equipped with a CCD imaging device. The imaging unit 10 captures an image of a monitoring space to be monitored at a predetermined time interval, converts the brightness of the monitoring space into an electrical signal representing a voltage value or a current value, and outputs the electrical signal to the image input I / F 12. To do. The image input I / F 12 includes an amplifier and an analog / digital converter. The image input I / F 12 receives an electrical signal from the imaging unit 10, amplifies the electrical signal with an amplifier, converts the electrical signal into a digital signal with an analog / digital converter, and monitors images (hereinafter referred to as other images) at predetermined time intervals. (Referred to as an input image for distinction from the image). The input image is a set of luminance values (pixel values) expressing brightness in 256 gradations. The generated input image is sent to the change area extraction unit 14 and the background image setting unit 24.

  Note that the image may be an image obtained by extracting a luminance signal from a color image, or an image obtained by extracting one of the color components from the color image. The images acquired by the imaging unit 10 may be acquired regardless of whether they are still images or moving images. A photoelectric conversion element such as a CMOS image sensor can be used instead of the CCD image sensor.

  It is also preferable to have an automatic gain control (AGC) function for automatically adjusting the dynamic range of the image. For example, it is preferable to perform adjustment so that the entire image uses a dynamic range of 256 gradations (0 to 255 gradations) as much as possible.

  The change area extraction unit 14 receives the background abnormal area setting information from the background abnormal area setting unit 22 and performs a change area extraction process based on the setting information. When the background abnormal area is not set in the background abnormal area setting unit 22, the change area extraction unit 14 reads the background image generated in the background image setting unit 24 and stored in the storage unit 30, and the image input I / O Background difference processing for calculating a difference value between corresponding pixels with the input image obtained from F12 is performed. Further, when the background abnormal area is set in the background abnormal area setting unit 22, the image area set as the background abnormal area is obtained from the input image newly obtained from the image input I / F 12 and before that. Other than the image area set as the background abnormal area, the inter-frame difference processing is performed to calculate the difference value between the corresponding pixels with the past input image (the input image of the frame immediately before or a predetermined number before). For the area of, background difference processing for calculating a difference value between corresponding pixels between the input image newly obtained from the image input I / F 12 and the background image is performed. It is preferable that past input images are stored and held in the storage unit 30 as necessary, and read out from the storage unit 30 for use. For an image subjected to background difference processing or inter-frame difference processing, if the absolute value of the difference value for each pixel is equal to or greater than a predetermined binarization threshold, a change pixel having a pixel value “1”, a predetermined binarization threshold If it is smaller, a binarized image having a pixel value “0” as an unchanged pixel is generated. At this time, it is preferable to generate a binarized image from which noise has been removed by performing filtering processing such as expansion and contraction or removal of a small area (independent change pixel removal). As a result, a change area composed of pixels in which the variation of the luminance value is equal to or greater than the binarization threshold between the newly obtained input image, the background image, and the past input image is extracted. Information on the position and shape of the change area is stored and held in the storage unit 30.

  In the labeling processing unit 16, the regions of changed pixels that collectively constitute one group in the binarized image obtained by the background difference processing or the inter-frame difference processing of the changed region extracting unit 14 are grouped as changed regions, In order to identify the change region, a labeling process is performed by assigning a unique label to make the “object candidate region”. The tracking processing unit 18 performs tracking processing of the object candidate region that has been subjected to labeling processing. In the tracking processing, an object candidate area extracted from the binarized image for the current input image is within a tracking search range set around the object candidate area extracted from the binarized image for the past input image. It is determined whether or not the object candidate area exists, and if there is a corresponding object candidate area, it is estimated that the image areas are obtained by capturing the same object based on the similarity of the feature amount such as the size and shape of the area. The object candidate areas are associated with each other.

  The background abnormal area detecting unit 20 has the positions and shapes of the object candidate areas matched in the tracking processing unit 18 between the predetermined frames substantially equal to each other, and the pixel values of the corresponding pixels between the predetermined frames If the candidate area changes more than a predetermined amount, the object candidate area is detected as a “background abnormal area”. This process is performed for each of the object candidate areas extracted by the background difference process and associated in the tracking processing unit 18. The processing in the background abnormal area detection unit 20 will be described in detail later.

  The background abnormal area setting unit 22 sets an area for executing the inter-frame difference based on the background abnormal area detected by the background abnormal area detection unit 20. The background abnormal area is set in units of pixels. Information on the position and shape of the background abnormal region set by the background abnormal region setting unit 22 is output as setting information to the change region extraction 14 and is stored and held in the storage unit 30.

  The background image setting unit 24 includes a background image generation unit and a background image update unit. The background image generation unit stores the input image captured at the start of the monitoring process or at a predetermined timing in the storage unit 30 as a background image, or weighted addition of the input image and the background image stored in the storage unit 30 The background image is generated or updated by processing such as storing the image as a new background image in the storage unit 30. The background image update unit partially updates the background image based on the background abnormal region set by the background abnormal region setting unit 22. The processing in the background image setting unit 24 will be described in detail later.

  The feature amount calculation unit 26 calculates a feature amount that is a parameter indicating the “humanity” of an object candidate region associated with the input image of a plurality of frames in the tracking processing unit 18 as a region where the same object is imaged. . Here, the movement distance, movement locus, number of continuous appearances, etc. of the object candidate area are calculated. However, the present invention is not limited to these. Image feature values such as luminance average values and luminance variance values of object candidate areas, geometric feature quantities such as area, aspect ratio, and barycentric position of object candidate areas, and background images A comparison value such as a normalized correlation value or a difference average value may be calculated as a feature amount.

  Specifically, the movement distance is obtained based on the distance between object candidate areas associated as areas where the same object is imaged by the tracking processing unit 18 for input images captured at different times. That is, the position coordinates (center of gravity coordinates) in the image stored for the object candidate areas labeled as the same object are read from the storage unit, and the position coordinates and the position coordinates of the corresponding object candidate area of the current input image are read. The actual distance to (centroid coordinates) is calculated as the actual distance movement amount of the object.

  In addition, a change in the velocity vector of the barycentric coordinates of the object candidate region associated with the plurality of input images obtained in the past as the region where the same object is imaged by the tracking processing unit 18 is calculated as a movement locus. In addition, the number of images (frames) continuously determined as a region where the tracking object 18 has captured the same object for a plurality of input images obtained in the past is calculated as the number of consecutive appearances.

  The abnormality determination unit 28 functions as a moving body determination unit that determines whether each object candidate region is an intruder based on the feature amount calculated by the feature amount calculation unit 26. If the abnormality determination unit 28 determines that at least one object candidate region by the intruder exists, an abnormality signal is output to the output unit 32.

  Specifically, the moving distance, moving trajectory, and number of consecutive appearances of the object candidate area are received from the feature amount calculation unit 26, and it is determined whether or not each object candidate area is an intruder based on these feature amounts. . When the moving distance is equal to or greater than a predetermined threshold, it can be determined that the person is “human”. In addition, the movement trajectory can be determined as “human” when a straight line or a smooth curve is drawn, and can be determined as “an object other than a person” when it greatly changes in a zigzag manner. In addition, when the number of continuous appearances is continuous for a predetermined threshold or more, it can be determined as “human”, and when it is intermittently smaller than the predetermined threshold, it is determined as “an object other than a person”. can do. Therefore, the condition that the movement distance of the object candidate region for a plurality of past input images is equal to or greater than a predetermined threshold, the number of consecutive appearances is equal to or greater than the predetermined threshold, and the movement locus is smoother than the predetermined threshold is satisfied. It is determined that there is an intruder in the monitoring space.

  Note that the process for determining the presence or absence of an intruder is not limited to this, and the determination is made by comparing other image feature values, geometric feature values, comparison values with the background image, etc. with a predetermined threshold value. May be. Alternatively, a plurality of feature amounts may be combined as appropriate, and the determination may be performed by logical product or logical sum of the determination results.

  The storage unit 30 stores and holds information used for each process of the image sensor 100, that is, an input image, a background image, various threshold values, information on an object candidate area, and information on an abnormal background area. These pieces of information are read and updated as appropriate in response to requests from each unit of the image sensor 100. The storage unit 30 includes a storage device such as a semiconductor memory, a magnetic disk device, and an optical disk device.

  The output unit 32 functions as means for notifying the outside of the abnormality when receiving the abnormality signal from the abnormality determining unit 28. For example, a warning is issued to the surroundings by a buzzer or the like, or a captured image or an alarm signal is transmitted to the monitoring center.

  Hereinafter, the intruder detection process by the image sensor 100 will be described with reference to the flowchart of FIG. Each process of the flowchart can be executed by each processing unit of the computer constituting the image sensor 100 by using a control program executable by the computer.

  In step S10, the current input image (hereinafter referred to as the current input image) captured by the imaging unit 10 at the start of monitoring is input to the background image setting unit 24 via the image input I / F 12, and the background image setting unit 24 The input image is initialized as a background image. In step S <b> 12, the current input image is input to the change area extraction unit 14 and the background image setting unit 24 via the image input I / F 12 by the imaging unit 10.

  In step S <b> 14, the change area extraction unit 14 extracts a change area from the current input image. Step S14 is made into a subroutine as shown in FIG. In step S14-1, a counter C for specifying each pixel included in the current input image is initialized. Here, each pixel included in the current input pixel is assigned a unique number and is specified by the value of the counter C. In step S14-2, it is determined whether or not the pixel specified by the counter C is set in the background abnormal region. The change area extraction unit 14 receives the setting information from the background abnormal area setting unit 22 and determines whether or not the pixel specified by the counter C is set as the background abnormal area. If the background abnormal area is set, the process proceeds to step S14-3. If the background abnormal area is not set, the process proceeds to step S14-4.

  In step S14-3, pixel values corresponding to each other between a current input image and a past input image (an input image of a frame immediately before or a predetermined number before) obtained for the pixel specified by the counter C are obtained. The difference value is calculated. On the other hand, in step S14-4, the background image stored in the storage unit 30 is read, and a difference value between corresponding pixel values between the input image and the background image is calculated for the pixel specified by the counter C. Is done. Thereafter, the process proceeds to step S14-5. In step S14-5, when the absolute value of the difference value calculated in step S14-3 or step S14-4 is equal to or greater than a predetermined binarization threshold, a change pixel having a pixel value “1”, a predetermined binary value If it is smaller than the binarization threshold, binarization processing is performed to make a non-changeable pixel having a pixel value “0”.

  In step S14-6, it is determined whether or not processing has been completed for all pixels in the image. If the process has been completed for all pixels, the process returns to step S16 of the main routine. If the process has not been completed for all pixels, the counter C is incremented by 1 in step S14-7 and step S14-2 is performed. Repeat the process.

  In step S16, the labeling processing unit 16 performs the labeling process as described above. In step S18, the tracking processing unit 18 performs the tracking process as described above. Here, the collected change pixels are set as an “object candidate region” with a unique label number. Hereinafter, the object candidate area for the current input image is referred to as the current object candidate area.

  In step S20, the background abnormal area detection unit 20 performs a background abnormal area detection process. Note that this process targets only the area extracted by the background difference process (background difference area) among the areas extracted as the object candidate areas in step S16, and the area extracted by the inter-frame difference process (between frames). (Difference area) is not targeted. Step S20 is made into a subroutine as shown in FIG. In step S20-1, a counter M for specifying the label number of the object candidate area, recording information regarding the position and shape of each object candidate area, and a counter K (M) indicating the number of consecutive background abnormal areas are initialized. Here, it is assumed that label numbers are assigned in order from 1 to each object candidate region extracted by the background difference process in step S16, and the counter M is initialized to 1. In addition, the counter K (M) indicating the number of consecutive background abnormal areas for the object candidate area with the label number already assigned in the past is maintained as it is, and the number of consecutive background abnormal areas for the object candidate area with the newly assigned label number is maintained. Is initially set to 0.

  In step S20-2, it is determined whether or not the object is a new object candidate area that has not been extracted as an object candidate area in the past labeling process. When it is determined in the tracking process in the tracking process in step S18 that there is an object candidate area that is an image area obtained by imaging the same object in the past based on the similarity of the feature amount such as the size and shape of the object candidate area. In step S20-3, the process proceeds to step S20-3. Otherwise, the process proceeds to step S20-13.

  In step S20-3, a background abnormal region candidate is detected. The process of step S20-3 is performed according to the flowchart shown in FIG. Here, the shape, position, and image of the current object candidate area detected by the tracking processing unit 18 and the object candidate area obtained with respect to the input image immediately before (one frame before) stored in the storage unit 30 will be described. The comparison is made to determine whether or not the object candidate region is a candidate for a background abnormal region whose background is abnormal based on the degree of coincidence of the shape, position and image content.

  In step S30-1, the shape of the object candidate area is obtained. A rectangular area circumscribing the current object candidate area whose label number matches the counter M is obtained, and the width W1 and height H1 of the rectangular area are calculated. Further, a rectangular area circumscribing the object candidate area in the immediately preceding input image associated with the current object candidate area is obtained by the tracking process in step S18, and the width W2 and height H2 of the rectangular area are calculated. These values are stored and held in the storage unit 30. In step S30-2, it is determined whether or not the shapes of the two object candidate areas match. Specifically, when the difference between the widths W1 and W2 of the two object candidate regions is equal to or smaller than a predetermined width threshold value, and the difference between the heights H1 and H2 is equal to or smaller than a predetermined height threshold value, the shape Are determined to match, and the process proceeds to step S30-3. Otherwise, it is determined that the shapes do not match, and the process proceeds to step S20-4.

  In addition, without using the width and height of the circumscribed rectangle, the Hausdorff distance for measuring the degree of coincidence of the binarized areas is calculated, and if the Hausdorff distance is equal to or less than a predetermined threshold, the shapes match. You may judge.

  In step S30-3, the movement amount of the object candidate area is obtained. The number of pixels in a region where the current object candidate region whose label number matches the counter M and the object candidate region corresponding to the immediately preceding input image associated therewith is obtained. Then, the ratio R1 of the number of pixels of the overlapping area to the total number of pixels of the current object candidate area specified by the counter M, and the pixels of the overlapping area with respect to the total number of pixels of the object candidate area in the immediately preceding input image associated therewith The number ratio R2 is calculated. In step S30-4, it is determined whether or not the positions of the two object candidate regions match based on the movement amount. If the ratio R1 is equal to or greater than the predetermined first ratio threshold and the ratio R2 is equal to or greater than the predetermined second ratio threshold, it is determined that the positions of the two object candidate areas match, and step S30- If not, it is determined that the positions do not match, and the process proceeds to step S20-4.

  In addition, without using the ratio of the number of pixels in the overlapping area, it may be determined that the positions match when the distance between the centroid coordinates of the two object candidate areas is equal to or less than a predetermined threshold.

  In step S30-5, a luminance difference is calculated to determine the degree of coincidence between the images of the two object candidate regions. The luminance difference between the pixel in the current object candidate area specified by the counter M in the current input image and the corresponding pixel in the object candidate area specified by the counter n in the immediately preceding input image is greater than or equal to a predetermined luminance threshold. Find the number of pixels. Then, a ratio D1 of the number of pixels with respect to the total number of pixels in the current object candidate area and a ratio D2 with respect to the total number of pixels of the object candidate area of the immediately preceding input image associated therewith are calculated. In step S30-6, it is determined based on the luminance difference ratios D1 and D2 whether the contents of the images of the two object candidate regions match. When the ratio D1 is equal to or greater than the predetermined first luminance difference threshold and the ratio D2 is equal to or greater than the predetermined second luminance difference threshold, it is determined that there is a change in the image contents of the two object candidate areas. The process proceeds to step S30-7. Otherwise, it is determined that there is no change in the content of the image, and the process proceeds to step S20-4.

  Note that without using the luminance difference ratio, it may be determined that there is a change when the correlation between the two object candidate regions is equal to or less than a predetermined threshold. Further, when the mean square error between the two object candidate areas is equal to or greater than a predetermined threshold, it may be determined that there is a change.

  In step S30-7, the current object candidate area specified by the counter M is determined as a background abnormal area candidate. Thereafter, the process proceeds to step S20-4.

  In step S20-4, it is determined whether or not the current object candidate area specified by the counter M is a background abnormal area candidate. If the current object candidate area is a background abnormal area candidate, the process proceeds to step S20-5, and if not, the process proceeds to step S20-11. In step S20-5, the position and shape of the newly set background abnormal region candidate are stored in the storage unit 30. In step S20-6, the counter K (M) of the number of consecutive background abnormal areas for the counter M is incremented. In step S20-7, it is determined whether or not the counter K (M) of the background abnormal region continuation count for the counter M is equal to or greater than a predetermined continuation count threshold. If the counter K (M) is greater than or equal to the predetermined continuous number threshold, the process proceeds to step S20-8, and if not, the process proceeds to step S20-13.

  In step S20-8, background abnormal region determination processing is performed based on a plurality of input images captured in the past. The process of step S20-8 is performed according to the flowchart shown in FIG. Here, it is determined whether or not the position and shape when the object candidate area specified by the counter M is stored as a background abnormal area candidate over the past N times (N frames) of input images are unchanged. . That is, the invariance of the position and shape of the current object candidate area and the object candidate area of the previous frame can be determined by the background abnormal area candidate extraction process, but minute changes are accumulated between a plurality of frames. For a longer time, there is a possibility that the position and shape of the object candidate region have changed greatly. Therefore, a change over a long time is determined using a plurality of input images. Here, the value of N is set to a value equal to or less than the continuous number threshold in step S20-7.

  In step S <b> 32-1, the position, width, and height of the circumscribed rectangle of the object candidate area specified by the counter M determined as the background abnormal area candidate in the past N input images are read from the storage unit 30. By calculating the logical sum of these circumscribed rectangular areas, the area (sweep area W1) in which the background abnormal area candidates have been swept in the image in the past N times is obtained. In addition, the average circumscribed rectangular area W2 is obtained by calculating the average value of the width and height of the circumscribed rectangular area of the background abnormal area candidate in the past N times. In step S32-2, it is determined whether or not the sweep area W1 and the average circumscribed rectangular area W2 match. If the width difference between the sweep area W1 and the average circumscribed rectangular area W2 is equal to or smaller than a predetermined threshold and the height difference is equal to or smaller than the predetermined threshold, the process proceeds to step S32-3. If not, the process proceeds to step S20-9. In step S32-3, the current object candidate area specified by the counter M, which is a background abnormal area candidate, is set as the background abnormal area, and the process proceeds to step S20-9.

  In addition, the background abnormal region candidate detection process is not performed for each of a plurality of past input images, and the position, shape, and image of the object candidate region in all the input images are stored in the storage unit 30, and the information is stored in these pieces of information. On the basis of this, it is possible to determine whether or not the background abnormal region using a plurality of input images.

  In step S20-9, it is determined whether or not the background abnormal region candidate is set as the background abnormal region in step S20-8. If the background abnormal region is set, the process proceeds to step S20-10, and if not, the process proceeds to step S20-11. In step S20-10, the storage unit 30 stores the position and shape of the current object candidate area set as the background abnormal area. In order to simplify the processing, a circumscribed rectangle of the current object candidate area set as the background abnormal area may be newly set as the background abnormal area, and the position and shape thereof may be stored in the storage unit 30.

  In step S20-11, the object candidate area specified by the counter M is set not as a background abnormal area but as a significant object candidate area obtained by imaging a significant object that may be an intruder. In step S20-12, the record information of the position and shape of the object candidate area specified by the counter M determined as a significant object candidate area is deleted, and the background abnormal area candidate continuous count K (M) is initialized to zero. . In step S20-13, it is determined whether there is another object candidate region. If there is another object candidate that has not been processed yet, the counter M is incremented in step S20-14, and the process is repeated from step S20-2. If not, the process proceeds to step S22. .

  In step S22, a background image update process is performed in the background image setting unit 24 in accordance with the detection process result of the background abnormal area in step S20. Step S22 is made into a subroutine as shown in FIG. In step S22-1, it is determined whether or not a forced full screen update condition is satisfied. The forced full screen update condition is a condition that is satisfied when the ratio of the number of changed pixels to the total number of pixels by the background difference process is equal to or greater than a predetermined threshold. When such a condition is satisfied, it is considered that a sudden change in the illumination of the monitoring space has occurred, so it is preferable to update the entire screen of the background image to the latest input image. When the forced full screen update condition is satisfied, in step S22-2, the background image held in the storage unit 30 is updated with the current input image, and the process proceeds to step S24. If the forced full screen update condition is not satisfied, the process proceeds to step S22-3.

  In step S22-3, it is determined whether or not there is a change area due to the background difference. If a background difference area is extracted in the background difference process in step S14, the process proceeds to step S22-5. If not, all the pixels of the background image are updated in step S22-4. It is preferable that the update process of all the pixels of the background image in step S22-4 is a process in which the luminance values of the pixels of the current background image and the current input image are respectively weighted and the luminance values of the corresponding pixels are added. It is. For example, the luminance value of the pixel of the current background image is multiplied by 0.8, the luminance value of the pixel of the current input image is multiplied by 0.2, and the luminance value of the corresponding pixel is added to calculate a new background image. . The background image may be updated with the current input image. After the update process in step S22-4, the process proceeds to step S24. In step S22-5, a pixel other than the change region is updated. The information on the position and shape of the change area extracted in step S14 is read from the storage unit 30, and the current background image corresponding to the area excluding the change area and the luminance value of the pixel of the input image are respectively weighted to correspond. It is preferable that the luminance value of the pixels be added. For example, the luminance value of the pixel of the current background image is multiplied by 0.8, the luminance value of the pixel of the current input image is multiplied by 0.2, and the luminance value of the corresponding pixel is added to calculate a new background image. . The background image may be updated with the current input image. After the update process in step S22-5, the process proceeds to step S22-6.

  In step S22-6, it is determined whether or not a background abnormal region has been set in step S20. If the background abnormal area is set, the process proceeds to step S22-7, and if not, the process proceeds to step S24. In step S22-7, a background image update process is performed based on the inter-frame difference area and the background abnormal area. The updating process here is performed only for pixels corresponding to the area obtained by removing the inter-frame difference area from the background abnormal area in the background image. Information on the position and shape of the background abnormal region extracted in step S20 and the change region between frames extracted in step S14 is read from the storage unit 30, and the current corresponding to a region that is not a change region between frames in the background abnormal region. The luminance value of the pixel of the background image is updated with the luminance value of the pixel of the input image. As a result, only the abnormal region in the background image can be replaced with the region of the current input image to be normalized. Note that the circumscribing rectangle of the inter-frame difference area may be calculated, and the processing amount may be deleted by setting the pixels in the area obtained by removing the circumscribing rectangle from the background abnormal area in the background image as the update target. After the update process in step S22-7, the process proceeds to step S24.

  In step S24, the feature amount calculation unit 26 calculates the feature amount. As described above, the tracking processing unit 18 calculates a feature amount that is a parameter indicating “humanity” of an object candidate region associated with a plurality of frames. For example, the moving distance, moving trajectory, and number of continuous appearances of the object candidate area.

  In step S26, intruder detection processing is performed in the abnormality determination unit 28, and it is determined whether or not there is an intruder in the monitoring space. Specifically, the moving distance, moving trajectory, and number of consecutive appearances of the object candidate area are received from the feature amount calculation unit 26, and it is determined whether or not each object candidate area is an intruder based on these feature amounts. . When the moving distance of the object candidate region for a plurality of past input images is equal to or greater than a predetermined threshold, the number of consecutive appearances is equal to or greater than a predetermined threshold, and the condition that the movement trajectory is smoother than the predetermined threshold is satisfied It is determined that there is an intruder in the monitoring space. If there is an intruder, the process proceeds to step S28, and if not, the process is repeated from step S12.

  In step S28, an abnormality output process indicating an abnormality in the monitoring space is performed. The abnormality determination unit 28 outputs an abnormality signal indicating abnormality to the output unit 32. As a result, a warning is issued to the surroundings by a buzzer or the like, or a captured image or an alarm signal is transmitted to the monitoring center.

  As described above, according to the present embodiment, it is possible to detect an abnormal background image including an image area due to a moving object, and to automatically normalize the abnormal background image including an image area due to the moving object. Can do. Therefore, the sensitivity and accuracy of detecting a moving object such as an intruder can be improved.

  Hereinafter, an embodiment will be described with reference to FIG. FIG. 8 shows the input image, the background image, the extracted image extracted by the change region extraction unit 14, the background abnormal region, the detection result of the background abnormal region, and the abnormality determination result at the time 1 to 5 at the start of sensing from above. Yes. In addition, the intruder has moved from a region close to the imaging unit 10 to a region far from the imaging unit 10 in the monitoring space, and the intruder is captured in the input images at times 1 to 5 at the start of sensing.

  In the input images obtained at times 1 to 3, the intruder image (indicated by hatching) completely overlaps the intruder image included in the background image obtained at the start of sensing. Therefore, the change area (indicated by hatching) in the extracted image matches the intruder image included in the background image. Here, assuming that the threshold of the number of consecutive background abnormal region candidates for extracting the background abnormal region is set to 3, the change region included in the extracted image at time 3 is set as the background abnormal region.

  In the extraction image generation processing at times 4 and 5, inter-frame difference processing is executed in the region set as the background abnormal region (hatched region) at time 3, and background difference processing is performed in the other regions. As a result, at times 4 and 5, the change area different from the change area in the extracted images at times 1 to 3 even though the intruder image in the input image completely overlaps the intruder image in the background image. Is extracted. At time 5, the area other than the circumscribed rectangular area of the extracted image change area at time 4 in the image area of the intruder in the background image is replaced with the input image at time 4.

  By performing the processing as described above, at time 5, an abnormality is detected based on the difference in feature amount between the change area in the extracted image at time 4 and the change area in the extraction area at time 5, and an alarm for abnormality determination is detected. Will be issued.

  In the above embodiment, when the image sensor 100 detects the background abnormal area, the image sensor 100 performs inter-frame difference processing in the background abnormal area, and performs update processing for normalizing the background image. A configuration may be adopted in which an abnormality is output to the outside by the output unit 32 when a background abnormal region is detected without performing inter-frame difference processing or update processing. With such a configuration, the processing amount can be reduced while maintaining the accuracy of detecting an intruder.

  In other words, the background abnormal region detection unit 20 notifies the output unit 32 that a background abnormal region has been detected, and steps S14-2 and S14-3 are omitted in the processing of the change region extraction unit 14 (FIG. 3). The process proceeds from S14-1 and S14-7 to S14-4, and a notification process is added to the output unit 32 between S20-9 and S20-10 in the process of the background abnormal area detection unit 20 (FIG. 4). In the process of the background image setting unit 24 (FIG. 7), S22-6 and S22-7 are omitted.

It is a block diagram which shows the structure of the image sensor in embodiment of this invention. It is a flowchart which shows the monitoring process in embodiment of this invention. It is a flowchart which shows the change area | region extraction process in embodiment of this invention. It is a flowchart which shows the background abnormal area | region detection process in embodiment of this invention. It is a flowchart which shows a background abnormal area | region candidate detection process in embodiment of this invention. It is a flowchart which shows the background abnormal area | region determination process in embodiment of this invention. It is a flowchart which shows the background image update process in embodiment of this invention. It is a figure which shows the Example of the monitoring process in embodiment of this invention. It is a figure explaining the conventional monitoring process.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 10 Imaging part, 14 Change area extraction part, 16 Labeling process part, 18 Tracking process part, 20 Background abnormal area | region detection part, 22 Background abnormal area | region setting part, 24 Background image setting part, 26 Feature-value calculation part, 28 Abnormality determination part , 30 storage unit, 32 output unit, 100 image sensor.

Claims (4)

An image sensor that sequentially acquires a monitoring image obtained by imaging a monitoring space and detects a moving object existing in the monitoring space based on a change area extracted by a difference process between monitoring images having different acquisition times.
A background image generation unit that generates a background image based on a monitoring image acquired before a predetermined time;
A background difference area extraction unit that extracts a background difference area by a difference process between a plurality of monitoring images acquired after the predetermined time and the background image;
A background abnormal region detection unit that compares background difference regions in the plurality of monitoring images and determines whether or not the background image includes an abnormal region due to a moving object;
An image sensor comprising: The image sensor according to claim 1.
The change area is extracted by difference processing between monitoring images captured at different times in the abnormal area, and the background difference area is extracted as the change area outside the abnormal area. . The image sensor according to claim 1 or 2,
The background abnormal area detection unit compares the position and shape of a background difference area in the plurality of monitoring images and pixel values of pixels corresponding to each other in the plurality of monitoring images in the background difference area, and An image sensor characterized by determining that an abnormal region due to a moving object is included in the background image when the shapes substantially match and the pixel value fluctuates to a predetermined value or more. In the image sensor as described in any one of Claims 1-3,
A background image update unit that generates a new background image by updating a region obtained by removing a difference region between the plurality of monitoring images from an abnormal region included in the background image with a newly acquired monitoring image, An image sensor.

Images