JP6246508B2 - Moving body image extraction method and moving body image extraction apparatus - Google Patents

Description

  The present invention can efficiently detect a moving object included in surveillance camera video data captured by a monitoring camera installed in public transportation, an intersection, a building, or the like, and extract moving object image data. The present invention relates to an extraction method and a moving image extraction apparatus.

  2. Description of the Related Art Conventionally, a technique is known in which surveillance cameras are installed at public transportation, intersections, buildings, and the like, and moving objects such as persons and vehicles included in images captured by the surveillance cameras are detected. For example, Patent Document 1 discloses a technique for capturing a normal background image in advance and determining that an abnormality has occurred when the difference between the background image and the input image for each pixel exceeds a predetermined threshold. It is disclosed. For this reason, when the technique according to Patent Document 1 is applied to moving object detection, a background image that does not have a moving object is registered in advance, and a pixel that is a difference between the input image captured by the monitoring camera and the background image When the value exceeds a predetermined threshold, such a difference can be extracted as a moving object.

  However, although the thing of the said patent document 1 is effective when the illumination condition etc. of an imaging environment are unchanged, it is difficult to apply when an imaging environment changes every moment. This is because a background image must be prepared for each imaging environment such as time, weather, and season. For this reason, a technique for updating a background image following changes in the imaging environment is known.

  For example, in Patent Document 2, a background image is prepared in the same manner as in Patent Document 1, and a difference for each pixel of the input image and the background image is obtained, and this difference is exceeded when the difference exceeds a predetermined threshold value. A technique for extracting a difference as a moving object and updating a background image as needed when a predetermined condition is satisfied is disclosed.

Japanese Patent Laying-Open No. 2005-143052 JP-A-8-63683 JP 2009-10636 A

  However, when such a patent document 2 is used, a predetermined threshold value must be updated every time the imaging environment changes, and thus a problem that a great amount of human labor is required to set an appropriate threshold value. There is.

  More specifically, for example, since the difference in contrast between light and dark is large in the daytime period, the threshold is set in consideration of the difference in contrast, while the contrast in light and darkness is set in the nighttime period. Since the difference is small, there is a high possibility of erroneous detection if the same threshold is used as in the daytime. For this reason, the same threshold cannot be used in the daytime and at night, and the threshold needs to be changed according to the imaging environment. Unless an appropriate threshold value is set, a situation may occur in which a minute luminance difference between the input image and the background image is erroneously determined as a moving object, or a shadow portion of the moving object is erroneously determined as a part of the moving object.

  There is also a method of using a common threshold by correcting the contrast between daytime and nighttime to the same extent using conventional contrast enhancement technology (for example, correction technology based on Retinex theory or technology described in Patent Document 3). Although it is conceivable, noise is usually amplified with the correction of contrast, and this may cause problems such as false detection of moving objects and omission of detection under the influence of the amplified noise.

  The present invention is to solve the above-described problems of the prior art, and even if the imaging environment changes, an appropriate threshold value is set without requiring human labor, and a moving object image is efficiently extracted. It is an object of the present invention to provide a moving body image extraction method and a moving body image extraction apparatus.

In order to solve the above-described problems and achieve the object, the present invention is arranged in chronological order in which images are taken in an environment in which at least one of the amount of natural light or illumination light and the irradiation direction may fluctuate. A moving body image extraction method that uses a plurality of captured images and extracts a moving body image included in the captured images as a moving body image, obtains a background image from which the influence of the moving body is eliminated , and the background image A background image acquisition step of generating a variation coefficient indicating a luminance value and a variation degree of the luminance value for each pixel, and a luminance difference value for each pixel of the background image acquired by the background image acquisition step and the captured image And the calculated luminance difference value is compared with the variation coefficient to determine whether or not to adopt the pixel of the captured image in the difference image that is the difference between the background image and the captured image. While, a difference image generation step of generating a difference image, and a background image update step of updating the background image from the background image and the captured image, in a single body from the difference image generated by the difference image generation step A moving object candidate extraction step for extracting a possible partial image as a moving object candidate, and when a plurality of moving object candidates are extracted by the moving object candidate extraction step, the appropriate moving object candidates are narrowed down from the plurality of moving object candidates. A moving object candidate narrowing-down step.

  Further, in the present invention according to the above invention, the background image acquisition step analyzes the distribution of luminance values of the plurality of captured images for each pixel corresponding to the same position with respect to the plurality of captured images having different imaging timings. In this case, the luminance value indicating a specific value for the distribution of luminance values is excluded, and the luminance value of the pixels of the background image is calculated using the luminance values remaining without being excluded.

Further, the present invention uses a plurality of captured images arranged in time series in an environment where at least one of the amount of natural light or illumination light and the irradiation direction may fluctuate. A moving image extraction method for extracting an image of a moving object contained therein as a moving image, a background image acquiring step for acquiring a background image in which the influence of the moving object is eliminated, and the background image acquiring step A difference image generation step that generates a difference image between a background image and the captured image, and a luminance difference value that is a difference for each pixel between the captured image and the immediately preceding captured image that is the captured image captured immediately before the captured image. And calculating a distribution in the entire image of the pixels having the luminance difference value equal to or larger than a predetermined value, and calculating the luminance difference value and the entire image of the pixels having the luminance difference value equal to or larger than the predetermined value. Depending on the evaluation results of the fabric, using the reflection ratio of each of the pixels calculated and reflected ratio calculation step of calculating a reflection ratio of each pixel in the background image, in front Symbol reflecting ratio calculation step of the captured image A background image update step for generating the updated background image by performing weighted averaging of the captured image and the background image with the reflection ratio and (1-the reflection ratio) for each pixel, and the difference image A moving object candidate extracting step of extracting a partial image that may be one moving object as a moving object candidate from the difference image generated by the generating step; and a plurality of moving object candidates extracted by the moving object candidate extracting step. And a moving object candidate narrowing-down step of narrowing down an appropriate moving object candidate from a plurality of moving object candidates .

  The present invention further includes a shake correction step of correcting the shake of the imaging apparatus by comparing the captured image and the background image, and the difference image generating step is corrected by the shake correction step. A difference image that is a difference between the corrected captured image and the background image is generated.

  In the above invention, the present invention further includes a mask region setting step for setting a mask region to be excluded from extraction of the difference image with respect to the captured image, and the difference image generation step includes the mask region setting step. The pixels in the mask area set in step 1 are determined to have no difference, or the difference image generation step is not executed.

  Further, the present invention is characterized in that, in the above-mentioned invention, the difference image generation step generates the difference image using a threshold value corresponding to a state having the weakest contrast under the environment.

  Further, in the present invention according to the above invention, in the moving object candidate extraction step, the moving object extracted by extracting a pixel group of continuous objects as one moving object candidate from the difference image generated in the difference image generation step. A characteristic related to the luminance value of the candidate is evaluated, and a luminance value that is a shadow determination threshold value for determining whether or not the pixel of the object included in the moving object candidate is a shadow is determined from the evaluation result of the characteristic. The shadow is removed from the moving object candidate according to the luminance value which is the shadow determination threshold.

Further, the present invention uses a plurality of captured images arranged in time series in an environment where at least one of the amount of natural light or illumination light and the irradiation direction may fluctuate. A moving image extraction method for extracting an image of a moving object contained therein as a moving image, a background image acquiring step for acquiring a background image in which the influence of the moving object is eliminated, and the background image acquiring step A difference image generation step for generating a difference image between a background image and the captured image, a background image update step for updating the background image from the background image and the captured image, and the difference generated by the difference image generation step A moving object candidate extraction step for extracting a partial image that may be one moving object from the image as a moving object candidate, and the moving object candidate extraction step When said moving object candidate is extracted, as a group element candidates with inclusion relation, enumerating possible combinations of elements candidate in the group as a pattern, element candidate and the time series included in the listed patterns The moving object candidate narrowing-down step that narrows down the appropriate moving object candidates from the plurality of moving object candidates by comparing the moving object candidates extracted from the adjacent captured images based on the similarity with each other. wherein the containing and.

  Also, in the present invention according to the above invention, the moving object candidate extraction step may temporarily extract a pixel group of continuous objects as one moving object candidate from the difference image generated in the difference image generation step, Among the combinations of moving object candidates extracted as candidates, a fused moving object candidate obtained by fusing a plurality of moving object candidates determined to have the same moving object candidate is added to form a moving object candidate group having the inclusion relationship. It is characterized by that.

Further, the present invention uses a plurality of captured images arranged in time series in an environment where at least one of the amount of natural light or illumination light and the irradiation direction may fluctuate. A moving body image extracting apparatus that extracts a moving body image included therein as a moving body image, obtains a background image from which the influence of the moving body is eliminated , and varies luminance values and luminance values for each pixel of the background image. A background image acquisition unit that generates a coefficient of variation indicating the degree of brightness, a luminance difference value for each pixel between the background image acquired by the background image acquisition unit and the captured image, and the calculated luminance difference value by comparing the coefficient of variation, while determining whether to adopt the pixels of the captured image to the differential image which is a difference between the background image and the captured image, a difference image generation means for generating a difference image A background image updating unit that updates the background image from the background image and the captured image, and a partial image that may be one moving object is extracted from the difference image generated by the difference image generating unit as a moving object candidate. A moving object candidate extracting unit; and a moving object candidate narrowing unit that narrows down an appropriate moving object candidate from a plurality of moving object candidates when the moving object candidate extracting unit extracts a plurality of moving object candidates. To do.
Further, the present invention uses a plurality of captured images arranged in time series in an environment where at least one of the amount of natural light or illumination light and the irradiation direction may fluctuate. A moving body image extracting apparatus for extracting a moving body image included therein as a moving body image, the background image acquiring means for acquiring a background image from which the influence of the moving body has been eliminated, and the background image acquiring means acquired by the background image acquiring means A difference image generation unit that generates a difference image between a background image and the captured image, and a luminance difference value that is a difference for each pixel between the captured image and a previous captured image that is a captured image captured immediately before the captured image. And calculating the distribution of the pixels having the luminance difference value not less than a predetermined value in the entire image, and evaluating the distribution of the luminance difference value and the distribution of the pixels having the luminance difference value not less than the predetermined value in the entire image According to the reflection ratio calculation means for calculating the reflection ratio of the captured image to the background image for each pixel and the reflection ratio for each pixel calculated by the reflection ratio calculation means for each pixel. A background image update unit that generates the background image after update by weighted averaging the captured image and the background image with the reflection ratio and (1-the reflection ratio), and the difference image generation unit. A moving object candidate extracting unit that extracts a partial image that may be one moving object from the difference image as a moving object candidate, and a plurality of moving object candidates when the moving object candidate extracting unit extracts a plurality of moving object candidates. A moving object candidate narrowing means for narrowing down an appropriate moving object candidate from candidates is provided.

According to the present invention, can be an imaging environment and set the appropriate threshold without requiring human effort be varied, it extracts a moving object image efficiently with.

FIG. 1 is an explanatory diagram for explaining an overview of background image generation processing, update processing, and difference detection threshold setting processing according to moving object detection according to the present embodiment. FIG. 2 is an explanatory diagram for explaining the processing outline of the entire moving object detection processing according to the present embodiment. FIG. 3 is a block diagram illustrating an internal configuration of the moving object image extraction apparatus according to the present embodiment. FIG. 4 is an explanatory diagram for explaining an outline of difference image generation according to the present embodiment. FIG. 5 is a flowchart showing a difference image generation processing procedure performed by the moving object image extraction apparatus shown in FIG. FIG. 6 is a flowchart of a background image update process performed by the moving object image extraction apparatus shown in FIG. FIG. 7 is a flowchart of a moving object candidate extraction process procedure performed by the moving object image extraction apparatus shown in FIG. FIG. 8 is an explanatory diagram for explaining the concept of the moving object fusion processing shown in FIG. FIG. 9 is a flowchart showing a processing procedure of moving object candidate fusion by the moving image extracting apparatus shown in FIG. FIG. 10 is a flowchart showing a moving object candidate narrowing-down process procedure performed by the moving object image extraction apparatus shown in FIG.

  Exemplary embodiments of a moving object image extraction method and a moving object image extraction device according to the present invention will be described below in detail with reference to the accompanying drawings.

  First, an overview of background image generation processing, update processing, and difference detection threshold setting processing related to moving object detection according to the present embodiment will be described. FIG. 1 is an explanatory diagram for explaining an overview of background image generation processing, update processing, and difference detection threshold setting processing according to moving object detection according to the present embodiment.

  In order to detect a moving object from an input image of a monitoring target area captured by a monitoring camera installed in public transportation, an intersection, or a building, an image of a monitoring target area that does not include a moving object (hereinafter, “background image”) And a threshold for detecting a moving object from the difference between the background image and the input image is set in advance. Since the brightness and contrast of the entire image and the intensity of contrast are affected by changes in the imaging environment such as sunshine, the background image and the threshold are sequentially updated as the imaging environment such as sunshine changes.

  First, an initial background image generation process prepared in advance before receiving an input image to be detected by a moving object will be described. As shown in FIG. 1, the initial background image is generated using, for example, ten images (image # 1 to image # 10) prepared in advance. The images # 1 to # 10 are arbitrary 10 images obtained by capturing the monitoring target area during a period (for example, 10 minutes) in which the environmental change related to image capturing such as sunshine is relatively small. Such image # 1 to image # 10 may include a moving object. However, if a moving object is included at the same position in all 10 sheets, the moving object is regarded as a part of the background.

  Specifically, when generating the initial background image, the variation in the luminance value is examined for each pixel located at the same position in the image # 1 to the image # 10, and the pixel with the large variation is due to the reflection of the moving object. And calculating a variation coefficient indicating the average value of the luminance values of the remaining pixels and the degree of variation of the luminance values, and generating a background image having the calculated luminance value and the variation coefficient of the luminance values as pixel values.

  Next, in the difference image generation process, the difference image # 1 is generated using the initial background image and the input image # 1. Specifically, the luminance value of each pixel of the input image is compared with the luminance value included in the pixel value of the pixel of the initial background image corresponding to the pixel position of each pixel of the input image. When it is less than a predetermined multiple of the variation coefficient included in the pixel value of the pixel of the initial background image, the pixel value of the corresponding pixel of the difference image # 1 is set to zero, and the difference in luminance value is the predetermined multiple of the variation coefficient In the case above, the absolute value of the difference between the two luminance values is set as the pixel value of the pixel. The difference image # 1 is input information for subsequent moving object image extraction processing because there is a possibility that a part having a non-zero pixel value is a pixel that forms a moving object.

  Next, background image update processing will be described. Here, a background image (second generation background image) to be used next time is generated using the initial background image and the input image # 1. The pixel value of each pixel of the second-generation background image includes the luminance value and the variation coefficient of the luminance value, which are the pixel values of the initial background image, the pixel value of the input image # 1, and the reflection ratio for each pixel (hereinafter, (Referred to as “first reflection ratio”). This reflection ratio represents the ratio of input image # 1 to be reflected in the initial background image for each pixel. Specifically, the luminance value of the initial background image corresponding to a certain pixel is v1 (t), the coefficient of variation of the luminance value is σ (t), and the luminance value of the input image # 1 corresponding to the same pixel is v2 (t). If the luminance value of the second generation background image corresponding to the same pixel is v1 (t + 1) and the variation coefficient of the luminance value is σ (t + 1), the luminance value v1 (t + 1) of the second generation background image ) Is calculated by v1 (t + 1) = v1 (t) × (1− (reflection ratio)) + v2 (t) × (reflection ratio), and the variation coefficient σ (t + 1) of the luminance value of the second generation background image Is calculated by σ (t + 1) = σ (t) × (1− (reflection ratio)) + abs (v1 (t) −v2 (t)) × (reflection ratio). Here, abs (x) indicates a function that returns the absolute value of x.

  Since the input image # 1 is an image obtained by capturing the latest monitoring target area, the background portion is an image reflecting an environmental change such as sunshine, but there is a possibility that a moving object is partially included. There is. Considering this, the influence of the environmental change such as sunshine of the input image # 1 is reflected as much as possible in the background image, and the moving object information reflected in the input image # 1 is not reflected as much as possible in the background image.

  Specifically, by calculating the difference in luminance value between the input image and the previous image for each pixel and converting the difference using a predetermined normalization rule, if the difference is large, if the difference is small A second reflection ratio greater than 0 and less than 1 is calculated. In addition, the extent to which the difference between the entire input image and the previous image of the input image spreads over the entire image (hereinafter referred to as “difference spread value”) is calculated, and the difference spread value is determined according to a predetermined normalization rule. By performing conversion using the third reflection ratio, a third reflection ratio that is larger if the difference spread value is larger, smaller if the difference spread value is smaller, and greater than 0 and less than 1 is calculated. The first reflection ratio for each pixel is calculated by multiplying the second reflection ratio and the third reflection ratio for each pixel, and the first reflection ratio and the background image thus calculated are calculated. A second generation background image is generated using the input image.

  Similarly, a difference image # 2 is generated from the second generation background image and the input image # 2, and a third generation background image is generated from the second generation background image and the input image # 2. Similarly, every time an input image is input, the luminance value for each pixel of the background image and the coefficient of variation of the luminance value are updated, so that the background image and the threshold value are sequentially updated with changes in the environment such as sunlight. Is done.

  In this way, based on a plurality of images obtained by imaging the monitoring target region, a background image having a variation coefficient indicating the average luminance value and the degree of variation of the luminance value is generated for each pixel, and the pixels of the background image The luminance value for each pixel is compared with the luminance value of the pixel corresponding to the same position of the input image, and a moving object is extracted using a predetermined multiple of the variation coefficient for each pixel of the background image as a threshold value. In addition, every time an input image is extracted to extract a moving body image, the background image that automatically reflects the background image changes due to environmental changes such as sunshine to the background image is updated and set, so the imaging environment changes. Even if human labor is not required, an appropriate threshold value can be set and a moving body image can be efficiently extracted.

  Next, an overview of the entire moving object detection process according to the present embodiment will be described. FIG. 2 is an explanatory diagram for explaining the processing outline of the entire moving object detection processing according to the present embodiment.

  As shown in FIG. 1, the initial background image generation process is a process of generating an initial background image shown in (b) with 10 images captured from the monitoring target area shown in (a) as input. In the initial background image generation process, even if moving images are included in the 10 images (a) that are input as shown in FIG. 2, the influence of the moving objects is eliminated and the background image data shown in (b) Can be generated.

  In the difference image generation process, the luminance value of the background image shown in (b) and the input image shown in (c) are compared for each pixel, and the difference in luminance value is a predetermined multiple of the variation coefficient of the pixel of the background image. This is a process of extracting pixels having a difference equal to or greater than a threshold value. By this difference image generation processing, pixels corresponding to a background image that is not a moving object are excluded, and moving object candidates that are determined to be different from the background image are extracted. Further, in this embodiment, the moving object candidate is extracted even in the night where the contrast of the light and darkness is low in order to extract the moving object candidate from the input image obtained by capturing the same monitoring target area under various conditions regardless of the time, weather, and season. A threshold value sensitive to contrast is set so that can be extracted.

  The background image update process is a process for updating the background image shown in (b) using the background image shown in (b) and the input image shown in (c) as input information. Since the background image is affected by changes in the environment such as sunshine, the effect of the change in the environment is sequentially reflected in the background image even after being generated once by the initial background image generation processing. In the background image update process, even if a moving object is included in the input image shown in (c), the influence of the moving object is eliminated and the background image shown in (b) is updated.

  In consideration of the fact that the shadow of the moving object is extracted as a difference from the difference image generated by the difference image generation process, as shown in the difference image of FIG. A shadow removal process is performed from the difference image to generate a shadow-removed difference image shown in (e). This is because, by setting a threshold sensitive to contrast so that moving object candidates can be extracted even at night, shadows are extracted as a difference in a daytime image with strong contrast between light and dark.

  Next, the moving object candidate extraction process is a process for extracting the moving object candidate shown in (f) from the difference image after shadow removal shown in (e). Specifically, in the difference image after shadow removal shown in (e), adjacent difference pixels may be identified as a single moving object by a rectangle, and moving object candidates extracted as separate independent moving object candidates may be the same moving object. In consideration of the characteristics, a moving object candidate obtained by fusing a plurality of moving object candidates is extracted, and candidates having a predetermined size or less are excluded. The moving object candidate shown in (f) is generated from the difference image after shadow removal shown in (e), and indicates that four moving object candidates Obj1 to Obj4 have been extracted. The moving object candidate Obj1 is a moving object candidate obtained by fusing the moving object candidate Obj2 and the moving object candidate Obj3. At this stage, the moving object candidate Obj1 is one moving object or the moving object candidate Obj2 and the moving object candidate Obj3 are independent moving objects. Therefore, both candidates are included.

  Next, in the moving object candidate narrowing-down process, the moving object candidates shown in (f) include moving object candidates Obj1 obtained by fusing moving object candidates Obj2 and moving object candidates Obj3, moving object candidates Obj2 and moving object candidates Obj3 before fusion. Therefore, by determining whether the moving object candidate Obj1 is one moving object or whether the moving object candidate Obj2 and the moving object candidate Obj3 are separate independent moving objects, any inappropriate moving object candidates are deleted, This is a process for determining a moving object. In the moving object candidate narrowing-down process of FIG. 2, the moving object candidate Obj2 and the moving object candidate Obj3 are determined as independent separate moving objects from the moving object candidate shown in (f), and the moving object candidate Obj1 is deleted. This indicates that the moving object candidate has been determined. The moving object candidate narrowing-down process includes information on the moving object candidate shown in (f) extracted in the moving object candidate extraction process, and moving object candidate information extracted from the immediately preceding two images shown in (g) and (h). , (I) and (j) using the moving object candidate information extracted from the immediately following two images, the moving object candidate Obj2 and the moving object candidate Obj3 are independent moving objects, or the moving object candidate Obj1 is one moving object. Determine what is happening.

  The background image shown in (b), the difference image shown in (d), the difference image after shadow removal shown in (e), the moving object candidate shown in (f), and the two images immediately before shown in (g) and (h) For the convenience of explanation, the captured moving image candidate shown in (a), the moving image candidate of the two images immediately after shown in (i) and (j), and the narrowed moving object candidate shown in (k) Although it is expressed in the same manner as the image data of the input image shown in (c), the data format and data holding items are different from the captured image shown in (a) and the input image shown in (c). The background image shown in (b) has a luminance value for each pixel and a variation coefficient of the luminance value. In the difference image shown in (d), the pixel value of the portion where the difference is less than the predetermined value is zero, and the pixel value of the portion where the difference is the predetermined value or more is the absolute value of the difference between the luminance value of the background image and the luminance value of the input image It is data which has. The data format of the difference image after shadow removal shown in (e) is the same as that of the difference image shown in (d). The candidate moving object shown in (f), the candidate moving object of the immediately preceding two images shown in (g) and (h), the candidate moving object of the two immediately following images shown in (i) and (j), and (k) The narrowed moving object candidate to be shown has the position and size of a rectangle on the input image for each moving object candidate surrounded by a rectangle, pixel information of the moving object within the rectangle, and the like.

  Next, the internal configuration of the moving object image extraction device 30 according to the present embodiment will be described. FIG. 3 is a block diagram illustrating an internal configuration of the moving object image extraction device 30 according to the present embodiment. As shown in FIG. 3, the moving object image extraction device 30 is connected to the monitoring camera 10 via the network 20. In the present embodiment, an image captured by the monitoring camera 10 via the network 20 is input to the moving object image extraction device 30, but the present invention is not limited to this. An image read from an external medium such as a CD, DVD, USB memory, SD memory, and Compact Flash (registered trademark) memory, an image obtained by capturing an image displayed on a PC screen with a capture board, and the like are stored in the moving object image extraction device 30. It may be input.

  The moving body image extraction device 30 includes an I / F unit 31, a display unit 32, an input unit 33, a storage unit 34, and a control unit 35. The I / F unit 31 is an interface unit for receiving monitoring image data captured by the monitoring camera 10 via the network 20. The display unit 32 is a liquid crystal panel, a display device, or the like, and the input unit 33 is a keyboard, a mouse, or the like.

  The storage unit 34 is a storage device including a flash memory or a hard disk device, and includes accumulated image data 34a, background image data 34b, processing parameters 34c, difference image data 34d, and moving object candidate data 34e. The accumulated image data 34a is image data captured by the monitoring camera 10, and is recorded in association with the date and time.

  The background image data 34b is background image data excluding moving objects in the area captured by the monitoring camera 10, and is used when generating a differential image from the input image captured by the monitoring camera 10. The background image data 34b is generated by an initial background image generation unit 35b described later, and is sequentially updated by a background image update unit 35d described later. The background image data 34b includes an image width, an image height, pixel information, and the like. The pixel information includes information on the number of pixels calculated by (image width) × (image height), and the information on each pixel includes a variation coefficient indicating the degree of variation in the luminance value of the pixel and the luminance value. have.

  The processing parameter 34 c is a setting parameter for extracting a moving object candidate image from an image captured by the monitoring camera 10. The processing parameter 34c is, for example, mask area information. The mask area information is a setting of an area not subject to motion detection in the input image. When calculating the difference between the input image and the background image, the luminance values of the pixels corresponding to this mask area are not compared, and the pixel value of the difference image is fixedly set to zero meaning no difference. Alternatively, the difference calculation process for this mask area may not be executed.

  The difference image data 34d is difference data between the input image captured by the monitoring camera 10 and the background image, and corresponds to the difference image (d) in FIG.

  The moving object candidate data 34e is moving object candidates or moving object information extracted based on the difference image data 34d. The moving object candidate data 34e includes information on all moving object candidates extracted from the input image. For example, each extracted moving object candidate has a position on the upper left image of the rectangle of the moving object candidate, a width and a height of the rectangle, and the like.

  The control unit 35 is a control unit that controls the moving image extracting apparatus 30 as a whole, and includes an image acquisition unit 35a, an initial background image generation unit 35b, a difference image generation unit 35c, a background image update unit 35d, a shadow removal unit 35e, and moving object candidates. It has the extraction part 35f and the moving body candidate narrowing-down part 35g. Actually, programs corresponding to these functional units are stored in a ROM or non-volatile memory (not shown), and each program is executed by a CPU (Central Processing Unit), thereby executing a process corresponding to each program. It will be.

  The image acquisition unit 35a obtains the image data captured by the monitoring camera 10 via the network 20 and the I / F unit 31, and stores it in the accumulated image data 34a. Or the form which acquires reproduction signals, such as a video recorder, may be sufficient.

  The initial background image generation unit 35b performs the initial background image generation process described with reference to FIG. Specifically, a plurality of images captured by the monitoring camera 10 at the time of activation of the present apparatus within a predetermined period (for example, 10 minutes) in which environmental changes related to image capturing such as sunshine are relatively small are used as background information. Image data 34b is generated. The initial background image generation unit 35b generates a background image that excludes the influence of the moving object even if the moving object is reflected in any 10 images captured within a predetermined period (for example, 10 minutes). Specifically, the variation of the luminance value is examined for each pixel of the ten images, and the pixel having a large variation is considered to be a reflection of the moving object, and the pixel is excluded and the remaining pixel is used for each pixel. A variation coefficient indicating the average luminance value and the degree of variation of the luminance value is calculated, and a background image having the luminance value and the variation coefficient as a pixel value is stored in the background image data 34b.

  The difference image generation unit 35c receives the input image captured by the monitoring camera 10 after the initial background image generation unit 35b generates the background image data 34b, and compares the background image data 34b with the input image to obtain the difference image data. 34d is generated. A detailed processing procedure will be described later.

  The background image update unit 35d updates the background image data 34b based on the latest background image information obtained from the input image. However, since the moving object may be reflected in the input image, the background image update unit 35d performs the update process of the background image data 34b by eliminating the influence of the moving object being reflected. A detailed processing procedure will be described later.

  The shadow removing unit 35e removes the shadow by setting the difference value of the portion determined to be the influence of the shadow from the difference image data 34d generated by the difference image generating unit 35c to zero. Specifically, a group of adjacent difference pixels is set as a moving object candidate, and a threshold value for determining a shadow for each moving object candidate is calculated according to a predetermined rule. A pixel whose difference value for each pixel is equal to or less than the threshold value is a shadow. And the pixel value of the difference image determined to be a shadow is set to zero.

  The moving object candidate extraction unit 35f extracts a difference pixel group in which adjacent difference pixels are grouped as one moving object candidate from the difference image data 34d generated by the difference image generation unit 35c, and registers it in the moving object candidate data 34e. In addition, the moving object candidate extraction unit 35f extracts adjacent difference pixels as one moving object candidate from the difference image data 34d after shadow removal generated by the shadow removing unit 35e, and extracts a plurality of extracted independent moving object candidates. In consideration of the possibility that the moving object candidates are the same moving object, moving object candidates obtained by fusing a plurality of moving object candidates are also extracted, and then candidates having a predetermined size or less are excluded. A detailed processing procedure will be described later.

  The moving object candidate narrowing-down unit 35g is a process of deleting an inappropriate moving object candidate included in the moving object candidate data 34e registered by the moving object candidate extracting unit 35f and determining an appropriate moving object candidate. The moving object candidate narrowing-down unit 35g determines an improper moving object candidate using information on moving object candidates extracted from images before and after the time series of the input image, and deletes the improper moving object candidate from the moving object candidate data 34e. To do. A detailed processing procedure will be described later.

  Next, an outline of difference image generation according to the present embodiment will be described. FIG. 4 is an explanatory diagram for explaining an outline of difference image generation according to the present embodiment.

  The difference image is generated using the background image data 34b, which is the background image, and the input image. The background image data 34b has a luminance value and a variation coefficient indicating the degree of variation of the luminance value for each pixel. On the other hand, the input image has only a luminance value for each pixel. From the information for each pixel of the background image data 34b, the luminance value of the pixel corresponding to the same position if the moving object does not enter is a distribution in which the luminance value of the corresponding pixel of the background image data 34b is averaged and the variation coefficient varies. It is assumed that

  On the other hand, if the determination criterion for detecting the difference is a pixel in which the absolute value of the luminance value difference for each pixel has a difference that is twice or more the variation coefficient, the luminance value of the pixel of the background image is the v1 variation coefficient. Is σ and the luminance value of the corresponding pixel of the input image is v2, if v1-2σ <v2 <v1 + 2σ, it is determined that there is no influence by the moving object, and the difference value for each pixel of the difference image data 34d Set to zero. Further, when v2 ≧ 1 + 2σ or v2 ≦ v1-2σ, it is determined that the difference is due to the moving object, and the absolute value of (v1−v2) is set as the pixel value of the difference image data 34d.

  Next, a difference image generation processing procedure by the moving object image extraction device 30 shown in FIG. 1 will be described. FIG. 5 is a flowchart showing a processing procedure of difference image generation by the moving object image extraction device 30.

  First, the difference image generation unit 35c converts the input image captured by the monitoring camera 10 into a gray multi-tone image (step S201). Further, the difference image generation unit 35c performs noise cut processing by applying a predetermined Gaussian filter to the input image (step S202).

  Next, the difference image generation unit 35c determines whether or not the input image has a deviation of a predetermined number of pixels or less in the vertical, horizontal, or diagonal directions compared to the background image due to the shaking of the monitoring camera 10. If the occurrence of deviation is determined, the deviation of the input image is corrected (step S203).

  Next, one pixel in the input image is extracted (step S204), and it is determined whether or not the extracted pixel is within the mask area defined by the processing parameter 34c (step S205). If the pixel extracted in step S204 is within the mask region defined by the processing parameter 34c (step S205; Yes), zero is written in the pixel difference value for the pixel in the difference image data 34d (step S209), and step The process proceeds to S210.

  If the pixel extracted in step S204 is outside the mask area defined by the processing parameter 34c (step S205; No), the absolute value of the difference between the luminance value of the pixel of the input image and the luminance value of the pixel of the background image Is calculated (step S206). When the absolute value of the calculated difference between the brightness values is less than a threshold value (hereinafter referred to as a difference determination threshold value) that is a predetermined number times the variation coefficient corresponding to the pixel of the pixel information of the background image data 34b (step S207; Yes) ), Zero is written in the pixel difference value for the pixel in the difference image data 34d (step S209), and the process proceeds to step S210.

  If the absolute value of the calculated difference in luminance value is equal to or greater than the difference determination threshold (step S207; No), the absolute value of the calculated difference in luminance value is written in the pixel difference value for the pixel in the difference image data 34d. (Step S208), the process proceeds to Step S210.

  Next, it is determined whether or not the processing from step S204 to step S209 has been performed for all the pixels (step S210), and there is a pixel for which the processing from step S204 to step S209 has not been performed. In (Step S210; No), the process proceeds to Step S204. Further, when the processing of step S204 to step S209 has been performed for all the pixels (step S210; Yes), the background image update unit 35d obtains the background image data 34b from the input image and the background image data 34b. Update (step S211), the process is terminated. The detailed update procedure of the background image data 34b in step S211 will be described later.

  Next, a background image update processing procedure by the moving object image extraction device 30 shown in FIG. 5 will be described. FIG. 6 is a flowchart illustrating a processing procedure of background image update performed by the moving image extracting apparatus 30.

  The background image update unit 35d extracts one pixel to be processed from the previous input image and the current input image (step S301), and calculates the luminance values of the extracted pixels of the previous input image and the current input image. The absolute value of the difference is calculated (step S302). Specifically, the pixel of the previous input image is f (i, j) (0 ≦ i <x, 0 ≦ j <y), and the pixel of the current input image is g (i, j) (0 ≦ i <x , 0 ≦ j <y), and if the difference image pixel between the previous input image and the current input image is Δg (i, j) (0 ≦ i <x, 0 ≦ j <y), then g (i, j ) −f (i, j) is calculated and stored in Δg (i, j). When the value of Δg (i, j) calculated in this way is large, there is a high possibility that the pixel is affected by a moving object.

  Next, the background image update unit 35d has α (i, j) which is a reflection ratio of the current input pixel value g (i, j) to the background image data 34b from the value of Δg (i, j) for the pixel. Is determined (step S303). Specifically, by converting Δg (i, j) using a predetermined normalization rule, α (i, j) greater than 0 and less than 1 is smaller if the difference is large, large if the difference is small. Is calculated.

  If the processing in steps S301 to S303 has not been completed for all pixels (step S304; No), the process proceeds to step S301. When the processing of step S301 to step S303 has been completed for all the pixels (step S304; Yes), the reflection ratio α (i, j) corresponding to all the pixels has been determined. The process proceeds to step S305.

Next, the background image update unit 35d extracts pixels in which the absolute value of the difference in luminance value of each pixel in the difference image Δg (i, j) is equal to or greater than a predetermined value (step S305). A barycentric position of a pixel group having an absolute value equal to or larger than a predetermined value is calculated (step S306), and an average distance ave_d from the barycentric position to all the pixels extracted in step S305 is calculated (step S307). Furthermore, the background image update unit 35d calculates a ratio ρ of the number of pixels in which the absolute value of the difference extracted in step S305 is equal to or larger than a predetermined value to the total number of pixels of the input image (step S308).
When the average distance ave_d from the center of gravity and the ratio ρ of the total number of pixels of the input image whose absolute value of the difference is equal to or greater than a predetermined value are large, pixels having a difference in luminance value greater than or equal to the predetermined value are scattered over a wide range. Therefore, it can be assumed that it is not the influence of the moving object that causes local changes, but the influence of changes in the overall background image such as sunlight.

  The background image update unit 35d calculates the average distance ave_d from the barycenter of the pixels with the difference in luminance value greater than or equal to the predetermined value calculated in step S307 and the input image having the number of pixels in which the absolute value of the difference calculated in step S308 is equal to or greater than the predetermined value A product of the ratio ρ with respect to the entire pixels is obtained, and based on this value, a reflection ratio β of greater than 0 and less than 1 to the background image data 34b of the entire current input image is determined (step S309).

  The background image update unit 35d reflects the reflection ratio α (i, j) of the input image for each pixel determined in steps S301 to S303 to the background image data 34b and the background image data 34b for the entire input image determined in step S309. From the reflection ratio β, the luminance value h1 (i, j) (0 ≦ i <x, 0 ≦ j <y) for each pixel of the background image data 34b and the variation coefficient h2 (i, j) for each pixel ( 0 ≦ i <x, 0 ≦ j <y) is updated (step S310). Specifically, the luminance value for each pixel after the change is h1 (i, j) = h1 (i, j) × (1−α (i, j) × β) + g (i, j) × α ( i, j) × β. The variation coefficient for each pixel is h2 (i, j) = h2 (i, j) × (1−α (i, j) × β) + abs (g (i, j) −h1 (i, j)) × Calculated by α (i, j) × β. Note that abs (x) represents a function that returns the absolute value of x.

  In the present embodiment, an example in which the reflection ratio α (i, j) for each pixel and the reflection ratio β for the entire image are calculated from one immediately preceding image and the input image has been described. The reflection ratio α (i, j) for each pixel with respect to each previous image and the reflection ratio β of the entire image are calculated from the immediately preceding image and the input image, and the reflection ratio α (i, j corresponding to a plurality of previous images is calculated. ) May be the reflection ratio for each pixel, and the maximum value of the reflection ratio β corresponding to each immediately preceding image may be the reflection ratio of the entire image.

  Next, a processing procedure of moving object candidate extraction by the moving object image extraction device 30 shown in FIG. 2 will be described. FIG. 7 is a flowchart showing a processing procedure of moving object candidate extraction by the moving object image extraction device 30.

  The moving object candidate extraction unit 35f sets a difference pixel group obtained by collecting adjacent difference pixels from the difference image data 34d, which is the difference image shown in (a), generated by the difference image generation unit 35c from the background image data 34b and the input image. The first stage moving body candidate shown in (b) is extracted as one moving body candidate, and information relating to the pixels constituting the moving body candidate is registered in the moving body candidate data 34e for each extracted moving body candidate (step S401). The first stage moving body candidate shown in (b) shows an example in which seven moving body candidates Obj1 to Obj7 are extracted by the moving body candidate extraction process (first stage) in step S401.

  Next, the shadow removal unit 35e performs shadow removal processing for each of the extracted moving object candidates Obj1 to Obj7 shown in the first-stage moving object candidate in (b) (step S402), and eliminates the difference pixels determined as shadows. The difference image shown in (c) after the shadow removal is generated, and the content of the difference image data 34d is updated (step S403). The difference image after shadow removal shown in (c) shows an example in which the shadow included in the moving object candidates Obj2, Obj3 and Obj5 shown in the first stage moving object candidate shown in (b) is removed.

  Next, the moving object candidate extraction unit 35f performs a process similar to that in step S401 from the difference image after shadow removal illustrated in (c) stored in the difference image data 34d, thereby obtaining a moving object candidate from which shadow has been removed. The third stage moving object candidate shown in (d) is extracted, and the extracted moving object candidate is registered in the moving object candidate data 34e (step S404). The third stage moving body candidate shown in (d) is an example in which seven moving body candidates Obj1, Obj12, Obj13, Obj4, Obj15, Obj6 and Obj7 are extracted by the moving body candidate re-extraction process (third stage) in step S404. Is shown. In addition, the third stage moving body candidate shown in (d) is compared with the first stage moving body candidate shown in (b), and the moving body candidates Obj2, Obj3 and the first stage moving body candidate shown in (b) included shadows. An example is shown in which Obj5 is extracted as moving object candidates Obj12, Obj13, and Obj15 from which shadows have been removed as a result of the shadow removal processing in step S402.

  The third stage moving object candidate shown in (d) extracted in step S404 may actually have one moving object extracted as a plurality of moving object candidates. In consideration of this, the moving object candidate extraction unit 35f extracts moving object candidates that are determined to have the same moving object according to a predetermined rule, and selects moving object candidates obtained by fusing a plurality of moving object candidates. The fourth stage moving object candidate shown in the included (e) is generated and reflected in the moving object candidate data 34e (step S405). A detailed processing procedure of this processing will be described later. The fourth stage moving object candidate shown in (e) is determined by the moving object candidate fusion process (fourth stage) in step S405 to be likely to be the same moving object, and shown in (d). In this example, a moving object candidate Obj20 obtained by merging moving object candidates Obj12 and Obj13 is added to the moving object candidate indicated by the third stage moving object candidate.

  Next, the moving object candidate extraction unit 35f compares the upper limit value and the lower limit value of the size on the image to be detected from the fourth stage moving object candidate shown in (e), and the size exceeds the upper limit value. The moving object candidate that falls below the lower limit is excluded and reflected in the moving object candidate data 34e (step S406). The fifth stage moving object candidate shown in (f) is obtained by moving object candidates Obj1, Obj4, Obj6 and Obj7 existing in the fourth stage moving object candidate shown in (e) by the moving object candidate narrowing process (fifth stage) in step S406. An example in which the detection target moving object is deleted because it falls below the lower limit value of the size of the moving object to be detected is shown.

  Next, the concept of the moving object candidate fusion process shown in FIG. 7 will be described. FIG. 8 is an explanatory diagram for explaining the concept of moving object candidate fusion processing. In the description of the concept of the moving object candidate fusion process in FIG. 8, for the sake of convenience, in the moving object candidate re-extraction process (step S404) shown in FIG. 7, three moving object candidates Obj1 to Obj3 are extracted as shown in FIG. The explanation will be made assuming the case.

  First, a fusion rate between all moving object candidates shown in (a) is calculated, and a moving object candidate fusion rate table T1 is generated. Regarding the two moving object candidates i and moving object candidates j, the fusion rate of moving object candidates j with respect to moving object candidates i indicates the ratio of difference pixels of moving object candidates j within the rectangular range of moving object candidates i. The moving object candidate fusion rate table is a table of n rows and n columns when the number of moving object candidates is n, and an element of i row and j column indicates a fusion rate of moving object candidate j with respect to moving object candidate i. The moving object candidate fusion rate table T1 indicates that all elements of the fusion rate between moving object candidates are “0%” because the rectangles of all moving object candidates do not overlap the difference pixels of other moving object candidates. Yes. Since the fusion of moving object candidates is performed when the fusion rate is equal to or higher than a predetermined value (for example, 50%), the moving object candidate fusion process is not performed here.

  Next, as shown in (b), the rectangular range for each moving object candidate is enlarged by a predetermined amount (for example, several pixels in the vertical and horizontal directions). A fusion rate between moving object candidates is calculated within the expanded rectangular range of each moving object candidate, and a moving object candidate fusion rate table T2 is generated. The moving object candidate fusion rate table T2 indicates that the difference pixel of the moving object candidate Obj1 is included in the rectangular range of the moving object candidate Obj2, and the fusion rate between moving object candidates of other combinations is “0%”. ing. Since the fusion rate of the moving object candidates is not equal to or higher than a predetermined value (for example, 50%), the moving object candidate fusion process is not performed.

  Next, as shown in (c), the rectangular range for each moving object candidate is enlarged by a predetermined amount, the fusion rate between moving object candidates is calculated in the expanded rectangular range for each moving object candidate, and the moving object candidate fusion rate table T3 is generated. In the moving object candidate fusion rate table T3, the difference pixel of the moving object candidate Obj1 is included in the rectangular range of the moving object candidate Obj2, and the difference pixel of the moving object candidate Obj3 is included in “5%”. The fusion rate is “0%”. Here, since the fusion rate of the moving object candidate Obj2 and the moving object candidate Obj1 is “55%” and the moving object candidate has a fusion rate equal to or higher than a predetermined value (for example, 50%), the moving object candidate Obj1 and the moving object candidate Obj2 are Add Obj4 which is a fused moving object candidate.

  Next, the original moving object candidates Obj1 and Obj2 are excluded from the moving object candidates shown in (c) and the moving object candidate Obj4 generated by the fusion is added. All moving object candidates shown in (d) On the other hand, the fusion rate between moving object candidates is calculated to generate a moving object candidate fusion rate table T4. The moving object candidate fusion rate table T4 indicates that the difference pixel of the moving object candidate Obj3 is included in the rectangular range of the moving object candidate Obj4 and the fusion rate between moving object candidates of other combinations is “0%”. ing. Therefore, since the fusion rate of moving object candidates is not equal to or higher than a predetermined value (for example, 50%), the moving object candidate fusion process is not performed.

  Thereafter, the rectangular range of the moving object candidates is expanded up to a predetermined maximum number of moving object candidate rectangles, the fusion rate between moving object candidates is calculated, and when the fusion rate is a predetermined value (for example, 50%) or more, the moving object By repeating the candidate fusion, a fused moving object candidate is generated.

  If moving object candidates are merged in the moving object candidate fusion process described above, the fused moving object candidates are excluded from the moving object candidate fusion rate table, but the fused moving object candidates are not excluded from the moving object candidates. . Since the moving object candidate fusion process described here is a process for extracting a moving object candidate that may be one moving object, the moving object candidates are not narrowed down in this phase.

  Next, a processing procedure for moving object candidate fusion by the moving object image extraction device 30 shown in FIG. 7 will be described. FIG. 9 is a flowchart illustrating a moving object candidate fusion processing procedure performed by the moving object image extraction device 30.

  First, the moving object candidate extraction unit 35f initializes the number of expansions of the moving object candidate rectangle range to zero (step S501), and re-extracts the moving object candidate from step S404 of FIG. Calculate the fusion rate between all the moving object candidates of the extracted moving object candidates, create a moving object candidate fusion rate table (step S502), and have the maximum fusion rate in the created moving object candidate fusion rate table Is specified (step S503).

  With reference to the value of the element having the maximum fusion rate in the moving object candidate fusion rate table extracted in step S503, it is determined whether or not the maximum value of the fusion rate is equal to or greater than a fusion implementation reference value (for example, 50%). This is performed (step S504). If the maximum value of the fusion rate in the moving object candidate fusion rate table is greater than or equal to a reference value (for example, 50%) (step S504; Yes), the moving object candidate corresponding to the element of the maximum fusion rate in the moving object candidate fusion rate table is selected. A new moving object candidate is generated by merging and added to the moving object candidate data 34e (step S505).

  In addition, the moving object candidate extraction unit 35f excludes the moving object candidates merged from the fusion rate calculation target (step S506), and adds the moving object candidates generated by the fusion added to the moving object candidates. Create (step S507), and proceed to step S503.

  When the maximum value of the fusion rate in the moving object candidate fusion rate table is less than a reference value (for example, 50%) (step S504; No), the number of expansions of the moving object candidate rectangle range is equal to the predetermined maximum number of moving object candidate rectangles. If the number of expansions of the moving object candidate rectangle range is equal to the predetermined maximum number of moving object candidate rectangle expansions (step S508; Yes), the process is terminated. When the number of expansions of the moving object candidate rectangle range is not equal to the predetermined maximum number of moving object candidate rectangle expansions (step S508; No), the moving object candidate rectangle range included in the moving object candidate fusion rate table is expanded according to a predetermined rule. (Step S509).

  Further, the moving object candidate extraction unit 35f updates the moving object candidate fusion rate table in accordance with the change of the moving object candidate rectangular range (step S510), counts up the number of expansions of the moving object candidate rectangular range (step S511), The process proceeds to step S503.

  Next, a moving object candidate narrowing-down process procedure by the moving object image extraction device 30 shown in FIG. 2 will be described. FIG. 10 is a flowchart illustrating a moving object candidate narrowing-down process performed by the moving object image extraction device 30.

  The moving object candidate narrowing-down unit 35g selects moving object candidates such that moving object candidates Obj2 and Obj3 are included in the moving object candidate Obj1 in the group 1 shown in (a) from the moving object candidates extracted in the moving object candidate data 34e. All the similar groups are extracted with the moving object candidate group having the inclusion relation in the rectangular range as one group (step S601).

  Next, the moving object candidate narrowing-down unit 35g has a possible combination of independent moving objects (hereinafter referred to as “pattern”) from the inclusion relationship of moving object candidates for one unprocessed group extracted in step S601. Is created (step S602). In the group 1 shown in (a), there are three moving object candidates Obj1 to Obj3, and the moving object candidates Obj2 and Obj3 are included in the moving object candidate Obj1, and by the pattern creation process for each group in step S602, From group 1, pattern 1 in which moving object candidates Obj2 and Obj3 shown in (b) are independent moving objects, and moving object candidates Obj1 generated by the fusion of moving object candidates Obj2 and Obj3 shown in (c) are 1 An example in which a pattern 2 that is one moving object is created is shown.

  Next, the moving object candidate narrowing down unit 35g evaluates the similarity between the moving object candidate included in the pattern and the moving object candidate included in each of the two images before and after the input image for the unevaluated pattern created in step S602. The pattern is evaluated from the similarity evaluation of each moving object candidate (step S603). The pattern evaluation in step S603 in FIG. 10 includes one pattern generated in step S602, moving object candidates for the immediately preceding two input images shown in (d) and (e), and (f) and (g). The similarity of the pattern is evaluated using the moving object candidates of the two input images immediately after the input as input information. For example, similarity is evaluated based on distance, size, and the like.

  The moving object candidate narrowing-down unit 35g determines whether or not the evaluation of the pattern in step S603 has been completed for all the patterns created in step S602 (step S604), and if an unevaluated pattern remains (step S604). In step S604; No), the process proceeds to step S603. If all patterns have been evaluated (step S604; Yes), a pattern with high evaluation is determined among the patterns of the group created in step S602, and moving object candidates belonging to other patterns are determined from the moving object candidate data 34e. It deletes (step S605). The pattern determination process in step S605 of FIG. 10 compares the evaluation result of pattern 1 shown in (b) with the evaluation result of pattern 2 shown in (c), and moving object candidates of group 1 are shown in (h). This shows that the pattern 1 is determined.

  Next, the moving object candidate narrowing-down unit 35g determines whether or not the processing for all the groups extracted in step S601 has been completed (step S606), and if there is a group that has not yet been processed (step S606; No), go to step S602. If the processing for all the groups has been completed (step S606; Yes), the validity of the moving object candidate is evaluated by correlation calculation for all moving object candidates in the moving object candidate data 34e. The moving object candidate determined not to be deleted from the moving object candidate data 34e (step S607), and the process ends.

  As described above, in this embodiment, a background image having a variation coefficient indicating a luminance value and a degree of variation of each pixel is automatically generated by eliminating the influence of moving objects from a plurality of images, The difference image extraction between the background image and the input image is performed by using a predetermined multiple of the variation coefficient of each pixel as a threshold value for the difference in luminance value for each pixel, and each time the input image is input, the moving object reflected in the input image is extracted. Since it is configured to automatically reflect changes in the background image due to environmental changes such as sunshine while eliminating the influence, an appropriate threshold value can be set without requiring human labor even if the imaging environment changes, and moving image Can be extracted efficiently.

  In the above-described embodiment, the update process of the background image is performed according to a procedure different from the initial background image generation. However, the present invention is not limited to this, and the initial background is updated when the background image is updated. The background image may be generated in the same procedure as the image.

  Each configuration illustrated in the above-described embodiment is functionally schematic and does not necessarily need to be physically configured as illustrated. That is, the form of distribution / integration of each device is not limited to that shown in the figure, and all or a part thereof may be functionally or physically distributed / integrated in an arbitrary unit according to various loads or usage conditions. Can be configured.

  As described above, the moving object image extraction method and the moving object image extraction device according to the present invention set an appropriate threshold without requiring human labor even when the imaging environment changes, and efficiently extract moving object images. Suitable for that.

DESCRIPTION OF SYMBOLS 10 Surveillance camera 20 Network 30 Moving body image extraction apparatus 31 I / F part 32 Display part 33 Input part 34 Storage part 34a Accumulated image data 34b Background image data 34c Processing parameter 34d Difference image data 34e Moving body candidate data 35 Control part 35a Image acquisition part 35b initial background image generation unit 35c difference image generation unit 35d background image update unit 35e shadow removal unit 35f moving object candidate extraction unit 35g moving object candidate narrowing-down unit T1 to T4 moving object candidate fusion rate table

Claims (11)

Using a plurality of captured images arranged in time series in an environment where at least one of the amount of natural light or illumination light and the irradiation direction may fluctuate, a moving object included in the captured image A moving image extraction method for extracting an image as a moving image,
Obtaining a background image in which the influence of the moving object is eliminated , and generating a variation coefficient indicating a luminance value and a variation degree of the luminance value for each pixel of the background image; and
A luminance difference value for each pixel between the background image acquired by the background image acquisition step and the captured image is calculated, and the calculated luminance difference value and the variation coefficient are compared to calculate the background image and the imaging A difference image generation step of generating a difference image while deciding whether or not to adopt pixels of the captured image as a difference image that is a difference from the image;
A background image update step of updating the background image from the background image and the captured image;
A moving object candidate extraction step of extracting a partial image that may be one moving object as a moving object candidate from the difference image generated by the difference image generating step;
A moving object image extracting method comprising: a moving object candidate narrowing-down step for narrowing down an appropriate moving object candidate from a plurality of moving object candidates when a plurality of moving object candidates are extracted in the moving object candidate extracting step.   The background image acquisition step analyzes the luminance value distribution of the plurality of captured images for each pixel corresponding to the same position for a plurality of captured images with different imaging timings, and is specific to the luminance value distribution. 2. The moving object image extraction method according to claim 1, wherein a luminance value of a pixel in the background image is calculated by excluding a luminance value indicating a value and using a luminance value remaining without being excluded. Using a plurality of captured images arranged in time series in an environment where at least one of the amount of natural light or illumination light and the irradiation direction may fluctuate, a moving object included in the captured image A moving image extraction method for extracting an image as a moving image,
A background image acquisition step of acquiring a background image in which the influence of the moving object is eliminated;
A difference image generation step for generating a difference image between the background image acquired by the background image acquisition step and the captured image;
A luminance difference value that is a difference for each pixel between the captured image and the immediately preceding captured image that is the captured image captured immediately before the captured image is calculated, and a distribution of the pixels in which the luminance difference value is equal to or greater than a predetermined value in the entire image The reflection ratio for calculating the reflection ratio for each pixel in the background image of the captured image in accordance with the evaluation result of the distribution of the luminance difference value and the distribution of the entire pixels of the pixels having the luminance difference value equal to or greater than a predetermined value. A calculation step ;
Using said reflecting proportion for each pixel calculated in the previous SL reflecting proportion calculating step, a weighted average that out with the captured image for each pixel the and the said reflecting proportion background image and (1-the reflecting proportion) A background image update step for generating the background image after update by:
A moving object candidate extraction step of extracting a partial image that may be one moving object as a moving object candidate from the difference image generated by the difference image generating step;
When a plurality of moving object candidates are extracted by the moving object candidate extraction step, a moving object candidate narrowing step for narrowing down the appropriate moving object candidates from the plurality of moving object candidates;
Moving body image extraction method characterized by containing. Further comprising a shake correction step of correcting the shake of the imaging device by comparing the captured image and the background image;
The difference image generation step, moving body according to any one of claims 1-3, characterized in that to generate the difference image which is a difference between the corrected captured image and the background image corrected by the shake correction step Image extraction method. A mask area setting step of setting a mask area to be excluded from extraction of the difference image with respect to the captured image;
The difference image generation step, any claim 1-4, characterized in that said for the pixels in the mask area setting mask area set in step determines that no difference, or does not execute the difference image generation step The moving body image extraction method according to claim 1. The difference image generation step, moving object image according to any one of claims 1-5, characterized in that to generate the difference image using the threshold corresponding to the weakest state contrast in the environment Extraction method. In the moving object candidate extraction step, a pixel group of continuous objects is extracted as one moving object candidate from the difference image generated in the difference image generating step, and a characteristic related to the luminance value of the extracted moving object candidate is evaluated. Then, a luminance value that is a shadow determination threshold for determining whether or not the pixel of the object included in the moving object candidate is a shadow is determined from the evaluation result of the characteristic, and the luminance value that is the shadow determination threshold moving object image extraction method according to any one of claims 1-6, characterized in that the removal of shadows from the moving object candidate. Using a plurality of captured images arranged in time series in an environment where at least one of the amount of natural light or illumination light and the irradiation direction may fluctuate, a moving object included in the captured image A moving image extraction method for extracting an image as a moving image,
A background image acquisition step of acquiring a background image in which the influence of the moving object is eliminated;
A difference image generation step for generating a difference image between the background image acquired by the background image acquisition step and the captured image;
A background image update step of updating the background image from the background image and the captured image;
A moving object candidate extraction step of extracting a partial image that may be one moving object as a moving object candidate from the difference image generated by the difference image generating step;
When a plurality of moving object candidates are extracted by the moving object candidate extracting step, moving object candidates having an inclusion relationship are listed as a group, and combinations of moving object candidates that can be included in the group are listed as patterns, Appropriate moving object candidates from a plurality of moving object candidates by comparing appropriate moving object candidates and moving object candidates extracted from captured images adjacent in time series based on similarity Moving object candidate narrowing step to narrow down
Moving object image extraction method characterized by including the. In the moving object candidate extraction step, a pixel group of continuous objects is once extracted as one moving object candidate from the difference image generated in the difference image generating step, and a combination of moving object candidates extracted as another candidate is extracted. a medium, according to claim 8, characterized in that the same moving object candidate possibilities is the add fusion body candidates that combine multiple elements candidates is determined of the moving object candidate with the inclusion relation group Moving image extraction method. Using a plurality of captured images arranged in time series in an environment where at least one of the amount of natural light or illumination light and the irradiation direction may fluctuate, a moving object included in the captured image A moving image extraction device for extracting an image as a moving image,
A background image acquisition unit that acquires a background image from which the influence of the moving object has been eliminated , and generates a variation coefficient indicating a luminance value and a degree of variation of the luminance value for each pixel of the background image;
A luminance difference value for each pixel between the background image acquired by the background image acquisition unit and the captured image is calculated, and the calculated luminance difference value and the variation coefficient are compared to calculate the background image and the imaging A difference image generating means for generating a difference image while determining whether or not to adopt the pixel of the captured image as a difference image that is a difference from the image;
Background image update means for updating the background image from the background image and the captured image;
Moving object candidate extraction means for extracting a partial image that may be one moving object as a moving object candidate from the difference image generated by the difference image generating means;
A moving object image extracting apparatus comprising: moving object candidate narrowing means for narrowing down an appropriate moving object candidate from a plurality of moving object candidates when a plurality of moving object candidates are extracted by the moving object candidate extracting means.   Using a plurality of captured images arranged in time series in an environment where at least one of the amount of natural light or illumination light and the irradiation direction may fluctuate, a moving object included in the captured image A moving image extraction device for extracting an image as a moving image,
  Background image acquisition means for acquiring a background image in which the influence of the moving object is eliminated;
  Difference image generation means for generating a difference image between the background image acquired by the background image acquisition means and the captured image;
  A luminance difference value that is a difference for each pixel between the captured image and the immediately preceding captured image that is the captured image captured immediately before the captured image is calculated, and a distribution of the pixels in which the luminance difference value is equal to or greater than a predetermined value in the entire image The reflection ratio for calculating the reflection ratio for each pixel in the background image of the captured image in accordance with the evaluation result of the distribution of the luminance difference value and the distribution of the entire pixels of the pixels having the luminance difference value equal to or greater than a predetermined value. A calculation means;
  By using the reflection ratio for each pixel calculated by the reflection ratio calculation means, the captured image and the background image are weighted averaged by the reflection ratio and (1-the reflection ratio) for each pixel. Background image update means for generating the background image after update;
  Moving object candidate extraction means for extracting a partial image that may be one moving object as a moving object candidate from the difference image generated by the difference image generating means;
  Moving object candidate narrowing means for narrowing down an appropriate moving object candidate from a plurality of moving object candidates when a plurality of moving object candidates are extracted by the moving object candidate extracting means;
A moving image extraction apparatus comprising:

Images