JP2017107407A - Image processing device, control program, and foreground image specification method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technique capable of improving specification accuracy in specifying a foreground image in an input moving image.SOLUTION: A second map update part 42 executes copy processing for adopting a first frequency value in a slight movement correspondence area corresponding to a slight movement occurrence area in a first map 410 as a second frequency value corresponding to the first frequency value in a second map. The second map update part 42 decreases a second frequency value other than a second frequency value to be subjected to copy processing by a prescribed value in the second map 420. A foreground specification part 34 specifies a foreground image on the basis of the second map 420 subjected to update processing.SELECTED DRAWING: Figure 3

Description

  The present invention relates to an image processing technique.

  As described in Patent Documents 1 to 5, various image processing techniques have been proposed.

Japanese Patent Laying-Open No. 2015-158837 JP2015-125696A JP2013-65151A JP 2011-123742 A JP 2010-3177 A

  As described in Patent Documents 1 and 2, etc., when specifying a foreground image in an input moving image, it is desired that the foreground image can be specified with high accuracy.

  Therefore, the present invention has been made in view of the above-described points, and an object thereof is to provide a technique capable of improving the specifying accuracy when specifying a foreground image in an input moving image.

  One aspect of the image processing apparatus is an image processing apparatus that identifies an image of a detection target moving object in an input moving image as a foreground image, and pixels included in a frame image of the input moving image based on a first background model. A first determination unit that performs a first determination process for tentatively determining whether the pixel is a foreground pixel or a background pixel, and a plurality of first values respectively corresponding to the plurality of pixels included in the frame image A first map update unit for performing a first map update process for updating the first map arranged according to the position of the plurality of second values respectively corresponding to the plurality of pixels included in the frame image, A second map update unit that performs a second map update process that updates a second map arranged according to the position of the pixel, and a first background model update process that updates the first background model based on the frame image A first background model update unit; and a fine movement specifying unit that performs a first specifying process for specifying a fine movement generation area in which a fine movement of the detection target moving object is generated in a shooting region that is reflected in the input moving image, The map update unit corresponds to the first map in the first map that corresponds to the pixel determined to be the foreground pixel in the first determination process included in the frame image in the first map update process. The value is increased by a first predetermined value, and the first value in the first map corresponding to the pixel determined to be the background pixel in the first determination process is included in the frame image. The second map updating unit is identified in the first map when the fine movement occurrence area is identified in the first identifying process in the second map updating process. Microtremor generation area Copy processing is performed in which the first value in the corresponding fine movement corresponding area is adopted as the second value corresponding to the first value in the second map, and the copy is performed in the second map. The second value other than the second value to be processed is decreased by a third predetermined value, and the image processing apparatus performs the first determination process, the first map update process, and the second map update. A foreground specifying unit that repeatedly executes the process, the first background model update process, and the first specifying process, and specifies the foreground image based on the second map on which the second update process has been performed;

  In one aspect of the image processing apparatus, the third predetermined value is greater than the second predetermined value.

  Further, in one aspect of the image processing apparatus, the second map update unit includes a plurality of independent multiples corresponding to an area where the detection target moving object is likely to exist in the imaging area in the first map. The second map update unit corresponds the first value in the detection target corresponding area including the fine movement corresponding area to the first value in the second map. This is adopted as the second value.

  Further, in one aspect of the image processing device, the second map update unit has a first threshold value greater than or equal to the first threshold value in the detection target corresponding area including the fine movement corresponding area. The first value larger than the threshold value is adopted as the second value corresponding to the first value in the second map.

  In one aspect of the image processing apparatus, the second map update unit has an area of the detection target corresponding region including a plurality of fine motion corresponding regions that are independent from each other or greater than or equal to the second threshold value. Is larger, a circumscribed rectangle of the plurality of fine movement corresponding regions is obtained, and among the first values in the detection target corresponding region, the obtained first value in the circumscribed rectangle is determined as the second map. Is adopted as the second value corresponding to the first value.

  In one aspect of the image processing apparatus, the second map update unit has an area of the detection target corresponding area including a plurality of fine movement corresponding areas independent from each other or the second threshold value. Is larger than the detection target corresponding area, a fine movement grouping process is performed in which a plurality of fine movement corresponding areas that are close to each other are set as one fine movement group, and the second map update unit is configured to detect the detection target corresponding area. For each fine movement group obtained by the fine movement grouping process for the corresponding area, a circumscribed rectangle of a plurality of fine movement corresponding areas constituting the fine movement group is obtained, and the first circumscribed rectangle in the detection target corresponding area is obtained for each obtained circumscribed rectangle. Among the values, the first value in the circumscribed rectangle is adopted as the second value corresponding to the first value in the second map.

In one aspect of the image processing apparatus, the fine movement specifying unit generates a difference image between the first and second frame images of the input moving image, and a plurality of independent moving object regions in the difference image are determined as a moving object region. A first grouping process is performed in which a plurality of moving body regions that are close to each other are set as one first group for the moving body region group, and the fine movement specifying unit is configured to perform the first grouping process. When there is an isolated moving object region in the moving object region group that does not have a moving object region nearby, the isolated moving object region is set as one first group and obtained as a result of the first grouping process on the difference image. For each of the plurality of first groups, a first inclusion region that satisfies the first and second conditions is specified from among the first inclusion regions that include the moving body regions that constitute the first group. Performing a second identification processing, the first condition is a condition that the area of the first inclusion region is less than the third threshold value or less or the third threshold value,
The second condition is a condition that the first inclusion area that does not satisfy the first condition does not exist within a predetermined distance from the first inclusion area. A corresponding area corresponding to the first inclusion area specified in the second specifying process is set as the fine movement occurrence area.

  In one aspect of the image processing apparatus, the fine movement identifying unit repeatedly performs a second grouping process in which a group determination process for determining one second group for a processing target is repeated in the first grouping process. The fine movement identifying unit performs processing in a plurality of moving body areas constituting the moving body area group in the first group determination process in the first second grouping process, and the fine movement identifying section includes: In the group determination process, if there is another moving object region whose distance from the reference moving object region is equal to or less than the fourth threshold value or less than the fourth threshold value in the processing target, the reference moving object When the region and the other moving object region are determined as one second group and the other moving object region does not exist in the processing target, the reference moving object region Is determined as one second group, and in the second grouping process, the fine movement specifying unit performs the second and subsequent group determination processes, and the processing target in the most recent group determination process The group determination process is executed with the remaining part excluding the second group determined in the most recent group determination process as the new process target, and the fine movement specifying unit performs the second grouping process. When the group determination process is repeatedly executed until the second group cannot be determined, the next second grouping process is executed, and the fine movement specifying unit executes the second and subsequent second grouping processes. If so, for each of the second groups determined in the most recent second grouping process, the moving object area constituting the second group The second inclusion area including the new moving object area, the new moving object area as the processing target, the first group determination process of the second grouping process is executed, and the fine movement specifying unit When the second grouping process is repeatedly executed until the number of the second groups determined by the grouping process does not decrease below the number of the second groups determined by the previous second grouping process , Ending the first grouping process, the fine movement specifying unit, for each of the second group finally obtained in the first grouping process, the moving body area constituting the second group, Let it be a first inclusion region.

  In one aspect of the image processing apparatus, it is temporarily determined whether a pixel included in the frame image of the input moving image is a foreground pixel or a background pixel based on a second background model different from the first background model. A second determination unit that performs a second determination process; and a second background model update unit that updates the second background model based on the frame image by a method different from the first background model update unit. The foreground specifying unit specifies the foreground image based on the second map and the result of the second determination process.

  Further, in one aspect of the image processing device, the first background model update unit is included in the first background model according to a frequency at which a similar pixel value appears in a certain pixel in the frame image. The background information corresponding to a certain pixel is updated.

  In one aspect of the image processing apparatus, the first background model is a background model based on a mixed normal distribution obtained by modeling pixel values of the past frame image, and the first background model update unit includes the first background model In the first background model update process, based on a pixel value of a certain pixel of the frame image, the mixed normal distribution corresponding to the certain pixel is updated, and from the plurality of normal distributions included in the mixed normal distribution after the update The normal distribution used as the background information corresponding to the certain pixel is determined.

  Further, in one aspect of the image processing device, the first background model update unit includes a frame image in the fine movement corresponding region of the first map when the second map update unit executes the copy process. A learning rate when updating the mixed normal distribution according to the pixel corresponding to the first value is decreased.

  Further, in one aspect of the image processing device, the second background model update unit, based on the pixel value when the pixel value of the pixel included in the frame image shows the same value continuously for a certain period of time The second background model is updated.

  Further, one aspect of the control program is a control program for causing a computer to specify an image of a moving object to be detected in an input moving image as a foreground image, and (a) the input based on a background model Tentatively determining whether a pixel included in a frame image of a moving image is a foreground pixel or a background pixel; and (b) a plurality of first values respectively corresponding to a plurality of pixels included in the frame image, Updating the first map arranged according to the positions of the plurality of pixels, and (c) a plurality of second values respectively corresponding to the plurality of pixels included in the frame image at the positions of the plurality of pixels. A step of updating the second map arranged accordingly, (d) a step of updating the background model based on the frame image, and (e) a shooting region reflected in the input moving image, In the step (b), in the step (b-1), the determination process included in the frame image is performed. A step of increasing the first value in the first map corresponding to the pixel determined to be the foreground pixel by a first predetermined value; and (b-2) included in the frame image, A step of decreasing the first value in the first map corresponding to the pixel determined to be the background pixel by the determination process by a second predetermined value, and in the step (c), (C-1) When the fine movement occurrence area is specified by the specifying process, the first value in the fine movement corresponding area corresponding to the specified fine movement generation area in the first map is set to the second value. Corresponds to the first value in the map (C-2) In the second map, a second predetermined value other than the second value at which the copy process is executed is set to a third predetermined value. And a step of repeatedly executing the steps (a) to (e), and (f) specifying the foreground image based on the second map updated in the step (c). Is for further execution.

  Further, one aspect of the foreground image specifying method is a foreground image specifying method in an image processing apparatus, wherein (a) a pixel included in a frame image of the input moving image is a foreground pixel or a background based on a background model (B) updating a first map in which a plurality of first values respectively corresponding to a plurality of pixels included in the frame image are arranged in accordance with the positions of the plurality of pixels; (C) updating a second map in which a plurality of second values respectively corresponding to a plurality of pixels included in the frame image are arranged in accordance with the positions of the plurality of pixels; A step of updating the background model based on the frame image; and (e) a specifying process for specifying a fine motion generation region in which a fine motion of the detection target moving body is generated in a photographing region reflected in the input moving image. With processes In the step (b), (b-1) the first value in the first map corresponding to the pixel determined to be the foreground pixel in the determination process, included in the frame image, A step of increasing by a first predetermined value; and (b-2) the first map in the first map corresponding to a pixel included in the frame image and determined to be the background pixel in the determination process. A step of decreasing the value by a second predetermined value, wherein the step (c) is specified in the first map when the fine movement occurrence region is specified in (c-1) the specifying process. Executing a copy process that employs the first value in the fine movement corresponding area corresponding to the fine movement generation area as the second value corresponding to the first value in the second map; (C-2) In the second map, the copy process is Reducing the second value other than the second value to be performed by a third predetermined value, the steps (a) to (e) are repeatedly executed, and (f) the step (c) The method further includes the step of specifying the foreground image based on the second map updated in step).

  It is possible to improve the accuracy of specifying the foreground image in the input moving image.

It is a figure which shows the structure of an image processing system. It is a figure which shows an example of a frame image. It is a figure which shows the structure of an image processing apparatus. It is a figure for demonstrating a background model. It is a figure for demonstrating appearance pixel value information. It is a figure for demonstrating operation | movement of the determination part. It is a figure for demonstrating operation | movement of the determination part. It is a flowchart which shows a pixel unit background update process. It is a flowchart which shows an initial learning process. It is a flowchart which shows operation | movement of an image processing apparatus. It is a flowchart which shows a fine movement specific process. It is a figure which shows an example of the moving body area | region group in a difference image. It is a figure for demonstrating operation | movement of a fine movement specific | specification part. It is a figure for demonstrating operation | movement of a fine movement specific | specification part. It is a figure which shows the outline | summary of a 1st grouping process. It is a flowchart which shows a group determination process. It is a figure for demonstrating operation | movement of a fine movement specific | specification part. It is a figure for demonstrating operation | movement of a fine movement specific | specification part. It is a figure for demonstrating operation | movement of a fine movement specific | specification part. It is a figure for demonstrating operation | movement of a fine movement specific | specification part. It is a figure for demonstrating operation | movement of a fine movement specific | specification part. It is a figure for demonstrating operation | movement of a fine movement specific | specification part. It is a figure for demonstrating operation | movement of a fine movement specific | specification part. It is a figure for demonstrating operation | movement of a fine movement specific | specification part. It is a figure for demonstrating operation | movement of a fine movement specific | specification part. It is a figure for demonstrating operation | movement of a fine movement specific | specification part. It is a figure for demonstrating operation | movement of a fine movement specific | specification part. It is a figure for demonstrating operation | movement of a fine movement specific | specification part. It is a figure for demonstrating operation | movement of a fine movement specific | specification part. It is a figure for demonstrating operation | movement of a fine movement specific | specification part. It is a figure for demonstrating operation | movement of a fine movement specific | specification part. It is a figure for demonstrating operation | movement of a fine movement specific | specification part. It is a figure for demonstrating operation | movement of a fine movement specific | specification part. It is a figure for demonstrating operation | movement of a fine movement specific | specification part. It is a figure which shows an example of a 1st frequency map. It is a figure which shows an example of a 2nd frequency map. It is a figure for demonstrating operation | movement of a 2nd map update part. It is a figure for demonstrating operation | movement of a 2nd map update part. It is a figure for demonstrating operation | movement of a 2nd map update part. It is a figure for demonstrating operation | movement of a 2nd map update part. It is a figure for demonstrating operation | movement of a foreground specific part. It is a figure for demonstrating an example of the method of calculating | requiring the shortest distance between the external shapes of two rectangular areas. It is a figure which shows the structure of the modification of an image processing apparatus. It is a flowchart which shows operation | movement of a background model update part.

<Configuration of image processing system>
FIG. 1 is a diagram showing a configuration of the image processing system 1. The image processing system 1 is a system that detects a detection target moving object existing in a predetermined space using image processing. The image processing system 1 is introduced into, for example, an office or a school. The image processing system 1 detects a person existing in a room such as an office. The moving object to be detected may be other than a person.

  As shown in FIG. 1, the image processing system 1 includes a camera 2 and an image processing device 3. The camera 2 is attached to the indoor ceiling, for example. The camera 2 is an omnidirectional camera, for example, and takes a bird's-eye view of the room. Therefore, the shooting range of the camera 2 is indoors. The camera 2 captures and outputs the moving image 100 at a predetermined frame rate (for example, 30 fps). The moving image 100 output from the camera 2 is input to the image processing device 3. That is, a plurality of frame images constituting the moving image 100 are sequentially input to the image processing apparatus 3. Hereinafter, the moving image 100 may be referred to as an “input moving image 100”. Further, the frame image of the input moving image 100 may be referred to as “input frame image 101”.

  In this example, the input moving image 100 is a color image, but may be a grayscale image. Further, the camera 2 may be attached to a place other than the ceiling. The camera 2 may be other than an omnidirectional camera.

  FIG. 2 is a diagram illustrating an example of the input frame image 101. The input frame image 101 is also called a fisheye image. As shown in FIG. 2, a plurality of desks 330 and a plurality of chairs 340 are arranged on the floor 320 of the room 300 that is reflected in the input frame image 101. There are a plurality of people 200 in the room 300. Specifically, in the room 300, there are two persons 200a and 200b standing beside the wall 350 and one person 200c sitting on a chair 340. Accordingly, the input frame image 101 includes three people 200a, 200b, and 200c.

  The image processing apparatus 3 detects the person 200 existing in the shooting range of the camera 2, that is, the room 300 by specifying the image of the person 200 in the input moving image 100 as the foreground image.

  In this example, the image processing apparatus 3 does not detect all the people present in the room 300 but detects a stationary person, in other words, a person who has not moved among the persons present in the room 300. In other words, the image processing device 3 specifies the image of the stationary person 200 (the image of the person 200 not moving) in the input moving image 100 as the foreground image. As a stationary person, for example, as shown in FIG. 2, a person 200c sitting on a chair 340, a person standing in the room 300, a person reading a book in front of a bookshelf in the room 300 Etc.

  The image processing apparatus 3 is, for example, a kind of computer, and includes a CPU 4+ (Central Processing Unit) and a storage unit 5 as shown in FIG. It can be said that the image processing apparatus 3 has a kind of circuit configuration. The storage unit 5 is configured by a non-transitory recording medium that can be read by the CPU 4, such as a ROM (Read Only Memory) and a RAM (Random Access Memory). The storage unit 5 stores a control program 6 for controlling the image processing apparatus 3 (computer). Various function blocks are formed in the image processing apparatus 3 by the CPU 4 executing the control program 6 in the storage unit 5.

  Note that some or all of the functions of the image processing apparatus 3 may be realized by a hardware circuit including a logic circuit or the like that does not require a program (software) to realize the function. The storage unit 5 may include a computer-readable non-transitory recording medium other than the ROM and RAM. The storage unit 5 may include, for example, a small hard disk drive and an SSD (Solid State Drive).

<Configuration of image processing apparatus>
FIG. 3 is a diagram showing the configuration of the image processing apparatus 3. As shown in FIG. 3, the image processing apparatus 3 includes a preprocessing unit 30, a determination unit 31, a background model update unit 32, a fine movement specification unit 33, a foreground specification unit 34, a first map update unit 41, and a second map update. A portion 42 is provided.

  The preprocessing unit 30 performs preprocessing on the input moving image 100 from the camera 2 and outputs the input moving image 100 after the preprocessing as the input moving image 110. Specifically, the preprocessing unit 30 performs grayscale conversion and size reduction processing on the input frame images 101 sequentially input from the camera 2, and sequentially outputs the processed input frame images 101 as input frame images 111. To do. Therefore, the input moving image 110 output from the preprocessing unit 30 is a grayscale image. Depending on the format of the input moving image 100 output from the camera 2, the preprocessing unit 30 is not necessary. Hereinafter, the pixel “input pixel” of the input frame image 111 of the input moving image 110 may be called.

  The determination unit 31 performs provisional determination processing that tentatively determines whether the input pixel of the input frame image 111 is the foreground pixel or the background pixel based on the background model 500 in the storage unit 5. The background model 500 is information that models the background that appears in the input moving image 110. The determination unit 31 determines an input pixel indicating a person as a foreground pixel without distinguishing whether the input pixel is a pixel indicating a moving person or a pixel indicating a stationary person.

  The background model update unit 32 performs background model update processing for updating the background model 500 based on the input frame image 111 and appearance pixel value information 510 described later in the storage unit 5.

  Based on the input moving image 110, the fine movement specifying unit 33 performs a fine movement specifying process for specifying a fine movement generation region in which a fine movement of a person who is a detection target moving body is generated in a shooting area captured in the input moving image 110.

  The first map update unit 41 performs a first map update process for updating the first map 410 in the storage unit 5 based on the determination result in the determination unit 31. The first map 410 is information indicating the distribution of the foreground pixel specific frequency (detection frequency) in the input frame image 111. Hereinafter, the first map 410 may be referred to as a “first frequency map 410”.

  The second map update unit 42 performs a second map update process for updating the second map 420 in the storage unit 5 based on the fine movement occurrence area specified by the fine movement specifying unit 33 and the first frequency map 410. The second map 420 is information indicating the distribution of the specific frequency of the foreground pixels in the area where the fine movement occurrence area in the shooting range is captured in the input frame image 111. Hereinafter, the second map 420 may be referred to as a “second frequency map 420”.

  Based on the second frequency map 420, the foreground identification unit 34 performs a foreground identification process that identifies an image of a still person in the input frame image 111 as a foreground image.

  Each time the input frame image 101 is input from the camera 2, the image processing apparatus 3 sets the input frame image 101 as a processing target, and performs preprocessing, provisional determination processing, background model update processing, fine motion identification processing, Map update processing, second map update processing, and foreground identification processing are performed. Hereinafter, the input frame image 111 obtained when the preprocessing unit 30 performs the preprocessing on the input frame image 101 to be processed may be referred to as “the input frame image 111 to be processed”.

<Detailed operation of image processing apparatus>
Next, detailed operation of each component of the image processing apparatus 3 will be described. In order to facilitate understanding of the operation of the image processing apparatus 3, first, the background model, the provisional determination process, and the background model update process will be described.

<Background model>
In this example, the background model 500 is a background model based on a mixed normal distribution (MoG) in which pixel values of past frame images are modeled. An example of such a background model is disclosed in Patent Documents 4 and 5 described above, for example. The pixel value in this example is, for example, luminance.

  FIG. 4 is a diagram showing the configuration of the background model 500. As shown in FIG. 4, in the background model 500, at least one background information 501 is associated with each of a plurality of pixels constituting the input frame image 111. The background information 501 is represented by a normal distribution. Hereinafter, the background information 501 may be referred to as “background normal distribution 501”.

  On the other hand, appearance pixel value information 510 is stored in the storage unit 5. Appearance pixel value information 510 is information regarding pixel values that have appeared in past frame images. As shown in FIG. 5, the appearance pixel value information 510 includes a plurality of mixed normal distributions 511 respectively corresponding to a plurality of pixels constituting the input frame image 111.

  The mixed normal distribution 511 models past pixel values for the corresponding pixels. The mixed normal distribution 511 indicates the appearance probability for each pixel value for the corresponding pixel. In other words, the mixed normal distribution 511 indicates the appearance frequency for each pixel value for the corresponding pixel.

  The mixed normal distribution 511 is represented by a plurality of normal distributions 512 and a plurality of weights w set to the plurality of normal distributions 512, respectively. A part of the plurality of normal distributions 512 included in the mixed normal distribution 511 is adopted as the background normal distribution 501 corresponding to the pixels corresponding to the mixed normal distribution 511. Specifically, among the plurality of normal distributions 512 included in the mixed normal distribution 511, the normal distribution 512 having a weight w greater than the first threshold is the background normal corresponding to the pixel corresponding to the mixed normal distribution 511. The distribution 501 is adopted.

<Details of provisional judgment processing>
In the provisional determination process, the determination unit 31 tentatively determines whether each input pixel of the input frame image 111 to be processed is a foreground pixel or a background pixel. When the determination unit 31 tentatively determines whether a certain input pixel is a foreground pixel or a background pixel, the background normal distribution 501 corresponding to the input pixel in the background model 500 and appearance pixel value information The mixed normal distribution 511 corresponding to the input pixel at 510 is used. Hereinafter, the input pixel to be described may be referred to as “target input pixel”.

  In the plurality of normal distributions 512 included in the mixed normal distribution 511 corresponding to the target input pixel, the determination unit 31 may be referred to as a normal distribution 512 that matches the pixel value of the target input pixel (hereinafter, referred to as “compatible normal distribution 512”). It is determined whether or not there exists. The determination unit 31 determines that the target input pixel is a foreground pixel when the matching normal distribution 512 does not exist in the plurality of normal distributions 512 included in the mixed normal distribution 511 corresponding to the target input pixel.

  On the other hand, the determination unit 31 corresponds to the target input pixel in the background model 500 when the matching normal distribution 512 exists in the plurality of normal distributions 512 included in the mixed normal distribution 511 corresponding to the target input pixel. In the background normal distribution 501, it is determined whether or not there is a background normal distribution 501 that matches the relevant normal distribution 512. The determination unit 31 determines that the target input pixel is the foreground pixel when the background normal distribution 501 corresponding to the target input pixel in the background model 500 does not have the background normal distribution 501 that matches the matching normal distribution 512. To do. On the other hand, when the background normal distribution 501 corresponding to the target input pixel in the background model 500 includes the background normal distribution 501 that matches the matching normal distribution 512, the determination unit 31 determines that the target input pixel is the background. It is determined that it is a pixel.

  When the pixel value of the input pixel is within a predetermined range of the normal distribution 512, the determination unit 31 sets the normal distribution 512 as the matched normal distribution 512. Here, when the average value and the variance value of the normal distribution are expressed by μ and σ, respectively, the predetermined range is, for example, a range of μ ± 3σ of the normal distribution 512. FIG. 6 is a diagram illustrating an example of the conforming normal distribution 512 that matches the pixel value 115 of the input pixel. FIG. 7 is a diagram illustrating an example of a normal distribution 512 that does not match the pixel value 115 of the input pixel.

  Note that that the pixel value of the input pixel matches the normal distribution 512 means that the background model 500 also matches the same background normal distribution 501 as the normal distribution 512 corresponding to the input pixel. Therefore, when the background normal distribution 501 corresponding to the pixel value of the target input pixel does not exist in the background normal distribution 501 corresponding to the target input pixel in the background model 500, the determination unit 31 determines that the target input pixel is the foreground pixel. It can be said that it is determined. In addition, when the background normal distribution 501 that matches the pixel value of the target input pixel exists in the background normal distribution 501 corresponding to the target input pixel in the background model 500, the determination unit 31 determines that the target input pixel is the background pixel. It can be said that it is determined.

<Details of background model update processing>
In the background model update process, the background model update unit 32 determines the input pixel in the appearance pixel value information 510 based on the result of the provisional determination process in the determination unit 31 for an input pixel in the input frame image 111 to be processed. Is updated, and the background information 501 (background normal distribution 501) corresponding to the certain input pixel in the background model 500 is updated based on the updated mixed normal distribution 511. This process is called “pixel unit background update process”. The background model update unit 32 performs a pixel unit background update process for each pixel of the input frame image 111 in the background model update process.

  FIG. 8 is a flowchart showing the pixel unit background update processing. The pixel unit background update process will be described with reference to FIG. Hereinafter, the mixed normal distribution 511 corresponding to the target input pixel of the input frame image 111 to be processed is referred to as “target mixed normal distribution 511”. The background information 501 corresponding to the target input pixel is referred to as “target background information 501”.

  As shown in FIG. 8, in step s1, the background model update unit 32 updates the target mixed normal distribution 511 based on the result of the provisional determination process in the determination unit 31 for the target input pixel. In step s2, the background model update unit 32 updates the target background information 501 based on the updated target mixed normal distribution 511. Step s1 includes steps s11 to s14.

  In step s11, the background model update unit 32 updates the weight w of each normal distribution 512 included in the target mixed normal distribution 511. The background model update unit 32 updates the weight w according to the following equation (1).

  In equation (1), w0 and w1 indicate the current weight w and the updated weight w, respectively. M in the formula (1) is a value set to 1 or 0. Among the plurality of normal distributions 512 included in the target mixed normal distribution 511, M = 1 when the determination unit 31 updates the weight w of the matching normal distribution 512 that is determined to match the pixel value of the target input pixel. Is set. On the other hand, among the plurality of normal distributions 512 included in the target mixed normal distribution 511, when the determination unit 31 updates the weight w of the normal distribution 512 determined not to match the pixel value of the target input pixel, M = 0 is set.

  Α in the equation (1) is a learning rate, and 0 <α <1. The learning rate α indicates how much the result of the provisional determination process is affected to the current value when the weight w is updated based on the current value. As the learning rate α is larger, the updated weight w is more affected by the result of the provisional determination process. Therefore, the weight w is less likely to change as the learning rate α is smaller.

  When the plurality of weights w of the target mixed normal distribution 511 are updated, the background model update unit 32 normalizes each weight w so that the sum of the updated weights w becomes “1”.

  As can be understood from Equation (1), among the plurality of normal distributions 512 included in the target mixed normal distribution 511, the weight w of the normal distribution 512 that matches the pixel value of the target input pixel increases, and the pixel of the target input pixel The weight w of the normal distribution 512 that does not match the value becomes smaller.

  After step s11, when the background model update unit 32 determines in step s12 that the adaptive normal distribution 512 is included in the target mixed normal distribution 511, in step s13, the average value μ and the variance value of the adaptive normal distribution 512 are determined. Update σ.

  The background model updating unit 32 updates the average value μ of the conforming normal distribution 512 according to the following equation (2). Further, the background model update unit 32 updates the variance value σ of the conforming normal distribution 512 according to the following equation (3).

  In the equation (2), μ0 and μ1 indicate an average value μ before update and an average value μ after update, respectively. In the equation (3), σ0 and σ1 indicate the variance value σ before update and the variance value σ after update, respectively. X in the expressions (2) and (3) indicates the pixel value of the target input pixel.

  In equations (2) and (3), β represents a learning rate, and 0 <β <1. The learning rate β in Expression (2) indicates how much the pixel value X of the target input pixel affects the current value when the average value μ is updated based on the current value. Yes. As the learning rate β increases, the updated average value μ is more influenced by the pixel value X of the target input pixel. Therefore, the smaller the learning rate β, the less the average value μ changes. Further, the learning rate β in Expression (3) indicates how much the pixel value X of the target input pixel affects the current value when the variance value σ is updated based on the current value. Show. As the learning rate β is larger, the updated variance value σ is more influenced by the pixel value X of the target input pixel. Therefore, the variance value σ is less likely to change as the learning rate β is smaller.

  The learning rates α and β are expressed by the following equations (4) and (5), for example.

  As shown in Expression (4), the learning rate α is a value obtained by adding a constant A to the reference value REF. Further, as shown in Expression (5), the learning rate β is a value obtained by adding a constant B to the reference value REF. The reference value REF is a fixed value and is commonly used for the learning rates α and β. The constant A is a value specific to the learning rate α. The constant B is a value specific to the learning rate β. The constant A and the constant B are different from each other.

  On the other hand, if the background model update unit 32 determines in step s12 that the target mixed normal distribution 511 does not include the matched normal distribution 512, the background model update unit 32 executes step s13.

  In step s13, the background model update unit 32 newly creates a normal distribution in which the pixel value of the target input pixel is the average value μ. Then, the background model update unit 32 deletes one normal distribution 512 of the plurality of normal distributions 512 included in the target mixed normal distribution 511, and replaces the deleted normal distribution 512 with a newly created normal distribution. use. The weight w and variance value σ of the newly created normal distribution match the weight w and variance value σ of the deleted normal distribution 512, respectively. The background model update unit 32 deletes, for example, the normal distribution 512 having the smallest weight w from the plurality of normal distributions 512 included in the target mixed normal distribution 511.

  When the background model update unit 32 executes Step s13, the update of the target mixed normal distribution 511 is completed. That is, step s1 ends. Further, when the background model update unit 32 executes Step s14, the update of the target mixed normal distribution 511 is completed. That is, step s1 ends.

  After step s1, the background model updating unit 32 updates the target background information 501 based on the updated target mixed normal distribution 511 in step s2. Specifically, the background model update unit 511 determines a normal distribution 512 to be used as the background information 501 from the plurality of normal distributions 512 included in the updated target mixed normal distribution 511. The background model update unit 511 uses, for example, a normal distribution 512 having a weight w greater than the first threshold as background information 501 among the plurality of normal distributions 511 included in the updated target mixed normal distribution 511. Decide what to do. In the plurality of normal distributions 511 included in the updated target mixed normal distribution 511, when there are a plurality of normal distributions 512 having a weight w greater than the first threshold value, the plurality of normal distributions 512 are used as background information. It is determined to be used as 501. Then, the background model update unit 32 replaces the current target background information 501 with the determined new background information 501. Thereby, the target background information 501 is updated.

  As can be understood from the above description, when the frequency of appearance of the same pixel value in the target input pixel increases, the normal distribution 512 corresponding to the pixel value appearing in the target input pixel has the background information 501 corresponding to the target input pixel. Will be adopted as. Therefore, if the frequency at which the same pixel value appears in the target input pixel increases, the target input pixel is determined as the background pixel in the determination unit 31. On the other hand, if the frequency of appearance of similar pixel values in the target input pixel decreases, the background normal distribution 501 corresponding to the pixel value is deleted. Therefore, if the frequency of appearance of similar pixel values in the target input pixel decreases, the target input pixel is determined as a foreground pixel in the determination unit 31.

  As described above, the background model update unit 32 updates the background information 501 corresponding to the target input pixel included in the background model 500 according to the frequency at which the same pixel value appears in the target input pixel. Hereinafter, such background information update is referred to as “frequency-type background update”.

  By using the background model 500 and its update method as described above, the background model 500 can be updated appropriately even when the background changes temporarily or when the background changes suddenly. It becomes.

<Operation Stage of Image Processing Device>
The operation stage of the image processing apparatus 3 includes a preparation stage and an actual operation stage. The image processing apparatus 3 performs an initial learning process for the background model 500 in the preparation stage. In the initial learning process of the background model 500, the background model 500 is repeatedly updated based on the input moving image 100 in which the room 300 in which no person exists is captured. In the image processing apparatus 3, when the initial learning process is completed, the operation stage shifts from the preparation stage to the actual operation stage. The image processing device 3 specifies the foreground image in the input moving image 100 in the actual operation stage. Hereinafter, the frame image 101 of the input moving image 100 in which the room 300 in which no person exists is used in the initial learning process is particularly referred to as a “learning frame image 101”. In the initial learning, learning frame images 101 are sequentially input from the camera 2 to the image processing device 3.

<Operation of Image Processing Device in Preparation Stage>
FIG. 9 is a flowchart showing the initial learning process performed in the preparation stage. As illustrated in FIG. 9, in step s21, the image processing apparatus 3 performs preprocessing, provisional determination processing, and background model update processing in step s22 on the learning frame image 101 input from the camera 2 as a processing target. . In the background model update process in step s22, the above-described pixel unit background update process is performed on each input pixel constituting the input frame image 111 corresponding to the learning frame image 101 output from the preprocessing unit 30. As a result, the background model 500 is updated once.

  After step s22, the image processing apparatus 3 determines whether the background model 500 has been updated a predetermined number of times. When determining that the background model 500 has been updated a predetermined number of times, the image processing device 3 ends the initial learning process. On the other hand, when the image processing apparatus 3 determines that the background model 500 has not been updated a predetermined number of times, the image processing apparatus 3 executes step s21 again and sets the learning frame image 101 newly input from the camera 2 as a new processing target. In step s22, preprocessing, provisional determination processing, and background model update processing are performed. Thereafter, the image processing apparatus 3 operates in the same manner.

  As described above, the background model 500 is repeatedly updated based on the input moving image 100 in which the room 300 in which no person exists is present, whereby the background model 500 corresponding to the background in the room 300 is stored in the storage unit 5. The

<Operation of Image Processing Device in Actual Operation Stage>
FIG. 10 is a flowchart showing the operation of the image processing apparatus 3 in the actual operation stage. As illustrated in FIG. 10, when the input frame image 101 is input from the camera 2 in step s31, the image processing apparatus 3 sets the input frame image 101 as a processing target. Next, in step s32, the preprocessing unit 30 performs preprocessing on the input frame image 101 to be processed, and outputs the input frame image 101 to be processed after the preprocessing as the input frame image 111 to be processed. To do.

  Next, in step s33, the determination unit 31 performs provisional determination processing based on the input frame image 111 to be processed.

  In step s34, the background model update unit 32 performs background model update processing based on the input frame image 111 to be processed.

  Next, in step s35, the fine movement specifying unit 33 performs a fine movement specifying process based on the input frame image 111 to be processed.

  Next, in step s36, the first map update unit 41 performs a first map update process for updating the first frequency map 410 based on the result of the provisional determination process in the determination unit 31.

  Next, in step s37, the second map update unit 42 updates the second frequency map 420 based on the fine movement occurrence area specified in step s35 and the first frequency map 410 updated in step s36. 2 Map update processing is performed.

  Next, in step s38, the foreground specifying unit 34 performs a foreground specifying process for specifying a still person's image as a foreground image based on the second frequency map 420 updated in step s37.

  When a new input frame image 101 is input from the camera 2 (step s31), the image processing device 3 executes steps s32 to s38 again with the input frame image 101 as a new processing target. The image processing apparatus 3 operates similarly thereafter, and executes steps s32 to s38 each time the input frame image 101 is input from the camera 2 (step s31).

<Details of microtremor identification processing>
Next, the fine movement specifying process in step s35 will be described in detail.

  As described above, in the frequency-type background update employed in the background model update process according to the present embodiment, if the frequency at which the same pixel value appears in the target input pixel increases, the target input pixel is determined by the determination unit. At 31, the pixel is determined as a background pixel. When a person stays still for a while at a certain place in the shooting region, the frequency with which the same pixel value appears in each pixel corresponding to the place in the input frame image 111 increases. Accordingly, when a person is stationary for a while in the shooting region, the determination unit 31 is more likely to erroneously determine that the pixel of the person's image in the input frame image 111 is the background pixel.

  On the other hand, a stationary person (a person who has not moved) such as a person sitting in a chair is almost always not moving completely but moving slightly (moving small). . For example, a person who is sitting on a chair and working or standing is moving a hand, head, body or leg slightly.

  In the present embodiment, the image processing apparatus 3 identifies a fine movement occurrence region where human tremors are occurring in the imaging region, and uses the identification result and the determination result in the determination unit 31 to determine the processing target. In the input frame image 111, an image of a stationary person is specified as a foreground image. This makes it possible to specify the foreground image more correctly.

  FIG. 11 is a flowchart showing the fine movement specifying process. As shown in FIG. 11, in step s351, the fine movement specifying unit 33 generates a difference image 600 between the input frame image 111 to be processed and the input frame image 111 N frames before. N is set to “3”, for example. For each pixel, the fine movement specifying unit 33 obtains a difference between the pixel value of the input frame image 111 to be processed and the pixel value of the input frame image 111 N frames before, and sets the difference value of each pixel obtained thereby to 2 A difference image 600 is generated by digitization. Therefore, the difference image 600 is a binarized image.

  Next, in step s352, the fine motion specifying unit 33 specifies a plurality of independent moving body regions in the difference image 600 as a moving body region group. In step s352, the fine movement specifying unit 33 first performs labeling using 4-connection or 8-connection on the portion (high luminance region) where the pixel value is “1” in the difference image 600. And the fine movement specific | specification part 33 calculates | requires the circumscribed rectangle about each independent area | region (each connection area | region) obtained by labeling. Then, the fine movement specifying unit 33 sets each circumscribed rectangle as one moving body area (moving pixel area) 601. Thereby, a plurality of independent moving body regions 601 in the difference image 600 are specified as the moving body region group 610.

  FIG. 12 is a diagram illustrating an example of the moving object region group 610 in the difference image 600. A moving body region group 610 shown in FIG. 12 includes moving body regions 601a to 601h.

  Next, in step s353, the fine movement specifying unit 33 performs a first grouping process on the moving object area group 610, in which a plurality of moving object areas 601 that are close to each other are set as one first group 602. In the first grouping process, when the moving object region group 610 includes an isolated moving object region 601 in which no other moving object region 601 is present nearby, one isolated group moving object region 601 alone constitutes one first group 602. .

  FIG. 13 is a diagram illustrating a result of the first grouping process performed on the moving object region group 610 illustrated in FIG. In the example of FIG. 13, a plurality of moving body regions 601a to 601e that are close to each other constitute one first group 602A. In addition, a plurality of moving body regions 601g and 601f that are close to each other form one first group 602B. In the example of FIG. 13, the moving object region 601h is an isolated moving object region 601, and therefore only one moving object region 601h constitutes one first group 602C.

  Next, in step s354, the fine movement specifying unit 33 selects the first of the first inclusion areas 603 including the moving body area 601 constituting the first group 602 for each first group 602 obtained in step s353. The first inclusion region 603 that satisfies the second condition is specified.

  In this example, a circumscribed rectangle of the plurality of moving body regions 601 constituting the first group 602 is defined as a first inclusion region 603 that includes the plurality of moving body regions 601. In addition, when the first group 602 is composed of one moving object region 601, the one moving object region 601 becomes the first inclusion region 603 as it is. In the example of FIG. 14, the circumscribed rectangles of the moving body regions 601a to 601e constituting the first group 602A are the first inclusion regions 603A, and the circumscribed rectangles of the moving body regions 601f and 601g constituting the first group 602B are the first. It is an inclusion area 603B. Further, the moving body region 601h constituting the first group 602C is a first inclusion region 603C.

  The first condition in step s354 is a condition that the area of the first inclusion region 603 is small. Specifically, the first condition is a condition that the area of the first inclusion region 603 is equal to or smaller than the second threshold value. The first condition may be a condition that the area of the first inclusion region 603 is less than the second threshold value. Hereinafter, the first condition is referred to as “size condition”.

  The second condition is a condition that another first inclusion region 603 having a large area does not exist in the vicinity (periphery) of the first inclusion region 603. Specifically, the second condition is a condition that there is no first inclusion region 603 that does not satisfy the size condition within a predetermined distance from the first inclusion region 603. Hereinafter, the second condition is referred to as “isolation condition”.

  In the example of FIG. 14, the area of the first inclusion region 603A is larger than the second threshold value, and the first inclusion region 603A does not satisfy the size condition. On the other hand, the areas of the first inclusion regions 603B and 603C are equal to or smaller than the second threshold value, and the first inclusion regions 603B and 603C satisfy the size condition. Further, since the first inclusion region 603B exists in the vicinity of the first inclusion region 603A and the first inclusion region 603A exists in the vicinity of the first inclusion region 603B, each of the first inclusion regions 603A and 603C is Does not satisfy the isolation condition. On the other hand, since there is no other first inclusion region 603 in the vicinity of the first inclusion region 603B, the first inclusion region 603B satisfies the isolation condition. Therefore, among the first inclusion areas 603A to 603C, the first inclusion area 603 that satisfies the size condition and the isolation condition becomes the first inclusion area 603B. Hereinafter, the first inclusion region 603 that satisfies the size condition and the isolation condition may be referred to as a “matching inclusion region 603”.

  After step s354, in step s355, the fine movement specifying unit 33 sets a corresponding area corresponding to the matching inclusion area 603 specified in step s354 in the imaging area as a fine movement generation area. In the example of FIG. 14, the corresponding area corresponding to the first inclusion area 603c in the imaging area is the fine movement generation area. That is, in the imaging region, a part of the region that exists at the same relative position as the relative position of the first inclusion region 603c in the difference image 600 is the fine motion generation region. If a plurality of matching inclusion areas 603 are specified in step s354, a plurality of corresponding areas respectively corresponding to the plurality of matching inclusion areas in the imaging area are fine movement generation areas. Therefore, in step s355, a plurality of fine movement occurrence regions may be specified.

  Here, from the difference image 600, a plurality of moving body regions 601 that are close to each other according to the movement of the same person may be obtained. On the other hand, there is a high possibility that the moving object region 601 corresponding to the movement of a certain person and the moving object region 601 corresponding to the movement of another person exist away from each other. Therefore, there is a high possibility that the plurality of moving object regions 601 that are close to each other correspond to the same person's movement. Therefore, one first inclusion region 603 that includes a plurality of moving object regions 601 that are close to each other can be viewed as a moving object region according to the movement of one person. Hereinafter, the first inclusion region 603 may be referred to as a “moving object region 603”.

  In addition, when a person is moving in a passage or the like, the person moves greatly, and thus the area of the moving object area 603 (first inclusion area 603) corresponding to the movement of the person increases. On the other hand, when a person is moving slightly, the area of the moving body region 603 (first inclusion region 603) corresponding to the movement of the person is reduced. Therefore, when the moving object region 603 (first inclusion region 603) satisfies the size condition, the moving object region 603 is highly likely to correspond to the fine movement of one person.

  In addition, a plurality of moving body regions 603 corresponding to the same person's movement may be obtained from the difference image 600. For example, a moving object region 603 corresponding to a certain person's head and a moving object region 603 corresponding to the movement of the certain person's leg may be obtained. A plurality of moving body regions 603 corresponding to the movement of the same person are close to each other. When a certain person is moving, a plurality of moving body regions 603 corresponding to the movement of the certain person may have a moving body region 603 having a large area in the vicinity of the moving body region 603 having a small area. For example, let us consider a case where a person wearing a pattern with a small batch on his chest is moving in a passage or the like. In this case, since the movement of clothes without a pattern hardly appears in the difference image 600, the moving area 603 having a small area corresponding to the movement of the batch attached to the clothes and the area corresponding to the movement of the head or the like is large. The moving body region 603 may be obtained separately. At this time, the moving body region 603 having a large area corresponding to the movement of the head or the like exists in the vicinity of the moving body region 603 having a small area corresponding to the movement of the batch. On the other hand, in the plurality of moving body regions 603 corresponding to the fine movement of the same person, the area of each moving body region 603 is small, and there is a low possibility that the moving body region 603 having a large area exists in the vicinity of the moving body region 603 having a small area. Therefore, when the moving body region 603 satisfies the isolation condition, the moving body region 603 is unlikely to be the moving body region 603 corresponding to the movement of the person, and is likely to be the moving body area 603 corresponding to the fine movement of the person.

  As described above, in this example, when the moving object region 603 satisfies the size condition and the isolation condition, it is determined that the moving object region 603 is the moving object region 603 corresponding to the fine movement of one person, and among the imaging regions. The corresponding area corresponding to the moving object area 603 that satisfies the size condition and the isolation condition is a fine movement occurrence area corresponding to one person. By setting the corresponding area corresponding to the moving object area 603 that satisfies the size condition and the isolation condition among the imaging areas as the fine movement generation area, the fine movement generation area can be appropriately specified. Note that the plurality of fine movement generation areas specified by the fine movement specifying unit 33 may correspond to a plurality of persons or may correspond to the same person.

<First grouping process>
In step s353 described above, the fine movement specifying unit 33 obtains the distance between the two moving object areas 601 for all the combinations of the two moving object areas 601, and uses the distances related to the obtained all combinations to perform the first grouping. Processing can be performed. That is, the fine movement identifying unit 33 identifies a plurality of moving body regions 601 whose distances are equal to or less than a predetermined value using the distances associated with all the obtained combinations, and the identified plurality of moving body regions 601 are close to each other. A plurality of identified moving body regions 601 can be determined as one first group 602 as the plurality of moving body regions 601.

  However, the processing for obtaining the distance between the two moving object regions 601 for all combinations of the two moving object regions 601 is heavy.

  Therefore, a method for performing the first grouping process that can reduce the number of times of obtaining the distance between the two moving object regions 601 will be described below.

  FIG. 15 is a diagram showing an outline of an example of the first grouping process. As shown in FIG. 15, in the first grouping process, the fine movement specifying unit 33 repeatedly executes the second grouping process in which the group determination process for determining one second group for the processing target is repeated.

  FIG. 16 is a flowchart showing the group determination process. In the second grouping process, the fine movement specifying unit 33 repeatedly performs the group determination process for the processing target while changing the processing target.

  As shown in FIG. 16, in step s 451, the fine movement specifying unit 33 determines a reference moving object area 601 (hereinafter referred to as “reference moving object area 601”) from the moving object area 601 to be processed. Any moving body region 601 among the moving body regions 601 to be processed may be set as the reference moving body region 601.

  Here, when performing the first group determination process in the first second grouping process, the fine movement specifying unit 33 sets all of the plurality of moving object areas 601 constituting the moving object area group 610 as a processing target.

  Next, in step s452, the fine movement identifying unit 33 determines whether there is another moving body area 601 whose distance from the reference moving body area 601 is equal to or smaller than the third threshold value in the moving body area 601 to be processed. That is, the fine movement specifying unit 33 determines whether or not there is another moving body area 601 at a position close to the reference moving body area 601 in the processing target.

  If the fine movement specifying unit 33 determines that there is another moving body area 601 whose distance from the reference moving body area 601 is equal to or smaller than the third threshold value in the processing target, the reference moving body area 601 and the other moving body are in step s453. The region 601 is determined as one second group 620. On the other hand, if the fine movement specifying unit 33 determines that there is no other moving body area 601 whose distance from the reference moving body area 601 is equal to or smaller than the third threshold value in the processing target, only the reference moving body area 601 is present in step s454. Are determined as one second group 620. When only one moving object region 601 is configured as the processing target, there is no other moving object region 601 whose distance from the reference moving object region 601 is equal to or less than the third threshold value in the processing target.

  In step s452, the fine movement specifying unit 33 may determine whether or not there is another moving body area 601 whose distance from the reference moving body area 601 is less than the third threshold value in the processing target. In this case, if there is another moving body region 601 whose distance from the reference moving body region 601 is less than the third threshold value in the processing target, the fine moving body specifying unit 33 and the other moving body. The region 601 is determined as one second group 620. In addition, when there is no other moving object region 601 whose distance from the reference moving object region 601 is less than the third threshold in the processing target, the fine movement specifying unit 33 sets only the reference moving object region 601 as one second object. Determine as group 620.

  In the second grouping process, when the second grouping process is performed, the fine movement specifying unit 33 determines the second group determined in the latest group determination process from the process target in the latest group determination process. The remaining part that has been removed is subjected to group determination processing as a new processing target.

  When the group determination process is repeatedly executed until the second group cannot be determined in the second grouping process, the fine movement specifying unit 33 executes the next second grouping process. Specifically, the fine movement specifying unit 33 removes the second group determined in the most recent group determination process from the processing target in the second grouping process, and as a result, the moving object region 601 does not remain, and the next When the processing target cannot be determined, the next second grouping process is executed.

  A specific example of the first second grouping process will be described below using the moving object region group 610 shown in FIG. In the first group determination process in the first second grouping process, the fine movement specifying unit 33 sets a plurality of moving body areas 601a to 601h constituting the moving body area group 610 as processing targets. The fine movement specifying unit 33 determines the reference moving body area 601 from the moving body areas 601a to 601h to be processed (step s451). For example, the fine movement specifying unit 33 sets the moving body area 601 a as the reference moving body area 601.

  Next, the fine movement specifying unit 33 determines whether there is another moving body area 601 whose distance from the moving body area 601a that is the reference moving body area 601 is equal to or smaller than the third threshold value in the moving body areas 601a to 601h. (Step s452). At this time, as shown in FIG. 17, the fine movement specifying unit 33 obtains the distance between the moving object area 601a and each of the moving object areas 601b to 601h. And the fine movement specific | specification part 33 determines whether each calculated | required distance is below a 3rd threshold value. Thereby, in the moving body areas 601a to 601h, it is determined whether or not there is another moving body area 601 whose distance from the moving body area 601a is equal to or smaller than the third threshold value. In the example of FIG. 17, the moving body regions 601 whose distance from the moving body region 601a is equal to or smaller than the third threshold value are the moving body regions 601b and 601c. As shown in FIG. 18, the fine movement specifying unit 33 determines the moving body areas 601a to 601c as one second group 620A1 (step s453). Thereby, the first group determination process is completed.

  Next, the fine movement identifying unit 33 removes the remaining parts excluding the second group 620A1 determined in the most recent group determination process from the moving object areas 601a to 601h that are the current processing target, that is, the moving object areas 601d to 601h. A second group determination process is performed as a new process target.

  The fine movement specifying unit 33 determines the reference moving body area 601 from the moving body areas 601d to 601h to be processed in the second group determination process (step s451). For example, the fine movement specifying unit 33 sets the moving body area 601d as the reference moving body area 601.

  Next, the fine movement specifying unit 33 determines whether there is another moving body area 601 whose distance from the moving body area 601d is equal to or smaller than the third threshold value in the moving body areas 601d to 601h (step s452). At this time, as shown in FIG. 19, the fine movement specifying unit 33 obtains the distance between the moving object area 601d and each of the moving object areas 601e to 601h. And the fine movement specific | specification part 33 determines whether each calculated | required distance is below a 3rd threshold value. Thereby, in the moving body regions 601e to 601h, it is determined whether or not there is another moving body region 601 whose distance from the moving body region 601e is equal to or smaller than the third threshold value. In the example of FIG. 19, the moving object region 601 whose distance from the moving object region 601d is equal to or smaller than the third threshold value is the moving object region 601e. As shown in FIG. 20, the fine movement specifying unit 33 determines the moving body areas 601d and 601e as one second group 620B1 (step s453). Thereby, the second group determination process is completed.

  Next, the fine movement identifying unit 33 removes the remaining part excluding the second group 620B1 determined in the most recent group determination process from the moving object areas 601d to 601h that are currently processed, that is, the moving object areas 601f to 601h. A third group determination process is performed as a new process target.

  The fine movement specifying unit 33 determines the reference moving body area 601 from the moving body areas 601f to 601h to be processed in the third group determination process (step s451). For example, the fine movement specifying unit 33 sets the moving body area 601h as the reference moving body area 601.

  Next, the fine movement specifying unit 33 determines whether there is another moving body area 601 whose distance from the moving body area 601f is equal to or smaller than the third threshold value in the moving body areas 601f to 601h (step s452). At this time, as shown in FIG. 21, the fine movement specifying unit 33 obtains the distance between the moving body area 601h and each of the moving body areas 601f and 601g. And the fine movement specific | specification part 33 determines whether each calculated | required distance is below a 3rd threshold value. Thereby, in the moving body regions 601f and 601g, it is determined whether or not there is a moving body region 601 whose distance from the moving body region 601h is equal to or smaller than the third threshold value. In the example of FIG. 21, in the moving body regions 601f and 601g, there is no moving body region 601 whose distance from the moving body region 601h is equal to or smaller than the third threshold value. Therefore, as shown in FIG. 22, the fine movement specifying unit 33 determines the moving body area 601h that is the reference moving body area 601 as one second group 620C1 (step s454). Thereby, the third group determination process ends.

  Next, the fine movement identifying unit 33 removes the remaining parts excluding the second group 620C1 determined in the most recent group determination process from the moving object areas 601f to 601h that are currently processed, that is, the moving object areas 601f and 601g. A fourth group determination process is performed as a new process target.

  The fine movement specifying unit 33 determines the reference moving body area 601 from the moving body areas 601f and 601g to be processed in the fourth group determination process (step s451). For example, the fine movement specifying unit 33 sets the moving body area 601f as the reference moving body area 601.

  Next, the fine movement specifying unit 33 determines whether or not the moving body area 601g is the moving body area 601 whose distance from the moving body area 601f is equal to or smaller than the third threshold value (step s452). At this time, the fine movement specifying unit 33 obtains the distance between the moving body area 601f and the moving body area 601g as shown in FIG. Then, when the obtained distance is equal to or smaller than the third threshold value, fine movement identifying unit 33 determines that moving object region 601g is moving object region 601 whose distance from moving object region 601f is equal to or smaller than the third threshold value. On the other hand, if the obtained distance is less than the third threshold value, fine movement specifying unit 33 determines that moving object region 601g is not moving object region 601 whose distance from moving object region 601f is equal to or smaller than the third threshold value. In the example of FIG. 23, the moving body region 601g is a moving body region 601 whose distance from the moving body region 601h is equal to or smaller than a third threshold value. As shown in FIG. 24, the fine movement specifying unit 33 determines the moving body areas 601f and 601g as one second group 620D1 (step s454). This completes the fourth group determination process.

  Next, as a result of excluding the second group 620D1 determined in the most recent group determination process from the moving object areas 601f and 601g that are the current processing target, the fine movement specifying unit 33 does not leave any moving object area 601. The second group 620 cannot be determined, the first second grouping process is terminated, and the next second grouping process is executed.

  When executing the second grouping process for the second and subsequent times, the fine movement specifying unit 33 moves the moving object constituting the second group 620 for each second group 620 determined in the latest second grouping process. The second inclusion area 630 including the area 601 is set as a new moving object area 601. Then, the fine movement specifying unit 33 executes the first group determination process of the second grouping process with the determined new moving object region 601 as a processing target.

  In the first second grouping process shown in FIGS. 17 to 24 described above, the second groups 620A1 to 620D1 are determined. Therefore, the fine movement specifying unit 33 sets the second inclusion area 630 including the moving body area 601 constituting the second group 620 for each of the second groups 620A1 to 620D1 as a new moving body area 601. Specifically, as shown in FIG. 25, the second inclusion area 630A1 including the moving object areas 601a to 601x constituting the second group 620A1 is set as a new moving object area 601A1. Further, the second inclusion region 630B1 including the moving body regions 601d and 601e constituting the second group 620B1 is set as a new moving body region 601B1. Further, the second inclusion area 630C1 including the moving body area 601h constituting the second group 620C1 is set as a new moving body area 601C1. Then, the second inclusion region 630D1 including the moving body regions 601f and 601g constituting the second group 620D1 is set as a new moving body region 601D1. In this example, for the second group 620 composed of a plurality of moving body regions 601 as in the second group 620A1, 620B1, 620D1, the circumscribed rectangle of the plurality of moving body regions 601 is defined as a second inclusion region 630. In addition, for the second group 620 configured by one moving body region 601 as in the second group 620C1, the one moving body region 601 is used as the second inclusion region 630 as it is. The fine movement specifying unit 33 performs the first group determination process of the second grouping process for the second time with the new moving object areas 601A1, 601B1, 601C1, and 601D1 as processing targets.

  The fine movement specifying unit 33 performs the second grouping until the number of second groups 620 determined in the second grouping process does not decrease below the number of second groups 620 determined in the preceding second grouping process. When the process 620 is repeatedly executed, the first grouping process is terminated. In other words, when the fine movement identifying unit 33 repeatedly executes the second grouping process until the second group 620 composed of the plurality of moving object regions 601 cannot be determined in the second grouping process, the first grouping process Exit.

  In the example shown in FIG. 25 described above, the processing objects in the first group determination process of the second grouping process for the second time are moving object areas 601A1, 601B1, 601C1, and 601D1. In the first group determination process of the second grouping process for the second time, the fine movement specifying unit 33 sets, for example, the moving body area 601A1 among the moving body areas 601A1, 601B1, 601C1, and 601D1 as the reference moving body area 601.

  Next, the fine movement specifying unit 33 determines whether there is another moving body area 601 whose distance from the moving body area 601A1 is equal to or smaller than the third threshold in the moving body areas 601A1, 601B1, 601C1, and 601D1 (Step S1). s452). At this time, as shown in FIG. 26, the fine movement specifying unit 33 obtains the distance between the moving object area 601A1 and each of the moving object areas 601B1, 601C1, and 601D1. And the fine movement specific | specification part 33 determines whether each calculated | required distance is below a 3rd threshold value. In the example of FIG. 26, the moving body region 601 whose distance from the moving body region 601A1 is equal to or smaller than the third threshold value is the moving body region 601B1. As shown in FIG. 27, the fine movement specifying unit 33 determines the moving body areas 601A1 and 601B1 as one second group 620A2 (step s453). Thereby, the first group determination process is completed.

  Next, the fine movement specifying unit 33 removes the second group 620A2 determined in the most recent group determination process from the moving object areas 601A1, 601B1, 601C1, and 601D1 that are currently processed, that is, the moving object area 601C1. , 601D1 as a new process target, the second group determination process is performed.

  In the second group determination process, for example, the fine movement specifying unit 33 sets the moving body area 601C1 as the reference moving body area 601 out of the moving body areas 601C1 and 601D1. Next, the fine movement specifying unit 33 determines whether there is another moving body area 601 whose distance from the moving body area 601C1 is equal to or smaller than the third threshold in the moving body areas 601C1 and 601D1 (step s452). At this time, as shown in FIG. 28, the fine movement specifying unit 33 obtains the distance between the moving body area 601C1 and the moving body area 601D1. And the fine movement specific | specification part 33 determines whether the calculated | required distance is below a 3rd threshold value. In the example of FIG. 28, in the moving body regions 601C1 and 601D1, there is no other moving body region 601 whose distance from the moving body region 601C1 is equal to or less than the third threshold. Therefore, as shown in FIG. 29, the fine movement specifying unit 33 determines the moving body area 601C1 that is the reference moving body area 601 as one second group 620B2 (step s454). Thereby, the second group determination process is completed.

  Next, the fine movement specifying unit 33 adds the remaining part excluding the second group 620B2 determined in the most recent group determination process from the moving object areas 601C1 and 601D1 that are the current processing targets, that is, the moving object area 601D1 to a new one. A third group determination process is performed as a processing target. The fine movement specifying unit 33 sets the moving body area 601D1 as the reference moving body area 601. In the third group determination process, the processing target is configured by one moving body region 601D1, and therefore there is another moving body region 601 whose distance from the reference moving body region 601 is equal to or smaller than the third threshold value in the processing target. do not do. Therefore, as shown in FIG. 30, the fine movement specifying unit 33 determines the moving body area 601D1 that is the reference moving body area 601 as one second group 620C2 (step s454). Thereby, the third group determination process ends.

  Next, as a result of excluding the second group 620C2 determined by the most recent group determination process from the moving object region 601D1 that is the current processing target, the fine movement specifying unit 33 does not leave any moving object region 601. The group 620 cannot be determined, and the second second grouping process is terminated and the third second grouping process is executed.

  When performing the second grouping process for the third time, the fine movement specifying unit 33 performs the second group for each of the second groups 620A2, 620B2, and 620C2 determined by the second grouping process for the second time most recently. The second inclusion region 630 including the moving object region 601 constituting the 620 is set as a new moving object region 601. As shown in FIG. 31, the inclusion area 630A2 including the moving body areas 601A1 and 601B constituting the second group 620A2 becomes a new moving body area 601A2, and the inclusion area 630B2 including the moving body area 601C1 constituting the second group 620B2. Becomes a new moving body region 601B2, and the inclusion region 630C2 including the moving body region 601D1 constituting the second group 620C2 becomes a new moving body region 601C2. The fine movement specifying unit 33 executes the first group determination process of the third grouping process for the third time, with the determined new moving object areas 601A2, 601B2, and 601C2 as processing targets.

  In the first group determination process of the second grouping process for the third time, the fine movement specifying unit 33 sets, for example, the moving body area 601A2 among the moving body areas 601A2, 601B2, and 601C2 to be processed as the reference moving body area 601. Next, the fine movement specifying unit 33 determines whether there is another moving body area 601 whose distance from the moving body area 601A2 is equal to or smaller than the third threshold value in the moving body areas 601A2, 601B2, and 601C2 (step s452). . At this time, as shown in FIG. 32, the fine movement specifying unit 33 obtains the distance between the moving object area 601A2 and each of the moving object areas 601B2 and 601C2. And the fine movement specific | specification part 33 determines whether each calculated | required distance is below a 3rd threshold value. In the example of FIG. 32, in the moving body regions 601A2, 601B2, and 601C2, there is no moving body region 601 whose distance from the moving body region 601A2 is the third threshold or less. Therefore, as shown in FIG. 33, the fine movement identifying unit 33 determines the moving body area 601A2 that is the reference moving body area 601 as one second group 620A3 (step s454). Thereby, the first group determination process is completed.

  Next, the fine movement specifying unit 33 removes the second group 620A3 determined in the most recent group determination process from the moving object areas 601A2, 601B2, and 601C2 that are currently processed, that is, the moving object areas 601B2 and 601C2. As a new process target, the second group determination process is performed. In the second group determination process, for example, the moving object region 601B2 of the moving object regions 601B2 and 601C2 to be processed is set as the reference moving object region 601. Next, the fine movement specifying unit 33 determines whether there is another moving body area 601 whose distance from the moving body area 601B2 is equal to or smaller than the third threshold in the moving body areas 601B2 and 601C2 (step s452). In the example of FIG. 32, in the moving body regions 601B2 and 601C2, there is no moving body region 601 whose distance from the moving body region 601B2 is equal to or less than the third threshold. Therefore, the fine movement specifying unit 33 determines the moving body area 601B2 that is the reference moving body area 601 as one second group 620B3. Thereby, the second group determination process is completed.

  Next, the fine movement specifying unit 33 adds the remaining part excluding the second group 620B3 determined in the most recent group determination process from the moving object areas 601B2 and 601C2 that are currently processed, that is, the moving object area 601C2 to a new one. A third group determination process is performed as a processing target. Then, the fine movement specifying unit 33 sets the moving body area 601C2 as the reference moving body area 601. In the third group determination process, since the processing target is configured by one moving body region 601C2, there is another moving body region 601 whose distance from the reference moving body region 601 is equal to or smaller than the third threshold value in the processing target. do not do. Therefore, the fine movement specifying unit 33 determines the moving body area 601C2 that is the reference moving body area 601 as one second group 620C3 (step s454). Thereby, the third group determination process ends.

  As shown in FIG. 34, in the second second grouping process, second groups 620A3, 620B3, and 620C3 are determined.

  The number of second groups 620 determined in the third second grouping determination process is not smaller than the number of second groups 620 determined in the second second grouping process. In other words, each of the second groups 620A3, 620B3, and 620C3 determined in the third second grouping process includes one moving object region 601, and in the third second grouping determination process, a plurality of The second group 620 constituting the moving object region 601 is not determined. Therefore, the fine movement specifying unit 33 performs the second operation until the number of second groups 620 determined in the second grouping process does not decrease below the number of second groups 620 determined in the preceding second grouping process. This means that the grouping process 620 has been repeatedly executed, and the first grouping process is terminated. In other words, the fine movement specifying unit 33 repeatedly executes the second grouping process until the second group 620 composed of the plurality of moving body regions 601 cannot be determined, and ends the first grouping process.

  In the first grouping process as described above, the first group 602 described above is formed by the moving body region 601 in the moving body region group 610 that is the basis of the moving body region 601 that constitutes the finally obtained second group 620. Composed. In the example of FIG. 34, the first group 602A (see FIG. 14) is composed of the moving body regions 601a to 601e in the moving body region group 610, which is the basis of the moving body region 601A2 that forms the second group 620A3. Further, the first group 602C (see FIG. 14) is configured by the moving body region 601h in the moving body region group 610, which is the basis of the moving body region 601B2 constituting the second group 620B3. And the 1st group 602B (refer FIG. 14) is comprised by the moving body area | region 601f and 601g in the moving body area | region group 610 used as the origin of the moving body area | region 601C2 which comprises the 2nd group 620C3.

  The fine movement specifying unit 33 sets the moving body area constituting the second group 620 for each second group 620 determined in the last second grouping process in the first grouping process as the first inclusion area described above. 603. In the example of FIG. 34, the moving body region 601A2 constituting the second group 620A3 is the first inclusion region 603A (see FIG. 14). In addition, the moving body region 601B2 constituting the second group 620B3 is the first inclusion region 603C (see FIG. 14). And the moving body area | region 601C2 which comprises the 2nd group 620C3 becomes the 1st inclusion area | region 603B (refer FIG. 14).

  As described above, in the first grouping process, by repeatedly executing the second grouping process in which the group determination process is repeatedly performed, the number of times of obtaining the distance between the moving object regions can be reduced. Therefore, the first grouping process is simplified.

<Details of the first map update process>
Next, the first map update process in step s36 will be described in detail.

  FIG. 35 shows the first frequency map 410. In the first frequency map 410, a plurality of first frequency values 411 respectively corresponding to a plurality of input pixels included in the input frame image 111 are arranged in accordance with the positions of the plurality of input pixels. The relative position of the first frequency value 411 in the first frequency map 410 matches the relative position of the input pixel corresponding to the first frequency value 411 in the input frame image 111.

  In the first map update process in step s36, the first map update unit 41 includes the first map corresponding to the input pixel included in the input frame image 111 and determined to be the foreground pixel in the provisional determination process in step s33. The frequency value 411 is increased by a predetermined value. This predetermined value is referred to as a “first increase amount”. On the other hand, the first map update unit 41 sets the first frequency value 411 corresponding to the input pixel determined to be the background pixel in the provisional determination process in step s33, included in the input frame image 111, by a predetermined value. Decrease. This predetermined value is referred to as a “first decrease amount”. The first increase amount and the first decrease amount may be the same as each other or different from each other. For the first frequency value 411, a minimum value and a maximum value are determined. The minimum value is, for example, zero. Therefore, when the first frequency value 411 corresponding to the input pixel determined to be the foreground pixel indicates the maximum value, the first frequency value 411 does not increase. Further, when the first frequency value 411 corresponding to the input pixel determined to be the background pixel indicates the minimum value, the first frequency value 411 is not decreased.

  By updating the first frequency map 410 in this way, in the first frequency map 410, the first frequency value 411 corresponding to the input pixel that is frequently determined to be the foreground pixel in the input frame image 111 is increased. . In other words, in the first frequency map 410, the first frequency value 411 corresponding to an area where people frequently appear in the imaging area increases. Therefore, by referring to the first frequency map 410, it is possible to specify an area where there is a high possibility that a person exists in the imaging area.

<Details of the second map update process>
Next, the second map update process in step s37 will be described in detail.

  FIG. 36 is a diagram showing the second frequency map 420. Similar to the first frequency map 410, in the second frequency map 420, a plurality of second frequency values 421 respectively corresponding to a plurality of input pixels included in the input frame image 111 are set according to the positions of the plurality of input pixels. Are lined up. The relative position of the second frequency value 421 in the second frequency map 420 coincides with the relative position of the input pixel corresponding to the second frequency value 421 in the input frame image 111.

  In the second map update process in step s37, the second map update unit 42 decreases each second frequency value 421 of the second frequency map 420 by a predetermined value when the fine movement occurrence area is not specified in step s35. . Hereinafter, this predetermined value is referred to as a “second decrease amount”.

  On the other hand, when the fine movement occurrence area is specified in step s35, the second map update unit 42 basically stores the fine movement corresponding area in the first frequency map 410 corresponding to the specified fine movement occurrence area. The first frequency value 411 is used as a second frequency value 421 corresponding to the first frequency value 411 in the second frequency map 420, and a copy process is executed. In other words, the second map updating unit 42 basically has a function in the fine movement corresponding area that exists at the same relative position as the relative position in the imaging area of the specified fine movement occurrence area in the first frequency map 410. The first frequency value 411 is used as a second frequency value 421 corresponding to the first frequency value 411 in the second frequency map 420, and a copy process is executed. Then, the second map updating unit 42 decreases the second frequency value 421 other than the second frequency value 421 for which the copy process is executed in the second frequency map 420 by the second decrease amount. Unlike the first frequency value 411, the second map updating unit 42 may increase the second frequency value 421 by a predetermined value, even though the first frequency value 411 may be adopted as the second frequency value 421. Absent. Hereinafter, the second map update process when the fine movement occurrence area is specified in step s35 will be described in detail.

<Second map update process when a fine movement occurrence area is specified>
FIG. 37 is a flowchart showing the second map update process when the fine movement occurrence area is specified. When the fine movement occurrence area is specified in step s35, the second map updating unit 42 includes a person (detection target moving object) in the imaging area in the first frequency map 410 updated in step s36 in step s371. A plurality of detection target corresponding areas that are independent of each other and that correspond to areas that are likely to be detected are identified.

  In step s371, the second map update unit 42 first binarizes the first frequency map 410 to generate a binarized map. Specifically, the second map updating unit 42 sets each first frequency value 411 larger than “0” to “1” in the first frequency map 410, and sets the first frequency value 411 of “0”. Set to “0” to generate a binarized map. Next, the second map update unit 42 performs labeling using 4-connection, 8-connection, or the like on the region where the value indicates “1” in the binarized map. Then, the second map update unit 42 specifies a corresponding region in the first frequency map 410 corresponding to the independent region for each independent region obtained by labeling from the binarized map. This corresponding area is referred to as an “independent area corresponding area”.

  The second map update unit 42 obtains a circumscribed rectangle of each independent area corresponding area for each independent area corresponding area in the first frequency map 410. Then, the second map updating unit 42 sets each circumscribed rectangle obtained as a detection target corresponding region.

  Here, the independent area corresponding area of the first frequency map 410 is an area where the first frequency value 411 indicates “1” or more. Therefore, in the first frequency map 410, the independent region corresponding region can be said to be a region corresponding to a region where there is a high possibility that a person exists in the imaging region. Therefore, in the first frequency map 410, the circumscribed rectangle of the independent area corresponding area, that is, the detection target corresponding area can also be said to be an area corresponding to an area where there is a high possibility that a person exists in the imaging area.

  The second map updating unit 42 sets each first frequency value 411 larger than the reference value (≧ 1) to “1” in the first frequency map 410, and the first frequency value 411 equal to or lower than the reference value. May be set to “0” to generate a binarized map.

  Next, in step s372, the second map updating unit 42 specifies a detection target corresponding region including a fine movement corresponding region in the detection target corresponding region specified in step s371. Hereinafter, the detection target corresponding area including the fine movement corresponding area is referred to as a “specific detection target corresponding area”. The specific detection target corresponding area may include a plurality of fine movement corresponding areas.

  Next, in step s373, the second map update unit 42 corresponds the first frequency value 411 in the specific detection target corresponding area specified in step s372 to the first frequency value 411 in the second frequency map 420. The second frequency value 421 is adopted. That is, the second map updating unit 42 uses the first frequency value 411 in the specific detection target corresponding region as the relative position of the first frequency value 411 in the first frequency map 410 in the second frequency map 420. The second frequency value 421 existing at the same relative position is adopted. Thereafter, adopting the first frequency value 411 in the specific detection target corresponding region as the second frequency value 421 corresponding to the first frequency value 411 in the second frequency map 420 is used in the specific detection target corresponding region. This is expressed as a copy of the first frequency value 411 to the second frequency map 420.

  The operation of the second map updating unit 42 when the second map updating unit 42 copies the first frequency value in the specific detection target corresponding region to the second frequency map 420 is whether the area of the specific detection target corresponding region is large. It depends on whether or not.

  When the second map updating unit 42 copies the first frequency value in the specific detection target corresponding area to the second frequency map 420, is the area of the specific detection target corresponding area equal to or greater than the fourth threshold value? Determine whether or not. When the area of the specific detection target corresponding region is less than the fourth threshold value, the second map update unit 42 determines that the area of the specific detection target corresponding region is small, and the plurality of first detection points in the specific detection target corresponding region. Of the frequency values 411, only the first frequency value 411 equal to or greater than the fifth threshold value is copied to the second frequency map 420. When all the first frequency values 411 of the specific detection target corresponding area whose area is less than the fourth threshold are less than the fifth threshold, the second map updating unit 42 The first frequency value 411 is not copied to the second frequency map 420. In addition, when the area of the specific detection target corresponding region is less than the fourth threshold value, the second map update unit 42 sets the fifth threshold value among the plurality of first frequency values 411 in the specific detection target corresponding region. Only a larger first frequency value 411 may be copied to the second frequency map 420.

  FIG. 38 is a diagram illustrating an example of a state in which the first frequency value 411 in the specific detection target corresponding region is copied to the second frequency map 420. As shown in FIG. 38, it is assumed that three fine movement occurrence areas 810a to 810c are specified in the imaging area 800 of the camera 2, and four detection target corresponding areas 416A to 416D are specified in the first frequency map 410. The second map updating unit 42 determines whether or not the area of the specific detection target corresponding area 416A including the fine movement corresponding area 415a corresponding to the fine movement occurrence area 810a in the first map 410 is equal to or larger than the fourth threshold value. To do. Further, the second map update unit 42 determines whether or not the area of the specific detection target corresponding region 416B including the fine motion corresponding region 415b corresponding to the fine motion occurrence region 810b in the first map 410 is equal to or larger than the fourth threshold value. Determine. Further, the second map updating unit 42 determines whether or not the area of the specific detection target corresponding region 416D including the fine motion corresponding region 415c corresponding to the fine motion occurrence region 810c in the first map 410 is equal to or larger than the fourth threshold value. Determine. In the example of FIG. 38, the areas of the specific detection target corresponding regions 416A, 416B, and 416D are less than the fourth threshold value. The detection target corresponding area 416C does not include the fine movement corresponding area.

  The second map update unit 42 copies the first frequency value 411 equal to or higher than the fifth threshold value among the first frequency values 411 in the specific detection target corresponding area 416 </ b> A to the second frequency map 420. The second map update unit 42 copies the first frequency value 411 equal to or higher than the fifth threshold value among the first frequency values 411 in the specific detection target corresponding area 416B to the second frequency map 420. Further, the second map update unit 42 copies the first frequency value 411 equal to or higher than the fifth threshold value among the first frequency values 411 in the specific detection target corresponding area 416D to the second frequency map 420. In the example of FIG. 38, some of the first frequency values 411 of the first frequency values 411 in the specific detection target corresponding area 416A are less than the fifth threshold value, and therefore the first frequency in the specific detection target corresponding area 416A. Only some of the first frequency values 411 of the values 411 are copied. Since all of the first frequency values 411 in the specific detection target corresponding area 416B are not less than the fifth threshold value, all of the first frequency values 411 in the specific detection target corresponding area 416C are copied. Since all of the first frequency values 411 in the specific detection target corresponding area 416D are less than the fifth threshold value, all of the first frequency values 411 in the specific detection target corresponding area 416D are not copied.

  The second map update unit 42 determines that the area of the specific detection target corresponding region is large when the area of the specific detection target corresponding region is equal to or larger than the fourth threshold, and handles fine movement included in the specific detection target corresponding region. Find the bounding rectangle of the region. Then, the second map updating unit 42 copies the first frequency value 411 equal to or higher than the fifth threshold value among the obtained first frequency values 411 in the circumscribed rectangle to the second frequency map 420. Note that the second map updating unit 42 may copy the first frequency value 411 larger than the fifth threshold value to the second frequency map 420 among the first frequency values 411 in the obtained circumscribed rectangle.

  When the specific detection target corresponding area includes only one fine movement corresponding area, the one fine movement corresponding area (rectangular area) becomes a circumscribed rectangle as it is.

  On the other hand, when only the plurality of fine movement corresponding areas are included in the specific detection target corresponding area, the second map updating unit 42 selects a plurality of fine movement corresponding areas that are close to each other with respect to the specific detection target corresponding area. A fine movement grouping process for one fine movement group is performed. The fine movement grouping process can be performed, for example, in the same manner as the first grouping process described above.

  For each fine movement group obtained as a result of the fine movement grouping process, the second map update unit 42 obtains a circumscribed rectangle of a plurality of fine movement corresponding areas constituting the fine movement group. Then, the second map updating unit 42 copies the first frequency value 411 equal to or higher than the fifth threshold value among the obtained first frequency values 411 in the circumscribed rectangle to the second frequency map 420. When the fine movement group is composed of only one fine movement corresponding area, the single fine movement corresponding area is directly a circumscribed rectangle.

  39 and 40 are diagrams illustrating an example of a state in which the first frequency value 411 in the fine movement group is copied to the second frequency map 420. FIG. In the example of FIG. 39, the area of the specific detection target corresponding area 416A including the three fine movement corresponding areas 415a to 415c is equal to or larger than the fourth threshold value. Three fine movement corresponding regions 415a to 415c constitute one fine movement group 417A. In this case, the second map updating unit 42 obtains a circumscribed rectangle 418A of the three fine movement corresponding regions 415a to 415c. Then, the second map updating unit 42 copies the first frequency value 411 equal to or higher than the fifth threshold value among the first frequency values 411 of the circumscribed rectangle 418 </ b> A to the second frequency map 420.

  In the example of FIG. 40, the area of the specific detection target corresponding area 416A including the six fine movement corresponding areas 415a to 415f is equal to or larger than the fourth threshold value. Three fine movement corresponding areas 415a to 415c constitute one fine movement group 417A, and three fine movement corresponding areas 415d to 415f constitute one fine movement group 417B. In this case, the second map updating unit 42 obtains a circumscribed rectangle 418A of the three fine movement corresponding areas 415a to 415c and a circumscribed rectangle 418B of the three fine movement corresponding areas 415d to 415f. Then, the second map updating unit 42 copies the first frequency value 411 equal to or higher than the fifth threshold value among the first frequency values 411 of the circumscribed rectangle 418A to the second frequency map 420, and the first frequency value 411 of the circumscribed rectangle 418B. Of the frequency values 411, the first frequency value 411 equal to or higher than the fifth threshold value is copied to the second frequency map 420.

  The second map update unit 42 determines whether or not the area of the specific detection target corresponding region is larger than the fourth threshold value. When the area of the specific detection target corresponding region is equal to or smaller than the fourth threshold value, When the area of the specific detection target corresponding region is determined to be small and the area of the specific detection target corresponding region is larger than the fourth threshold, it may be determined that the area of the specific detection target corresponding region is large.

  As described above, the first frequency value 411 in the specific detection target corresponding area of the first frequency map 410 is copied to the second frequency map 420. In the second frequency map 420, the second map updating unit 42 decreases the second frequency value 421 other than the second frequency value 421 for which the copy process is executed by the second decrease amount.

<Details of foreground identification processing>
As can be understood from the above description, the first frequency value 411 is large in the detection target corresponding region in the first frequency map 410 corresponding to the region in which the person is likely to exist in the imaging region. The specific detection target corresponding area corresponds to an area where there is a high possibility that a person exists in the imaging area, and includes a fine movement corresponding area corresponding to a fine movement occurrence area in the imaging area. Therefore, the specific detection target corresponding region is very likely to correspond to a region in the imaging region where a person who is moving slightly exists, and the first frequency value 411 is large. Therefore, by adopting the first frequency value 411 in the specific detection target corresponding region as the second frequency value 421 corresponding to the first frequency value 411 in the second frequency map 420, the second frequency map 420 The second frequency value 421 corresponding to an area where there is a finely moving person in the imaging area increases. Therefore, in the second frequency map 420, an area where the second frequency value 421 is larger than “0” is likely to correspond to the foreground image in the input frame image 111.

  Therefore, in the foreground specifying process in step s38, the foreground specifying unit 34 sets a region where the second frequency value 421 is larger than “0” in the second frequency map 420 as a foreground corresponding region. Then, the foreground identifying unit 34 identifies an area corresponding to the foreground corresponding area in the input frame image 111 to be processed as a foreground image.

  FIG. 41 is a diagram illustrating an example of a state in which a foreground image is specified based on the second frequency map 420. In the example of FIG. 41, the second frequency map 420 includes three regions 425a to 425c as regions where the second frequency value 421 is larger than “0”. The foreground identification unit 34 sets the areas 425a to 425c as the foreground corresponding areas 425a to 425c, respectively. Then, the foreground specifying unit 34 specifies an area corresponding to the foreground corresponding area 425a in the input frame image 111 to be processed as the foreground image 112a. In other words, the foreground specifying unit 34 specifies, as the foreground image 112a, a region that exists at the same relative position as the relative position of the foreground corresponding region 425a in the second frequency map 420 in the input frame image 111 to be processed. . Similarly, the foreground specifying unit 34 specifies a region corresponding to the foreground corresponding region 425b as the foreground image 112b in the input frame image 111 to be processed, and specifies a region corresponding to the foreground corresponding region 425c as the foreground image 112c. Basically, the foreground images 112a to 112c correspond to a plurality of different people, respectively. Therefore, in the example of FIG. 41, there are three people who are stationary in the imaging region.

<How to find the distance between rectangular areas>
The image processing apparatus 3 may obtain a distance between two rectangular areas in the fine movement specifying process, the second map update process, and the like. For example, in the above-described step s353, a first grouping process is performed on the moving object region group 610 to make a plurality of moving object regions 601 that are close to each other as one first group 602. The moving object region 601 is a rectangular region. For this reason, the image processing apparatus 3 needs to obtain a distance between two rectangular areas in the first grouping process. Further, in the second map update process, a fine movement grouping process is performed on the specific detection target corresponding area by using a plurality of fine movement corresponding areas that are close to each other as one fine movement group, and the fine movement corresponding area is a rectangular area. Therefore, the image processing apparatus 3 needs to obtain a distance between two rectangular areas in the fine movement grouping process.

  As the distance between the two rectangular areas, the distance between the center (centroid) of one rectangular area and the center (centroid) of the other rectangular area can be adopted. Further, as the distance between the two rectangular regions, the shortest distance between the outer shape of one rectangular region and the outer shape of the other rectangular region can be employed. An example of how to determine the shortest distance will be described below.

  FIG. 42 is a diagram for explaining how to obtain the shortest distance between the outer shape of one rectangular region and the outer shape of the other rectangular region. In the two rectangular areas, the image processing apparatus 3 uses one as the first rectangular area 701 and the other as the second rectangular area 702. 42, the image processing apparatus 3 includes a right upper area 710, a right lower area 711, a right right area 712, a right left area 713, a right upper area 714, an upper left area 715, a lower left area 716, and a lower right area. Consider 717.

  Here, the directly above area 710 is an area located directly above the first rectangular area 701. The directly below area 711 is an area located directly below the first rectangular area 701. The right area 712 is an area located right of the first rectangular area 701. The true left area 713 is an area located just to the left of the first rectangular area 701. The upper right area 714 is an area located at the upper right of the first rectangular area 701. The upper left area 715 is an area located at the upper left of the first rectangular area 701. The lower left area 716 is an area located at the lower left of the first rectangular area 701. The lower right area 717 is an area located at the lower right of the first rectangular area 701.

  When at least a part of the second rectangular area 702 exists in any one of the directly above area 710, the directly below area 711, the exactly right area 712, and the exactly left area 713, the image processing apparatus 3 performs the first rectangular area 701. The distance between the two opposing sides in the second rectangular area 702 is the shortest distance between the outer shape of the first rectangular area 701 and the outer shape of the second rectangular area 702. In other words, the image processing apparatus 3 determines that the first rectangular area 702 is present when at least a part of the second rectangular area 702 is located directly above, directly below, right or left of the first rectangular area 701. The distance between two opposing sides in 701 and the second rectangular area 702 is the shortest distance between the outer shape of the first rectangular area 701 and the outer shape of the second rectangular area 702.

  On the other hand, when the entire area of the second rectangular area 702 exists in only one of the upper right area 714, the upper left area 715, the lower left area 716, and the lower right area 717, the image processing apparatus 3 The distance between the nearest corners in the rectangular area 701 and the second rectangular area 702 is defined as the shortest distance between the outer shape of the first rectangular area 701 and the outer shape of the second rectangular area 702.

  As shown in FIG. 42, for example, the shortest distance between the outer shape of the second rectangular region 702a partially existing in the upper region 710 and the outer shape of the first rectangular region 701 is below the second rectangular region 702a. This is the distance Da between the side 702aa on the side and the side 701a on the upper side of the first rectangular region 701 facing the side 702aa.

  Further, as shown in FIG. 42, the shortest distance between the outer shape of the second rectangular region 702b and the outer shape of the first rectangular region 701 partially present in the right-right region 712 is the left side of the second rectangular region 702b. The distance Db between the side 702bb of the first rectangular region 701 and the side 701b on the right side of the first rectangular region 701.

  As shown in FIG. 42, the shortest distance between the outer shape of the second rectangular region 702c and the outer shape of the first rectangular region 701 existing only in the lower left region 716 is the upper right corner 702cc of the second rectangular region 702c. And the distance Dc between the left corner 701c of the first rectangular area 701.

  As described above, when the image processing apparatus 3 identifies a still person image in the input frame image 111 as a foreground image, the image processing apparatus 3 outputs the identification result. The result of specifying the foreground image in the image processing device 3 is used, for example, in air conditioning control in the room 300. The air conditioning control device that controls the air conditioning of the room 300 identifies a region in the room 300 where a stationary person exists based on the foreground image identification result of the image processing apparatus 3. And an air-conditioning control apparatus performs air-conditioning control so that the temperature of the specified area | region approaches fixed temperature. Note that the foreground image identification result in the image processing apparatus 3 may be used in addition to air conditioning control.

  As described above, in the image processing device 3, when the fine movement occurrence area is specified in the imaging area, the first frequency value 411 in the fine movement corresponding area corresponding to the specified fine movement occurrence area in the first frequency map 410 is obtained. In the second frequency map 420, the second frequency value 421 corresponding to the first frequency value 411 is adopted.

  On the other hand, even when the stationary person pixel included in the input frame image 111 is not determined as the foreground pixel by the determination unit 31 due to the update of the background model 500, the first frequency map 410 The first frequency value 411 of the region corresponding to the stationary person does not immediately become zero. That is, after the background model 500 is updated, the stationary human pixel included in the input frame image 111 is not determined as the foreground pixel by the determination unit 31, and then the stationary frame in the first frequency map 410 is stationary. It takes time until the first frequency value 411 of the region corresponding to the person becomes zero.

  Accordingly, in the second frequency map 420, the first frequency value 411 in the fine movement corresponding region corresponding to the fine movement occurrence region of the photographing region in the first frequency map 410 is copied to the second frequency map 420, so The second frequency value 421 in the area corresponding to the fine movement occurrence area where the fine movement of the person is occurring in the area is immediately after the determination unit 31 no longer detects the pixel of the stationary person as the foreground pixel. It will not be zero. For example, in the second frequency map 420, the second frequency value 421 in the area corresponding to the fine movement occurrence area where the fine movement of the person is occurring in the imaging area is determined by the determination unit 31 in the foreground. It will not go to zero for several hundred frames after it is no longer detected as a pixel. Therefore, even after the human pixel is no longer detected as the foreground pixel, the determination unit 31 specifies the still human image in the input frame image 111 as the foreground image based on the second frequency map 420. be able to. As a result, the accuracy of specifying the foreground image in the input moving image 110 is improved.

  In addition, a plurality of fine movement corresponding areas in the first frequency map 410 may correspond to the same person's fine movement. In this case, when the first frequency value 411 in the fine movement corresponding area is copied to the second frequency map 420 for each of the plurality of fine movement corresponding areas, a plurality of foreground images respectively corresponding to the plurality of fine movement corresponding areas are identified. Is done. The plurality of foreground images are images of the same person. When counting the number of people who are stationary in the shooting region based on the foreground image identification result in the image processing device 3, a plurality of foreground images identified in the image processing device 3 are recorded by one person. If it is an image or an image of a plurality of people, there is a possibility that the number of stationary people cannot be counted correctly.

  On the other hand, the detection target corresponding region in the first frequency map 410 is an independent detection target corresponding region corresponding to a region where there is a high possibility that a person exists in the imaging region, and thus corresponds to one person. Probability is high. In this example, in the image processing device 3, the first frequency value 411 in the detection target corresponding region (specific detection target corresponding region) including the fine motion corresponding region is copied to the second frequency map 420 in the first frequency map 410. Therefore, the foreground image corresponding to the detection target region in the input frame image 111 is highly likely to be an image of one person. Therefore, based on the foreground image specifying result in the image processing device 3, the number of still persons in the shooting area can be counted more correctly.

  Further, the determination unit 31 may erroneously determine that a human pixel is a background pixel due to a noise component or the like included in the input frame image 111. Therefore, in the specific detection corresponding region, the region where the first frequency value 411 is small is not very likely to correspond to the foreground pixel.

  In this example, among the first frequency values 411 in the specific detection target corresponding region, the first frequency value 411 that is equal to or higher than the fifth threshold value or larger than the fifth threshold value is copied to the second frequency map 420. The first frequency value 411 corresponding to the background pixel can be prevented from being copied to the second frequency map 420. Therefore, the accuracy of specifying the foreground image is further improved.

  In addition, one detection target corresponding region may be obtained corresponding to a plurality of people who are firmly present in the imaging region. Then, the area of the detection target corresponding region becomes large.

  On the other hand, even when the person no longer exists in the area where the person exists in the imaging area, the first frequency value 411 of the first frequency map 410 corresponding to the area does not immediately become zero. Therefore, in the imaging region, when a certain person from the plurality of people no longer exists from an area where a plurality of people are fixed, the imaging region corresponds to the detection target corresponding area corresponding to the plurality of people. The first frequency value 411 corresponding to the area where the certain person is present becomes a large value for a while. Therefore, when all of the first frequency values 411 in the detection target corresponding region having a large area are copied to the second frequency map 420, the first frequency value 411 corresponding to the region where the person no longer exists in the imaging region is obtained. The second frequency map 420 may be copied.

  In this example, when the area of the specific detection target region including the plurality of fine movement corresponding regions is equal to or larger than the fourth threshold value or larger than the fourth threshold value, the second map update unit 42 Instead of copying all the first frequency values 411 to the second frequency map 420, only the first frequency values 411 in the circumscribed rectangles of the plurality of fine movement corresponding regions included in the specific detection target region are used as the second frequency map. It is copied to 420. Thereby, it is possible to suppress copying of the first frequency value 411 corresponding to the area where the person no longer exists in the imaging area to the second frequency map 420. Therefore, the accuracy of specifying the foreground image is further improved.

  Also, a plurality of fine movement corresponding regions corresponding to the same person's fine movement are close to each other. Therefore, in the specific detection target area corresponding to a plurality of persons who are present together, there is a high possibility that a plurality of fine movement corresponding areas existing close to each other correspond to the same person's fine movement.

  In this example, the second map updating unit 42 performs a fine movement grouping process in which a plurality of fine movement corresponding areas that are close to each other are set as one fine movement group for a specific detection target corresponding area including a plurality of fine movement corresponding areas. ing. Then, the second map updating unit 42 copies the first frequency value 411 in the circumscribed rectangle of the plurality of fine movement corresponding areas constituting the fine movement group to the second frequency map 420 for each obtained fine movement group. Yes. Since one fine movement group is likely to correspond to one person, it is possible to obtain a plurality of foreground images respectively corresponding to a plurality of persons who are present together. Therefore, based on the foreground image specifying result in the image processing device 3, the number of still persons in the shooting area can be counted more correctly.

  In addition, the specific detection target corresponding area includes a first fine movement group including P1 (P1 ≧ 2) fine movement corresponding areas that are close to each other, and P2 (P2 ≧ 2) fine movement corresponding areas that are close to each other. When the first fine movement group is separated from the second fine movement group, the region between the first fine movement group and the second fine movement group in the first frequency map 410 is a foreground pixel. Is less likely to correspond. Therefore, when the first frequency value 411 in the circumscribed rectangle of the (P1 + P2) fine movement corresponding regions constituting the first and second fine movement groups is copied to the second frequency map 420, the first frequency value 411 corresponding to the background pixel is copied. May be copied to the second frequency map 420.

  In this example, the second map updating unit 42 copies, for each obtained fine movement group, the first frequency value 411 in the circumscribed rectangle of the plurality of fine movement corresponding areas constituting the fine movement group to the second frequency map 420. Therefore, the first frequency value 411 of the region between the two fine movement groups, that is, the region that is unlikely to correspond to the foreground pixel, can be suppressed from being copied to the second frequency map 420. Therefore, the accuracy of specifying the foreground image is further improved.

  Note that the first frequency value 411 in the specific detection target corresponding area is not copied to the second frequency map 420, but only the first frequency value 411 in the fine movement corresponding area is obtained as the second frequency without obtaining the detection target corresponding area. You may copy to the map 420. FIG. In this case, instead of copying all of the first frequency values 411 in the fine movement corresponding area to the second frequency map 420, the first frequency value 411 in the fine movement corresponding area is equal to or higher than the fifth threshold value or A first frequency value 411 larger than the fifth threshold value may be copied to the second frequency map 420.

  Also, instead of copying the first frequency value 411 that is greater than or equal to the fifth threshold value or greater than the fifth threshold value among the first frequency values 411 in the specific detection target corresponding region, All the first frequency values 411 in the specific detection target corresponding area may be copied to the second frequency map 420.

  Further, all of the first frequency values 411 in the specific detection target area or the first frequency values 411 in the specific detection target area are obtained without obtaining circumscribed rectangles of the plurality of fine movement corresponding areas included in the specific detection target area. Of these, the first frequency value 411 that is greater than or equal to the fifth threshold value or greater than the fifth threshold value may be copied to the second frequency map 420.

  Further, without performing the fine movement grouping process on the specific detection target corresponding area, all of the first frequency values 411 in the specific detection target area or the first frequency values 411 in the specific detection target area, The first frequency value 411 greater than or equal to the fifth threshold value or greater than the fifth threshold value may be copied to the second frequency map 420.

  Further, the second decrease amount for the second frequency map 420 may be larger than the first decrease amount for the first frequency map 410. When the second decrease amount is small, in the second frequency map 420, the second frequency value 421 of the corresponding area corresponding to the area where no person already exists in the imaging area is unlikely to be small. Therefore, there is a possibility that the corresponding area will always be the foreground corresponding area. In addition, when the first reduction amount is large, the pixel corresponding to the stationary person in the first frequency map 410 after the determination unit 31 does not determine the pixel of the stationary person as the foreground pixel. The first frequency value 411 is likely to be small. Therefore, after the determination unit 31 no longer determines the pixel of the person who is stationary as the foreground pixel, the time until the foreground identification unit 34 does not specify the image of the person who is stationary as the foreground image is shortened. End up. By making the second reduction amount larger than the first reduction amount, in the second frequency map 420, the second frequency value 421 of the corresponding region corresponding to the region where no person already exists in the imaging region is likely to be small. Furthermore, the first frequency value 411 of the area corresponding to the stationary person in the first frequency map 410 is difficult to decrease. Therefore, in the second frequency map 420, it is possible to prevent the corresponding area corresponding to the area where no person in the shooting area already exists as the foreground corresponding area, and the foreground specifying unit 34 is stationary. It is possible to lengthen the time until the human image is no longer specified as the foreground image. As a result, the foreground image specifying accuracy is further improved.

  In the above example, the frequency type background update using MOG is used as the background model update method in the image processing apparatus 3, but other types of frequency type background update may be used. For example, background update using a codebook method may be used. Also, background updates other than frequency type background updates may be used.

<Various modifications>
Hereinafter, various modified examples of the image processing apparatus 3 will be described.

<Combination of frequency type background update and continuous type background update>
FIG. 43 is a diagram showing the configuration of the image processing apparatus 3 according to this modification. The image processing apparatus 3 according to this modification includes a determination unit 37 and a background model update unit 38, as compared with the image processing apparatus 3 shown in FIG.

  Each time the input frame image 111 is output from the preprocessing unit 30, the determination unit 37 tentatively determines whether each input pixel of the input frame image 111 is a foreground pixel or a background pixel based on the background model 550. judge.

  The background model 550 is a background model different from the background model 500 and is stored in the storage unit 5. The background model 550 includes a plurality of pieces of background information respectively corresponding to a plurality of input pixels constituting the input frame image 111. The background information included in the background model 550 is composed of pixel values of background pixels (hereinafter referred to as “background pixel values”).

  In the flowchart shown in FIG. 10 described above, the determination unit 37 tentatively determines whether the input pixel of the input frame image 111 to be processed is a foreground pixel or a background pixel after step s32 and before step s38. A second provisional determination process is performed to determine automatically. In the second provisional determination process, the determination unit 37 determines whether or not the pixel value of the target input pixel of the input frame image 111 to be processed matches the background pixel value corresponding to the target input pixel in the background model 550. To do. When the pixel value of the target input pixel matches the background pixel value corresponding to the target input pixel, the determination unit 37 tentatively determines that the target input pixel is a background pixel. On the other hand, when the pixel value of the target input pixel does not match the background pixel value corresponding thereto, the determination unit 37 tentatively determines that the target input pixel is a foreground pixel. Hereinafter, the provisional determination process in the determination unit 31 may be referred to as a “first provisional determination process”.

  The determination unit 37 temporarily determines that the target input pixel is the background pixel when the absolute value of the difference between the pixel value of the target input pixel and the corresponding background pixel value is less than the sixth threshold value. It may be determined that the target input pixel is a foreground pixel when the absolute value is equal to or greater than the sixth threshold value. The determination unit 37 temporarily determines that the target input pixel is the background pixel when the absolute value of the difference between the pixel value of the target input pixel and the background pixel value corresponding thereto is equal to or smaller than the sixth threshold value. If the absolute value is larger than the sixth threshold value, it may be provisionally determined that the target input pixel is a foreground pixel.

  The background model update unit 38 updates the background model 550 based on the input frame image 111 each time the input frame image 111 is output from the preprocessing unit 30. The background model update unit 38 updates the background model 550 by a method different from that of the background model update unit 32. When the pixel value of the target input pixel included in the input frame image 111 shows the same value continuously for a certain period, the background model update unit 38 determines the target input pixel in the background model 550 based on the pixel value. The background pixel value corresponding to is updated. The background model update unit 38 similarly updates each background pixel value of the background model 550.

  FIG. 44 is a flowchart showing the operation of the background model update unit 38 regarding the target input pixel. The background model update unit 38 executes the process shown in FIG. 44 every time the input frame image 111 is output from the preprocessing unit 30. In addition, the background model update unit 38 executes the processing shown in FIG. 44 for each input pixel of the input frame image 111.

  As illustrated in FIG. 44, the background model update unit 38 determines whether or not the determination unit 37 tentatively determines that the target input pixel of the input frame image 111 to be processed is the background pixel in step s41. To do. If the background model update unit 38 determines that the determination unit 37 has provisionally determined that the target input pixel is a background pixel, it resets the count value to zero in step s44. On the other hand, if the background model update unit 38 determines that the target input pixel is a foreground pixel and is temporarily determined by the determination unit 37, the background model update unit 38 executes step s42.

  In step s42, the background model update unit 38 determines whether or not the determination unit 37 has temporarily determined that the target input pixel is the background pixel in the previous frame. That is, the background model update unit 38 determines whether or not the determination unit 37 tentatively determines that the target input pixel in the input frame image 111 immediately before the processing target input frame image 111 is a background pixel. to decide. When the background model update unit 38 determines that the target input pixel is tentatively determined to be the background pixel in the previous frame, in step s43, the background model update unit 38 determines the target input pixel of the input frame image 111 to be processed. The pixel value is set as a reference pixel value. In step s44, the background model update unit 38 resets the count value.

  As described above, when the determination unit 37 tentatively determines that the target input pixel is the foreground pixel in the current frame, the background model update unit 38 is the background pixel in the previous frame. When the determination unit 37 temporarily determines, the pixel value of the target input pixel in the current frame is set as the reference pixel value, and the count value is reset.

  On the other hand, when the background model update unit 38 determines in step s42 that the determination unit 37 temporarily determines that the target input pixel is the foreground pixel in the previous frame, the background model update unit 38 executes step s45. In step s45, the background model update unit 38 determines whether or not the pixel value of the target input pixel of the input frame image 111 to be processed is approximately the same as the reference pixel value. When the absolute value of the difference between the pixel value of the target input pixel and the reference pixel value in the input frame image 111 to be processed is less than the seventh threshold, the background model update unit 38 sets the pixel value of the target input pixel as the reference It is determined that the pixel value is approximately the same. If the absolute value is equal to or greater than the seventh threshold value, the background model update unit 38 determines that the pixel value of the target input pixel is not the same as the reference pixel value. The background model update unit 38 determines that the pixel value of the target input pixel is about the same as the reference pixel value when the absolute value of the difference between the pixel value of the target input pixel and the reference pixel value is equal to or smaller than the seventh threshold value. If the absolute value is larger than the seventh threshold value, it may be determined that the pixel value of the target input pixel is not the same as the reference pixel value.

  If the background model update unit 38 determines that the pixel value of the target input pixel is not comparable to the reference pixel value, it executes step s44 to reset the count value. On the other hand, when the background model update unit 38 determines that the pixel value of the target input pixel is approximately the same as the reference pixel value, the background model update unit 38 increases the count value by one in step s46. In step s47, the background model update unit 38 determines whether or not the count value is equal to or greater than the eighth threshold value. The eighth threshold value is set to “100”, for example.

  When the background model update unit 38 determines that the count value is equal to or greater than the eighth threshold value, the background model update unit 38 updates the background pixel value corresponding to the target input pixel in the background model 550 based on the pixel value of the target input pixel. For example, the background model update unit 38 updates the background pixel value by adopting the reference pixel value as a new background pixel value corresponding to the target input pixel in the background model 550. Note that the background model update unit 38 uses the average value of a plurality of pixel values (including the reference pixel value) for the target input pixel obtained up to the present after the reference pixel value is set as the target in the background model 550. You may employ | adopt as a new background pixel value corresponding to an input pixel.

  When the background model value is updated, the background model update unit 38 executes step s41 for the target input pixel in the next input frame image 111 (new input frame image 111 to be processed). Thereafter, the background model update unit 38 operates in the same manner.

  On the other hand, when the background model update unit 38 determines that the count value is less than the eighth threshold value, the background model update unit 38 executes step s41 for the target input pixel in the next input frame image 111. Thereafter, the background model update unit 38 operates in the same manner.

  The background model update unit 38 may determine whether or not the count value is larger than the eighth threshold value in step s47. In this case, the background model update unit 38 executes step s48 when the count value is larger than the eighth threshold value, and executes step s41 when the count value is equal to or smaller than the eighth threshold value.

  As the background model update unit 38 operates as described above, the pixel values of the input pixels that are tentatively determined to be foreground pixels by the determination unit 37 are approximately the same for a plurality of frames (constant period) continuously. In this case, the background information (background pixel value) corresponding to the input pixel in the background model 550 is updated based on the pixel value. Hereinafter, such background information update may be referred to as “continuous background update”.

  In the present modification, the foreground identification unit 34 identifies the foreground image based on the second frequency map 420 and the determination result by the determination unit 37. Hereinafter, the operation of the foreground identification unit 34 according to this modification will be described in detail.

  In step s38 of FIG. 10, the foreground specifying unit 34 sets a region in which the second frequency value 421 is larger than “0” in the second frequency map 420 as described above, in the same manner as described above. Then, the foreground identification unit 34 sets a region corresponding to the foreground corresponding region in the second frequency map 420 in the processing target input frame image 111 as the first foreground candidate region. In addition, the foreground specifying unit 34 sets, in the input frame image 111 to be processed, an area composed of a plurality of input pixels tentatively determined to be foreground pixels by the determination unit 37 as a second foreground candidate region. . Then, the foreground specifying unit 34 specifies, as the foreground image, an area that is the first foreground candidate area and the second foreground candidate area in the input frame image 111 to be processed.

  As described above, the foreground identification unit 34 is based not only on the second frequency map 420 updated based on the result of the first provisional determination process but also on the result of the second provisional determination process different from the first provisional determination process. Since the foreground image is specified, the foreground image can be specified more correctly.

  Here, as described above, the second frequency value 421 of the second frequency map 420 decreases by the second decrease amount. Therefore, in the second frequency map 420, an area where there is no longer a person in the imaging region. The corresponding second frequency value 421 does not immediately become zero. Therefore, when a certain part (such as a hand or head) of a stationary person is moving, not only the second frequency value 421 corresponding to the current position of the certain part but also the past position of the certain part. The second frequency value 421 may also be larger than “0”. Therefore, in the input frame image 111 to be processed, the region corresponding to the foreground corresponding region whose second frequency value 421 is larger than “0” in the second frequency map 420 is specified as the foreground image. There is a possibility that the range of the foreground image is slightly deviated from the current range in the shooting area of a stationary person.

  On the other hand, in the continuous background update, the background information corresponding to the input pixel in the background model 550 is updated only when the pixel value of the input pixel shows the same value continuously for a certain period. When a certain part of a stationary person moves, the background model 550 is not easily updated according to the movement of the certain part. Therefore, the determination unit 37 determines that the input pixel corresponding to the current position of the certain part is the foreground pixel in the input frame image 111 to be processed even when a certain part of the stationary person moves. While being able to determine, the input pixel according to the past position of the said part can be determined to be a background pixel.

  In the present modification, the foreground identification unit 34 identifies the foreground image based on the determination result of the determination unit 37 and the second frequency map 420, so that the range of the identified foreground image and the still image are static. It is possible to reduce the deviation from the current range in the shooting area for the person who is. Therefore, it is possible to more correctly specify the region where there is a person who is stationary in the shooting range.

  In the configuration shown in FIG. 3 described above, the determination unit 37 may be used instead of the determination unit 31, and the background model 550 may be used instead of the appearance pixel value information 510 and the background model 500. In this case, the first map update unit 41 updates the first frequency map 410 based on the determination result in the determination unit 37.

<Adjustment of learning rate>
In this modification, values α1 and β1 when no copy occurs and values α2 and β2 when a copy occurs are determined as the learning rates α and β when updating the mixed normal distribution 511. α1, β1, α2, and β2 are respectively represented by the following formulas (6) to (9), for example.

  As shown in Expression (6), the value α1 when no copy occurs is a value obtained by adding a constant A to the reference value REF1. Further, as shown in Expression (7), the value β1 when no copy occurs is a value obtained by adding a constant B to the reference value REF1. The reference value REF1 is a fixed value and is commonly used for α1 and β1.

  On the other hand, as shown in Expression (8), the value α2 at the time of occurrence of copying is a value obtained by adding a constant A to the reference value REF2. Further, as shown in Expression (9), the value β2 at the time of occurrence of copying is a value obtained by adding a constant B to the reference value REF2. The reference value REF2 is a fixed value and is commonly used for α2 and β2. Since RFE1> REF2, α2 is smaller than α1, and β2 is smaller than β1.

  The background model update unit 32 performs a learning rate determination process that determines the values of the learning rates α and β. The learning rate determination process is performed between step s37 and step s38 in the flowchart shown in FIG. The learning rate determination process will be described in detail below.

  The background model update unit 32 generates a mixed normal distribution 511 corresponding to the input pixel in the input frame image 111 corresponding to the first frequency value 411 copied by the second map update unit 42 to the second frequency map 420 in step s37. The learning rates α and β at the time of updating are set to the values α2 and β2 at the time of copying, respectively. The background model update unit 32 performs this process for each first frequency value 411 copied by the second map update unit 42 to the second frequency map 420 in step s37.

  On the other hand, the background model update unit 32 responds to the input pixel in the input frame image 111 corresponding to the first frequency value 411 that the second map update unit 42 did not copy to the second frequency map 420 in step s37. The learning rates α and β when updating the mixed normal distribution 511 are set to values α1 and β1 when no copy occurs. The background model update unit 32 performs this process for each first frequency value 411 that the second map update unit 42 did not copy to the second frequency map 420 in step s37.

  Thus, in this modification, the background model update unit 32 updates the learning rate α when updating the mixed normal distribution 511 corresponding to the input pixel corresponding to the first frequency value 411 copied to the second frequency map 420. , Β is made small (decreased). This makes it difficult to update the mixed normal distribution 511 corresponding to the input pixel corresponding to the first frequency value 411 copied to the second frequency map 420. Since the input pixel corresponding to the first frequency value 411 copied to the second frequency map 420 is likely to be a pixel of a stationary person, the mixed normal distribution 511 corresponding to the input pixel is updated. By becoming difficult, the determination unit 31 takes a long time until a pixel of a stationary person is not specified as a foreground pixel. Therefore, the accuracy of specifying the foreground image is further improved.

  As described above, the image processing system 1 and the image processing apparatus 3 have been described in detail. However, the above description is illustrative in all aspects, and the present invention is not limited thereto. The various modifications described above can be applied in combination as long as they do not contradict each other. And it is understood that the countless modification which is not illustrated can be assumed without deviating from the scope of the present invention.

DESCRIPTION OF SYMBOLS 3 Image processing apparatus 6 Control program 31, 37 Judgment part 32, 38 Background model update part 33 Fine movement specific part 34 Foreground specific part 41 1st map update part 42 2nd map update part 410 1st map (1st frequency map)
420 Second map (second frequency map)
500,550 Background model

Claims (15)

An image processing apparatus that identifies an image of a moving object to be detected in an input moving image as a foreground image,
A first determination unit that performs a first determination process that tentatively determines whether a pixel included in a frame image of the input moving image is a foreground pixel or a background pixel based on a first background model;
A first map update unit for performing a first map update process for updating a first map in which a plurality of first values respectively corresponding to a plurality of pixels included in the frame image are arranged according to the positions of the plurality of pixels; ,
A second map update unit for performing a second map update process for updating a second map in which a plurality of second values respectively corresponding to a plurality of pixels included in the frame image are arranged according to the positions of the plurality of pixels; ,
A first background model update unit for performing a first background model update process for updating the first background model based on the frame image;
A fine movement specifying unit that performs a first specifying process for specifying a fine movement generation area in which a fine movement of the detection target moving body is generated in a shooting area that is reflected in the input moving image;
In the first map update process, the first map update unit includes:
Increasing the first value in the first map corresponding to the pixel determined to be the foreground pixel in the first determination process included in the frame image by a first predetermined value;
The first value in the first map corresponding to the pixel determined to be the background pixel in the first determination process included in the frame image is decreased by a second predetermined value,
In the second map update process, the second map update unit includes:
When the fine movement occurrence area is specified in the first specifying process, the first value in the fine movement corresponding area corresponding to the specified fine movement generation area in the first map is set in the second map. Performing a copy process adopted as the second value corresponding to the first value;
In the second map, the second value other than the second value at which the copy process is executed is decreased by a third predetermined value,
The image processing apparatus repeatedly executes the first determination process, the first map update process, the second map update process, the first background model update process, and the first specifying process,
An image processing apparatus, further comprising: a foreground specifying unit that specifies the foreground image based on the second map on which the second update process has been performed. The image processing apparatus according to claim 1,
The image processing apparatus, wherein the third predetermined value is larger than the second predetermined value. An image processing apparatus according to any one of claims 1 and 2,
The second map update unit identifies a plurality of detection target corresponding regions independent of each other in the first map, corresponding to a region where the detection target moving object is likely to exist in the imaging region,
The second map updating unit employs the first value in the detection target corresponding area including the fine movement corresponding area as the second value corresponding to the first value in the second map. , Image processing device. The image processing apparatus according to claim 3,
The second map updating unit includes the first value greater than or equal to a first threshold value or greater than the first threshold value among the first values in the detection target corresponding region including the fine motion corresponding region. Is adopted as the second value corresponding to the first value in the second map. An image processing apparatus according to any one of claims 3 and 4,
When the area of the detection target corresponding area including a plurality of fine movement corresponding areas independent from each other is greater than or equal to a second threshold value or larger than the second threshold value, the second map updating unit A circumscribed rectangle of the corresponding area is obtained, and among the first values in the detection target corresponding area, the obtained first value in the circumscribed rectangle corresponds to the first value in the second map. An image processing apparatus employed as the second value. The image processing apparatus according to claim 5,
The second map update unit is configured to detect the detection target when the area of the detection target corresponding region including a plurality of fine motion corresponding regions independent from each other is greater than or equal to the second threshold value or larger than the second threshold value. A fine movement grouping process in which a plurality of fine movement corresponding areas that are close to each other is made into one fine movement group is performed on the corresponding area,
The second map update unit obtains circumscribed rectangles of a plurality of fine movement corresponding areas constituting the fine movement group for each fine movement group obtained by the fine movement grouping process for the detection target corresponding area, and calculates each circumscribed rectangle. For the first value in the detection target corresponding region, the first value in the circumscribed rectangle is employed as the second value corresponding to the first value in the second map. , Image processing device. An image processing apparatus according to any one of claims 1 to 6,
The fine movement specifying unit
Generating a difference image between the first and second frame images of the input moving image;
A plurality of independent moving object regions in the difference image are specified as a moving object region group,
A first grouping process is performed on the moving object region group, with a plurality of moving object regions that are close to each other as one first group,
In the first grouping process, when the isolated moving object region in which no moving object region exists nearby exists in the moving object region group, the fine moving specifying unit sets the isolated moving object region as one first group,
For each of the plurality of first groups obtained as a result of the first grouping process on the difference image, the first and second conditions are satisfied among the first inclusion regions including the moving object regions constituting the first group. Performing a second specifying process for specifying the first inclusion region;
The first condition is a condition that an area of the first inclusion region is a third threshold value or less or less than the third threshold value,
The second condition is a condition that there is no first inclusion area that does not satisfy the first condition within a predetermined distance from the first inclusion area.
The fine movement specifying unit is an image processing apparatus in which a corresponding area corresponding to the first inclusion area specified in the second specifying process is set as the fine movement generation area in the photographing area. The image processing apparatus according to claim 7,
In the first grouping process, the fine movement specifying unit repeatedly executes a second grouping process for repeatedly performing a group determination process for determining one second group for a processing target;
In the first group determination process in the first second grouping process, the fine movement specifying unit sets a plurality of moving body areas constituting the moving body area group as the processing target.
In the group determination process, the fine movement specifying unit
In the processing target, when there is another moving body area whose distance from the reference moving body area is equal to or smaller than the fourth threshold value or less than the fourth threshold value, the reference moving body area and the other moving body object A region is determined as one second group,
When the other moving object region does not exist in the processing target, the moving object region serving as the reference is determined as one second group,
In the second grouping process, the fine movement specifying unit is determined by the most recent group determination process from the process target in the most recent group determination process when performing the second or subsequent group determination process. The remaining portion excluding the second group is set as a new processing target, and the group determination process is executed.
In the second grouping process, the fine movement specifying unit repeatedly executes the group determination process until the second group cannot be determined, and then executes the second grouping process.
When executing the second grouping process after the second time, the fine movement specifying unit configures the second group for each of the second groups determined in the most recent second grouping process. The second inclusion area including the moving object area is set as a new moving object area, the new moving object area is set as the processing target, and the first group determination process of the second grouping process is executed.
The fine movement specifying unit may perform the first until the number of the second groups determined in the second grouping process does not decrease below the number of the second groups determined in the preceding second grouping process. When the two grouping process is repeatedly executed, the first grouping process is terminated,
The fine movement specifying unit is an image processing apparatus in which a moving body area constituting the second group for each of the second groups finally obtained in the first grouping process is the first inclusion area. An image processing apparatus according to any one of claims 1 to 8,
A second determination unit that performs a second determination process that tentatively determines whether a pixel included in the frame image of the input moving image is a foreground pixel or a background pixel based on a second background model different from the first background model When,
A second background model update unit that updates the second background model based on the frame image in a different manner from the first background model update unit;
The said foreground specific part is an image processing apparatus which specifies the said foreground image based on the said 2nd map and the result of the said 2nd determination process. An image processing apparatus according to any one of claims 1 to 9, wherein
The first background model update unit updates background information corresponding to the certain pixel included in the first background model according to the frequency with which a similar pixel value appears in a certain pixel in the frame image. , Image processing device. The image processing apparatus according to claim 10,
The first background model is a background model based on a mixed normal distribution obtained by modeling pixel values of the past frame image,
In the first background model update process, the first background model update unit updates the mixed normal distribution corresponding to the certain pixel based on a pixel value of the certain pixel of the frame image, and the mixed mixture after the update An image processing apparatus that determines a normal distribution to be used as the background information corresponding to a certain pixel from a plurality of normal distributions included in the normal distribution. The image processing apparatus according to claim 11,
The first background model update unit applies pixels corresponding to the first value in the fine motion corresponding area of the first map in the frame image when the second map update unit executes the copy process. An image processing apparatus that reduces a learning rate when the corresponding mixed normal distribution is updated. An image processing apparatus according to any one of claims 9 to 12,
The second background model update unit updates the second background model based on the pixel value when the pixel value of the pixel included in the frame image continuously shows the same value for a certain period of time. Processing equipment. A control program for causing a computer to specify an image of a moving object to be detected in an input moving image as a foreground image,
In the computer,
(A) tentatively determining whether a pixel included in the frame image of the input moving image is a foreground pixel or a background pixel based on a background model;
(B) updating a first map in which a plurality of first values respectively corresponding to a plurality of pixels included in the frame image are arranged according to the positions of the plurality of pixels;
(C) updating a second map in which a plurality of second values respectively corresponding to a plurality of pixels included in the frame image are arranged according to the positions of the plurality of pixels;
(D) updating the background model based on the frame image;
(E) performing a specifying process for specifying a fine movement occurrence area in which a fine movement of the detection target moving body is generated in a shooting area reflected in the input moving image;
In the step (b),
(B-1) Increasing the first value in the first map corresponding to the pixel determined to be the foreground pixel by the determination process included in the frame image by a first predetermined value. Process,
(B-2) Decrease the first value in the first map corresponding to the pixel determined to be the background pixel in the determination process included in the frame image by a second predetermined value. Process and
In the step (c),
(C-1) When the fine movement occurrence area is specified by the specifying process, the first value in the fine movement corresponding area corresponding to the specified fine movement generation area in the first map is set to the second value. Executing a copy process employed as the second value corresponding to the first value in the map;
(C-2) In the second map, executing a step of reducing the second value other than the second value for which the copy process is executed by a third predetermined value;
The steps (a) to (e) are repeatedly executed,
(F) A control program for further executing a step of specifying the foreground image based on the second map updated in the step (c). A foreground image specifying method in an image processing apparatus,
(A) tentatively determining whether a pixel included in the frame image of the input moving image is a foreground pixel or a background pixel based on a background model;
(B) updating a first map in which a plurality of first values respectively corresponding to a plurality of pixels included in the frame image are arranged according to the positions of the plurality of pixels;
(C) updating a second map in which a plurality of second values respectively corresponding to a plurality of pixels included in the frame image are arranged according to the positions of the plurality of pixels;
(D) updating the background model based on the frame image;
(E) performing a specifying process for specifying a fine movement occurrence area in which a fine movement of the detection target moving body is generated in a shooting area reflected in the input moving image;
The step (b)
(B-1) Increasing the first value in the first map corresponding to the pixel determined to be the foreground pixel by the determination process included in the frame image by a first predetermined value. Process,
(B-2) Decrease the first value in the first map corresponding to the pixel determined to be the background pixel in the determination process included in the frame image by a second predetermined value. A process,
The step (c)
(C-1) When the fine movement occurrence area is specified by the specifying process, the first value in the fine movement corresponding area corresponding to the specified fine movement generation area in the first map is set to the second value. Executing a copy process employed as the second value corresponding to the first value in the map;
(C-2) in the second map, a step of reducing the second value other than the second value at which the copy process is executed by a third predetermined value;
The steps (a) to (e) are repeatedly executed,
(F) A foreground image specifying method further comprising the step of specifying the foreground image based on the second map updated in the step (c).

Images