JP2018025914A - Detection device for individual in shot image - Google Patents

Abstract

PROBLEM TO BE SOLVED: To detect respective individuals with an individual in action discriminated from an individual in standstill on the basis of shot images of a fixed-point camera, and display the individuals with the individual in action discriminated from the individual in standstill.SOLUTION: A raw image input unit 110 is configured to sequentially input a shot raw image P(t) as an image of a frame unit, and a background difference image preparation unit 130 is configured to prepare a background difference image D(t) as a difference between the raw image P(t) and a background image BG. An inter-frame difference image preparation unit 140 is configured to prepare an inter-frame difference image F(t) as a difference between two raw images P(t) different in a time t. An overlapping presence or absence determination unit 150 is configured to determine presence or absence of an overlapping part with respect to each difference area within the inter-frame difference image F(t) as to respective difference areas in the background difference image D(t), and an individual recognition unit 160 is configured to recognize each individual as a standstill individual and active individual on the basis of the determination result. A detection individual display unit 170 is configured to display a standstill individual index or active individual index with the standstill individual index or active individual index overlapped on the raw image P(t).SELECTED DRAWING: Figure 1

Description

  The present invention relates to an apparatus for detecting an individual in a photographed image, and in particular, an individual that detects an individual existing in a predetermined photographing target region as a stationary individual or a moving individual based on the photographed image. The present invention relates to a detection device.

  An apparatus for detecting an object such as a person or a vehicle by analyzing a captured image of a surveillance camera or the like installed at a fixed point is widely used in various technical fields. When detecting an object such as a person or a vehicle that has entered a predetermined shooting target area, an area that normally forms a difference by taking a difference between a background image prepared in advance and an input image shot at the present time In addition, a method of recognizing that an object such as a person or a vehicle exists is employed. Various methods for recognizing the movement of an object on frame-by-frame captured images arranged in time series have been proposed, and a technique for tracking a moving individual such as a person or a vehicle has been put into practical use.

  For example, in Patent Document 1 below, a first difference process for obtaining a difference between an input image and a background image and a second difference process for obtaining a difference between two input images input at different times are performed. Discloses a moving object tracking device capable of tracking a moving object. Patent Document 2 discloses a first basic image that is taken with the elevator door closed in the absence of passengers, a second basic image that is taken immediately after the door is closed, and the current shooting. There is disclosed an elevator car monitoring apparatus having a function of determining whether a stationary object is present in a car and whether a dynamic object is present by obtaining a difference from the image. Further, Patent Document 3 has a function of detecting a moving object based on a difference between two input images input at different times, and detecting a pedestrian based on a preset pedestrian's stride and a walking cycle. A pedestrian detection apparatus having a pedestrian is disclosed.

Japanese Patent No. 5154461 Japanese Patent No. 4705862 Japanese Patent No. 5626099

  An image captured by a fixed-point camera includes people, vehicles, and the like as subjects, and it is important to recognize these subjects as independent individuals. In particular, it is necessary for a person in charge who performs a monitoring job to not only visually recognize each individual but also grasp whether each individual is performing some kind of operation or whether it is stationary. For example, for an observer who monitors the behavior of a person or vehicle while looking at the image taken by a fixed-point camera, not only recognizes each individual person or vehicle as an individual, but also whether the individual is in operation. It is important to distinguish whether it is stationary or not.

  Specifically, if it is possible to identify a person who has been stationary for a long time with respect to the images taken by a fixed-point camera installed in a crowded area, be aware of the possibility of a suspicious person or a victim. You can watch. In addition, if it is possible to distinguish between a parked vehicle and a moving vehicle for a captured image of a fixed point camera installed in the parking lot, it is possible to grasp the congestion situation, which can be useful for vehicle guidance. it can.

  As described above, various conventionally proposed devices can recognize an individual such as a person or a vehicle by analyzing a captured image, and can track a moving object. is there. However, it is impossible to distinguish and recognize whether each individual is a moving individual or a stationary individual, or to distinguish and display them on a display image. Patent Document 2 described above discloses a technique for determining whether an individual existing in an elevator car is a stationary object or a dynamic object. However, the technique is applied only in a limited place in an elevator car. It is a possible technology and cannot be used for general monitoring systems.

  Therefore, the present invention has an object to provide an individual detection device in a captured image that can detect an individual individual based on a captured image obtained by a fixed-point camera and distinguishing whether it is operating or stationary. It is another object of the present invention to provide a device for detecting an individual in a captured image that can distinguish and display a moving individual and a stationary individual.

(1) According to a first aspect of the present invention, in an apparatus for detecting an individual in a captured image that detects an individual present in a predetermined imaging target region as a stationary individual that is stationary or an operating individual that is operating,
An original image input unit for inputting each original image obtained as an image in frame units arranged in time series by photographing with respect to a predetermined photographing target area;
A background image storage unit that stores a background image of the imaging target area;
For each original image input by the original image input unit, a background difference image creation unit that creates a background difference image including a background difference region indicating a difference from the background image;
For each original image input by the original image input unit, an inter-frame difference image creating unit that creates an inter-frame difference image including an inter-frame difference region indicating a difference from different frames in time series, and
For each background difference area in the background difference image, an overlap presence / absence determination unit that determines the presence / absence of an overlapping portion with respect to the inter-frame difference area in the inter-frame difference image;
Based on the determination result by the duplication presence / absence determination unit, an individual recognition unit that recognizes an individual included in the original image as a stationary individual or a moving individual,
Is provided.

(2) According to a second aspect of the present invention, in the device for detecting an individual in the captured image according to the first aspect described above,
The individual recognition unit recognizes an individual occupying the background difference region, which is obtained as a result of the determination that there is no substantial overlap with the inter-frame difference region, by the duplication presence / absence determination unit as a stationary individual. It is what you have.

(3) According to a third aspect of the present invention, in the individual detection device in the captured image according to the second aspect described above,
A detected individual display unit for displaying a stationary individual index indicating an individual recognized as a stationary individual by the stationary individual recognition unit is further provided.

(4) According to a fourth aspect of the present invention, in the device for detecting an individual in the captured image according to the third aspect described above,
The detected individual display unit creates a stationary individual index indicating a stationary individual based on the background difference area occupied by the stationary individual.

(5) According to a fifth aspect of the present invention, in the individual detection device in the captured image according to the first aspect described above,
The individual recognition unit recognizes an individual that occupies a background difference region from which the determination result that at least a part of the region has a substantial overlap portion with respect to the inter-frame difference region is obtained by the overlap presence / absence determination unit as an operation individual It is made to have an operation individual recognition part which performs.

(6) According to a sixth aspect of the present invention, in the individual detection device in the captured image according to the fifth aspect described above,
The apparatus further includes a detected individual display unit that displays an operating individual index indicating an individual recognized as an operating individual by the operating individual recognition unit.

(7) According to a seventh aspect of the present invention, in the device for detecting an individual in the captured image according to the sixth aspect described above,
The detected individual display unit creates an operating individual index indicating the operating individual based on the background difference area occupied by the operating individual.

(8) According to an eighth aspect of the present invention, in the device for detecting an individual in the captured image according to the sixth aspect described above,
The detected individual display unit creates the motion individual index indicating the motion individual by correcting the inter-frame difference region occupied by the motion individual based on the background difference region occupied by the motion individual. is there.

(9) According to a ninth aspect of the present invention, in the individual detection device in the captured image according to the eighth aspect described above,
The detected individual display unit performs correction by performing a logical OR operation on the figures constituting the area for the inter-frame difference area occupied by the moving object and the background difference area occupied by the moving object. .

(10) According to a tenth aspect of the present invention, in the device for detecting an individual in the captured image according to the first aspect described above,
The individual recognition unit
A stationary individual recognizing unit that recognizes an individual occupying a background differential region obtained as a result of determination that there is no substantial overlapping portion with respect to the inter-frame difference region by the duplication presence / absence determining unit,
The motion individual recognition unit that recognizes an individual occupying the background difference region from which the determination result that at least a part of the region has a substantial overlap with the inter-frame difference region is obtained by the overlap presence / absence determination unit as a motion individual When,
It is made to have.

(11) According to an eleventh aspect of the present invention, in the device for detecting an individual in the captured image according to the tenth aspect described above,
A detected individual display unit for displaying a stationary individual index indicating an individual recognized as a stationary individual by a stationary individual recognition unit and a moving individual index indicating an individual recognized as a moving individual in a visually distinguishable manner. Furthermore, it is made to have.

(12) According to a twelfth aspect of the present invention, in the device for detecting an individual in the captured image according to the eleventh aspect described above,
The detected individual display unit creates a stationary individual index indicating a stationary individual based on the background difference area occupied by the stationary individual, and creates an operating individual index indicating the moving individual based on the background difference area occupied by the moving individual. It is what you do.

(13) According to a thirteenth aspect of the present invention, in the device for detecting an individual in the captured image according to the eleventh aspect described above,
The detected individual display unit creates a stationary individual index indicating a stationary individual based on the background difference area occupied by the stationary individual, and an operating individual index indicating the moving individual is calculated with respect to the inter-frame difference area occupied by the moving individual. This is created by performing correction based on the background difference area occupied by the moving individual.

(14) According to a fourteenth aspect of the present invention, in the individual detection apparatus in the captured image according to the thirteenth aspect described above,
The detected individual display unit performs correction by performing a logical OR operation on the figures constituting the area for the inter-frame difference area occupied by the moving object and the background difference area occupied by the moving object. .

(15) According to a fifteenth aspect of the present invention, in the device for detecting an individual in a captured image according to the third, fourth, and sixth to ninth aspects described above,
The detected individual display unit creates an individual detected image by superimposing a stationary individual index or a moving individual index on the original image, and displays the individual detected image on the display screen.

(16) According to a sixteenth aspect of the present invention, in the device for detecting an individual in the captured image according to the fifteenth aspect described above,
The detected individual display unit is arranged in the vicinity of the area occupied by the target individual with the stationary individual index or the moving individual index, and the inside is painted with a predetermined color or the closed area index in which the predetermined pattern is formed The individual detection image is created by superposing the closed region index on the original image.

(17) According to a seventeenth aspect of the present invention, in the device for detecting an individual in the captured image according to the sixteenth aspect described above,
When the closed area index is composed of a closed area that is painted in a semi-transparent color or has a partially spaced pattern, and the closed area index is superimposed on the original image, the original image is It is designed to show through through.

(18) According to an eighteenth aspect of the present invention, in the individual detection device in the captured image according to the fifteenth aspect described above,
The detected individual display unit is configured with a stationary individual index or a moving individual index composed of a frame-shaped index surrounding the target individual, and an individual detection image is created by superimposing the frame-shaped index on the original image. is there.

(19) According to a nineteenth aspect of the present invention, in the device for detecting an individual in the captured image according to the fifteenth aspect described above,
The detected individual display unit is composed of a stationary individual index or a moving individual index with a contour index indicating the contour line of the target individual, and an individual detection image is created by superimposing the contour index on the original image It is.

(20) According to a twentieth aspect of the present invention, in the individual detection device in the captured image according to the fifteenth aspect described above,
The detected individual display unit comprises a stationary individual index or a moving individual index by a mask image in which an aperture window is arranged in a target individual part and a part other than the aperture window is covered, and the mask image is superimposed on the original image Thus, an individual detection image is created.

(21) According to a twenty-first aspect of the present invention, in the individual detection device in the captured image according to the above-described eleventh to fourteenth aspects,
The detected individual display unit creates an individual detected image by superimposing a stationary individual index and a moving individual index on the original image, and displays the individual detected image on the display screen.

(22) According to a twenty-second aspect of the present invention, in the device for detecting an individual in the captured image according to the twenty-first aspect described above,
The detected individual display unit is configured by a first closed region index in which a stationary individual index is arranged in the vicinity of a region occupied by the target individual and the inside is painted with a predetermined color or a predetermined pattern is formed inside Then, the motion individual index is arranged in the vicinity of the area occupied by the target individual and is constituted by a second closed area index in which the inside is painted with a predetermined color or a predetermined pattern is formed inside, The closed region index and the second closed region index are closed regions in which different colors are applied or different patterns are formed, and the first closed region index and the second closed region index are formed on the original image. An individual detection image is created by superimposing region indexes.

(23) According to a twenty-third aspect of the present invention, in the individual detection device in the captured image according to the twenty-second aspect described above,
The first closed area index and the second closed area index are configured by closed areas that are painted with a semi-transparent color or partially formed with a gap pattern, and each closed area index is superimposed on the original image In this case, the original image is displayed through each closed region index.

(24) According to a twenty-fourth aspect of the present invention, in the device for detecting an individual in the captured image according to the twenty-first aspect described above,
The detected individual display unit configures the stationary individual index by a first frame-shaped index surrounding the target stationary individual, and configures the moving individual index by the second frame-shaped index surrounding the target moving individual, The first frame-shaped index and the second frame-shaped index are constituted by frame lines different from each other in at least one of line type, color, shape, and thickness, and the first frame-shaped index is displayed on the original image. The individual detection image is created by superimposing the index and the second frame-shaped index.

(25) According to a twenty-fifth aspect of the present invention, in the device for detecting an individual in the captured image according to the twenty-first aspect described above,
The detected individual display unit configures the stationary individual index by the first contour index indicating the contour line of the target stationary individual, and the moving individual index indicates the second contour index indicating the contour line of the target moving individual. The first contour index and the second contour index are configured by contour lines having at least one of line type, color, and thickness different from each other, and the first contour index is included in the original image. An individual detection image is created by superimposing the second contour index.

(26) According to a twenty-sixth aspect of the present invention, in the individual detection device in the captured image according to the above-described twenty-first to twenty-fifth aspects,
The detected individual display section
An original image display mode for displaying the original image on the display screen;
An all-individual display mode for displaying an individual detection image in which a stationary individual index, a motion individual index, and an original image are superimposed on a display screen;
A stationary individual display mode for displaying an individual detection image in which a stationary individual index and an original image are superimposed on a display screen;
An action individual display mode for displaying an individual detection image in which an action individual indicator and an original image are superimposed on a display screen;
These four display modes have a display function, and display is performed by switching to an arbitrary display mode.

(27) According to a twenty-seventh aspect of the present invention, in the individual detection device in the captured image according to the first to twenty-sixth aspects described above,
The background image storage unit has a function of creating and storing a background image based on a plurality of frame unit images input in the past by the original image input unit,
The background difference image creation unit creates a background difference image using the latest background image stored in the background image storage unit.

(28) According to a twenty-eighth aspect of the present invention, in the individual detection device in the captured image according to the first to twenty-seventh aspects described above,
The original image input unit samples and inputs images in frame units arranged in time series given from the outside at a predetermined sampling period,
The inter-frame difference image creation unit creates an inter-frame difference image by performing a difference operation on the sampled image in units of frames with the immediately preceding image in time series.

(29) According to a twenty-ninth aspect of the present invention, in the individual detection device in the captured image according to the first to twenty-eighth aspects described above,
The original image input unit inputs a color image composed of a set of pixels having pixel values of the three primary colors RGB as an original image, and converts the color image into a luminance image composed of a set of pixels having a pixel value indicating luminance Y. Process to convert,
The background image storage unit stores a background image composed of a collection of pixels having pixel values indicating luminance Y,
A background difference image creation unit binarizes a difference image composed of a collection of pixels having pixel values indicating an absolute value ΔY of a difference in luminance value between the luminance image and the background image using a predetermined threshold value. To create a background difference image,
The inter-frame difference image creation unit binarizes a difference image composed of a set of pixels having pixel values indicating an absolute value ΔY of a difference in luminance value between two frame-unit images using a predetermined threshold value. By doing so, an inter-frame difference image is created.

(30) According to a thirtieth aspect of the present invention, in the individual detection device in the captured image according to the twenty-ninth aspect described above,
The background difference image creation unit gives a pixel value “1” for a pixel whose absolute value ΔY of the luminance value difference is equal to or greater than the threshold value, and a pixel value “1” for a pixel whose absolute value ΔY of the luminance value difference is less than the threshold value. By giving “0”, a background difference image including a continuous area of pixels having a pixel value “1” as a background difference area is created,
The inter-frame difference image creation unit gives a pixel value “1” for a pixel for which the absolute value ΔY of the luminance value difference is greater than or equal to the threshold value, and a pixel value for the pixel for which the absolute value ΔY of the luminance value difference is less than the threshold value By giving “0”, an inter-frame difference image including a continuous area of pixels having a pixel value “1” as an inter-frame difference area is created.

(31) According to a thirty-first aspect of the present invention, in the device for detecting an individual in the captured image according to the first to thirtieth aspects described above,
Based on the background difference image and the inter-frame difference image, the duplication presence / absence determining unit creates an overlap confirmation image including an overlapping partial region indicating an overlapping portion of the background difference region and the inter-frame difference region,
When the individual recognition unit refers to a corresponding region on the overlap confirmation image for a specific background difference region included in the background difference image, and the overlapping partial region is not substantially included in the corresponding region When the individual occupying the specific background difference area is recognized as a stationary individual, and the overlapping partial area is substantially included in the corresponding area, the individual occupying the specific background difference area is defined as an operating individual. It is intended to be recognized.

(32) According to a thirty-second aspect of the present invention, in the device for detecting an individual in the captured image according to the thirty-first aspect described above,
When the area of the overlapping partial area included in the corresponding area does not satisfy a predetermined standard, the individual recognition unit determines that the overlapping partial area is not included in the corresponding area. is there.

(33) According to a thirty-third aspect of the present invention, in the device for detecting an individual in the captured image according to the thirty-second aspect described above,
The background difference image creation unit is a binary image composed of a collection of pixels having a pixel value “1” or “0”, and represents the background difference region as a continuous region of pixels having the pixel value “1”. Created background difference image,
The inter-frame difference image creation unit is a binary image composed of a collection of pixels having a pixel value “1” or “0”, and the inter-frame difference area is a continuous area of pixels having the pixel value “1”. Create an inter-frame difference image that represents
The duplication presence / absence determination unit performs a logical AND operation on the corresponding pixels with respect to the background difference image and the inter-frame difference image, thereby forming a binary consisting of a collection of pixels having the pixel value “1” or “0”. Create a duplicate confirmation image that is an image,
The individual recognition unit refers to the corresponding region on the overlap confirmation image for a specific background difference region included in the background difference image, and the pixel having the pixel value “1” among the pixels in the corresponding region is referred to. When the total number does not satisfy the predetermined standard, the individual occupying the specific background difference area is recognized as a stationary individual, and the total number of pixels having the pixel value “1” among the pixels in the corresponding area is predetermined. If it is equal to or more than the reference, an individual occupying the specific background difference area is recognized as an action individual.

(34) According to a thirty-fourth aspect of the present invention, in the device for detecting an individual in the captured image according to the first to thirty-third aspects described above,
A background difference area including a pixel difference information that the background difference image creating unit assigns a background difference area code to each background difference area and identifies pixels included in the background difference area corresponding to each background difference area code Process to create labeling information for
The inter-frame difference image creating unit assigns an inter-frame difference area code for each inter-frame difference area, and specifies pixel specifying information for specifying a pixel included in the inter-frame difference area corresponding to each inter-frame difference area code. Including the process of creating the labeling information of the inter-frame difference area,
When the overlapping presence / absence determining unit determines the presence / absence of an overlapping portion for a predetermined target background difference region, each frame difference region is determined for each target pixel specified by the pixel specifying information in the labeling information of the target background difference region. It is determined whether or not there is a pixel at the same position in the pixel specifying information in the labeling information, and if it exists, it is determined that there is an overlapping portion.

(35) According to a thirty-fifth aspect of the present invention, in the device for detecting an individual in the captured image according to the first to thirtieth aspects described above,
When the overlap presence / absence determination unit determines the presence / absence of an overlapping portion with respect to the inter-frame difference area in the inter-frame difference image for the background difference area in the background difference image, the circumscribing rectangle of the background difference area and the circumscribing of the inter-frame difference area A simple determination method for determining the presence or absence of an overlapping portion with respect to a rectangle is performed.

(36) According to a thirty-sixth aspect of the present invention, in the individual detection device in a photographed image according to the thirty-fifth aspect,
The individual recognition unit recognizes the individual occupying the background difference area from which the determination result is obtained that there is no overlapping part about the circumscribed rectangle by the simple determination method performed by the duplication presence / absence determination unit as a stationary individual,
A stationary individual index indicating an individual recognized as a stationary individual is created based on a circumscribed rectangle of a background difference area occupied by the stationary individual, and further includes a detected individual display unit for displaying the rectangle.

(37) According to a thirty-seventh aspect of the present invention, in the individual detection device in the captured image according to the thirty-fifth aspect,
The individual recognition unit recognizes the individual occupying the background difference area from which the determination result is obtained that there is an overlapping part about the circumscribed rectangle by the simple determination method performed by the duplication presence / absence determination unit, as an operation individual,
A motion individual index indicating an individual recognized as a motion individual is created based on a circumscribed rectangle of a background difference area occupied by the motion individual, and further includes a detected individual display unit for displaying the rectangle.

(38) According to a thirty-eighth aspect of the present invention, in the individual detection device in the captured image according to the thirty-fifth aspect,
The individual recognition unit recognizes an individual that occupies the inter-frame difference area obtained as a result of determination that there is an overlapping part for the circumscribed rectangle by the simple determination method performed by the duplication presence / absence determination unit, as an operation individual,
The motion individual index indicating the individual recognized as the motion individual is created by performing correction based on the circumscribing rectangle of the background difference region occupied by the motion individual on the circumscribing rectangle of the inter-frame difference region occupied by the motion individual. Further, a detected individual display unit for displaying this is further provided.

(39) According to a thirty-ninth aspect of the present invention, in the individual detection device in a photographed image according to the thirty-eighth aspect described above,
The detected individual display unit obtains an inclusion rectangle that includes both circumscribed rectangles for the circumscribed rectangle of the inter-frame difference area occupied by the moving individual and the circumscribed rectangle of the background difference area occupied by the moving individual, and based on the obtained inclusion rectangle The movement individual index is created.

  (40) According to a 40th aspect of the present invention, the individual detection device in the photographed image according to the first to 39th aspects is configured by incorporating a program in a computer.

(41) According to a 41st aspect of the present invention, there is provided a monitoring system in which an individual detection device in a captured image according to the above 1st to 39th aspects is incorporated.
A fixed-point camera that acquires images in frame units arranged in time series by performing moving image shooting on a predetermined shooting target area;
A computer that inputs a frame unit image acquired by a fixed point camera as an original image and executes a predetermined process;
A display device for displaying an image based on instructions from a computer;
Composed by
The computer functions as the individual detection device by an operation based on an installed program.

(42) According to a forty-second aspect of the present invention, there is provided a monitoring system in which an individual detection device in a captured image according to the first to thirty-ninth aspects described above is incorporated.
Consists of a client system and a server system connected to each other by a network,
The client system
A fixed-point camera that acquires images in frame units arranged in time series by performing moving image shooting on a predetermined shooting target area;
A client computer that inputs a frame-unit image acquired by a fixed-point camera as an original image and executes predetermined processing;
A display device that displays an image based on an instruction from a client computer,
Server system
A server computer that executes predetermined processing while communicating with a client computer via a network;
The client computer performs a part of processing necessary as the individual detection device by an operation based on the installed client computer program,
The server computer is configured to execute other part of the processing necessary as the individual detection device by an operation based on the installed server computer program.

  (43) According to a forty-third aspect of the present invention, a client computer program or a server computer program in the monitoring system according to the forty-second aspect is provided independently.

(44) A forty-fourth aspect of the present invention provides the monitoring system according to the forty-first or forty-second aspect described above,
The computer or the client computer issues an alarm to the outside based on the recognition result of the stationary object or the moving object.

(45) A forty-fifth aspect of the present invention is the monitoring system according to the forty-first aspect described above,
An image storage device for input that stores images in frames arranged in time series obtained by performing moving image shooting on a predetermined shooting target area instead of or in addition to the fixed point camera. Provided,
A computer inputs a frame unit image stored in an input image storage device as an original image and executes a predetermined process.

(46) According to a 46th aspect of the present invention, in the monitoring system according to the 41st or 45th aspect described above,
In place of the display device or in addition to the display device, an image storage device for output for storing an image given from a computer is provided,
The computer outputs the image to be output to the display device to the output image storage device and stores it.

(47) According to a 47th aspect of the present invention, in an individual detection apparatus for detecting a stationary individual that is stationary among individuals existing in a predetermined imaging target region,
An original image input unit for inputting an original image obtained as an image in frame units arranged in time series by photographing with respect to a predetermined photographing target region;
An individual recognition unit that recognizes whether each individual appearing on the original image is a stationary individual or a moving individual;
Based on the recognition result by the individual recognition unit, a detected individual display unit that superimposes and displays a stationary individual index indicating an individual recognized as a stationary individual on the original image;
Is provided.

(48) According to a 48th aspect of the present invention, in an individual detection apparatus for detecting an operating individual that is operating among individuals existing in a predetermined imaging target region,
An original image input unit for inputting an original image obtained as an image in frame units arranged in time series by photographing with respect to a predetermined photographing target region;
An individual recognition unit that recognizes whether each individual appearing on the original image is a stationary individual or a moving individual;
Based on the recognition result by the individual recognition unit, a detected individual display unit that superimposes and displays an operation individual index indicating an individual recognized as an operation individual on the original image,
Is provided.

(49) According to a 49th aspect of the present invention, in the individual detection device in the photographed image according to the 47th or 48th aspect described above,
The detected individual display unit is arranged in the vicinity of the area occupied by the target individual with the stationary individual index or the moving individual index, and the inside is painted with a predetermined color or the closed area index in which the predetermined pattern is formed The closed area index is superimposed on the original image and displayed.

(50) According to a 50th aspect of the present invention, in the device for detecting an individual in the photographed image according to the 49th aspect described above,
When the closed area index is composed of a closed area that is painted in a semi-transparent color or has a partially spaced pattern, and the closed area index is superimposed on the original image, the original image is It is designed to show through through.

(51) According to a fifty-first aspect of the present invention, in the individual detection device in the photographed image according to the forty-seventh or forty-eighth aspect described above,
The detected individual display unit configures the stationary individual indicator or the moving individual indicator as a frame-like indicator that surrounds the target individual and has a predetermined line type, color, shape, and thickness, and superimposes the frame-like indicator on the original image It is made to display.

(52) According to a 52nd aspect of the present invention, in the individual detection device in the photographed image according to the 47th or 48th aspect described above,
The detected individual display unit shows the contour of the target individual as a stationary individual index or a moving individual index, and is composed of a contour index having a predetermined line type, color, and thickness, and superimposes the contour index on the original image. Are displayed.

(53) According to a fifty-third aspect of the present invention, in an individual detection apparatus for detecting an individual existing in a predetermined imaging target region as a stationary stationary individual or an operating individual moving.
An original image input unit for inputting an original image obtained as an image in frame units arranged in time series by photographing with respect to a predetermined photographing target region;
An individual recognition unit that recognizes individual individuals appearing on the original image as stationary individuals or moving individuals;
Based on the recognition result by the individual recognition unit, on the original image, a stationary individual index indicating an individual recognized as a stationary individual and a moving individual index indicating an individual recognized as a moving individual are superimposed, and A detected individual display unit that displays the stationary individual index and the moving individual index in such a manner as to be visually distinguishable from each other;
Is provided.

(54) According to a fifty-fourth aspect of the present invention, in the device for detecting an individual in the photographed image according to the fifty-third aspect described above,
The detected individual display unit is configured by a first closed region index in which a stationary individual index is arranged in the vicinity of a region occupied by the target individual and the inside is painted with a predetermined color or a predetermined pattern is formed inside Then, the motion individual index is arranged in the vicinity of the area occupied by the target individual and is constituted by a second closed area index in which the inside is painted with a predetermined color or a predetermined pattern is formed inside, The closed region index and the second closed region index are closed regions in which different colors are applied or different patterns are formed, and the first closed region index and the second closed region index are formed on the original image. The area index is superimposed and displayed.

(55) According to a 55th aspect of the present invention, in the individual detection device in a photographed image according to the above-mentioned 54th aspect,
The first closed area index and the second closed area index are configured by closed areas that are painted with a semi-transparent color or partially formed with a gap pattern, and each closed area index is superimposed on the original image In this case, the original image is displayed through each closed region index.

(56) According to a fifty-sixth aspect of the present invention, in the individual detection device in the photographed image according to the fifty-third aspect described above,
The detected individual display unit configures the stationary individual index by a first frame-shaped index surrounding the target stationary individual, and configures the moving individual index by the second frame-shaped index surrounding the target moving individual, The first frame-shaped index and the second frame-shaped index are constituted by frame lines different from each other in at least one of line type, color, shape, and thickness, and the first frame-shaped index is displayed on the original image. The index and the second frame-shaped index are superimposed and displayed.

(57) According to a 57th aspect of the present invention, in the individual detection apparatus in a photographed image according to the 53rd aspect described above,
The detected individual display unit configures the stationary individual index by the first contour index indicating the contour line of the target stationary individual, and the moving individual index indicates the second contour index indicating the contour line of the target moving individual. The first contour index and the second contour index are configured by contour lines having at least one of line type, color, and thickness different from each other, and the first contour index is included in the original image. In addition, the second contour index is superimposed and displayed.

  (58) According to a 58th aspect of the present invention, the individual detection apparatus in the photographed image according to the 47th to 57th aspects is configured by incorporating a program in a computer.

  According to the individual detection device in the photographed image according to the present invention, whether or not each individual is in operation is determined by determining whether or not there is an overlap of regions in the background difference image and the inter-frame difference image. Based on the image taken by the fixed point camera, it is possible to detect each individual by distinguishing whether it is operating or stationary. In addition, it is possible to distinguish between a moving individual and a stationary individual on the display screen.

It is a block diagram which shows the structure of the detection apparatus of the individual in the picked-up image which concerns on fundamental embodiment of this invention. FIG. 2 is a diagram illustrating a specific example of a correspondence relationship between an original image P (t) input to the apparatus of FIG. 1 and an individual detection image Q (t) -S output from the apparatus of FIG. FIG. 2 is a diagram illustrating a specific example of a correspondence relationship between an original image P (t) input to the apparatus of FIG. 1 and an individual detection image Q (t) -M output from the apparatus of FIG. FIG. 2 is a diagram illustrating a specific example of a correspondence relationship between an original image P (t) input to the apparatus of FIG. 1 and an individual detection image Q (t) -SM output from the apparatus of FIG. It is a figure which shows the specific example of the background difference image D (t) produced by the background difference image creation part 130 of the apparatus of FIG. It is a figure which shows the specific example of the inter-frame difference image F (t) produced by the inter-frame difference image creation part 140 of the apparatus of FIG. It is a figure which shows the specific example of the duplication confirmation image C (t1) produced by the duplication existence determination part 150 of the apparatus of FIG. 1 at the time of the time t1. It is a figure which shows the specific example of the duplication confirmation image C (t2) produced by the duplication existence determination part 150 of the apparatus of FIG. 1 at the time of the time t2. It is a figure which shows the specific example of the duplication confirmation image C (t3) produced at the time of the time t3 by the duplication presence determination part 150 of the apparatus of FIG. It is a figure which shows the specific example of the duplication confirmation image C (t4) produced by the duplication presence determination part 150 of the apparatus of FIG. 1 at the time of the time t4. It is a figure which shows the specific example of duplication confirmation image C (t1)-C (t4) produced at each time of the time t1-t4 by the duplication presence determination part 150 of the apparatus of FIG. It is a figure which shows the 1st modification of the individual detection image Q (t) -SM of a stationary individual and an operation | movement individual shown in FIG. It is a figure which shows the 2nd modification of the individual detection image Q (t) -SM of a stationary individual and a movement individual shown in FIG. It is a figure which shows the 3rd modification of the individual detection image Q (t) -SM of a stationary individual and a moving individual shown in FIG. It is a figure which shows the modification of individual detection image Q (t) -S of a stationary individual shown in FIG. It is a figure which shows the modification of the individual detection image Q (t) -M of an operation | movement individual | organism | solid shown in FIG. It is a figure which shows four kinds of display modes by the detected individual display part 170 of the apparatus of FIG. It is a figure which shows the general procedure of the process which produces the binarization difference image W about the two color images U and V, and produces labeling information about the difference area. It is a figure which shows the principle of the duplication determination process by the duplication presence determination part 150 of the apparatus of FIG. 2 is a flowchart showing a procedure for creating a stationary individual index by a cooperative operation of the duplication presence / absence determining unit 150, stationary individual recognition unit 161, and detected individual display unit 170 of the apparatus of FIG. It is a figure which shows the specific example of the process shown to the flowchart of FIG. 2 is a flowchart illustrating a procedure for creating an action individual index by a cooperative operation of the duplication presence / absence determination unit 150, the action individual recognition unit 162, and the detected object display unit 170 of the apparatus of FIG. It is a figure which shows the specific example of the process shown to the flowchart of FIG. 7 is a flowchart showing a modified example of a procedure for creating an action individual index by a cooperative operation of the duplication presence / absence determination unit 150, the action individual recognition unit 162, and the detected object display unit 170 of the apparatus of FIG. FIG. 25 is a first diagram illustrating a specific example of the process illustrated in the flowchart of FIG. 24. FIG. 25 is a second diagram illustrating a specific example of the process illustrated in the flowchart of FIG. 24. FIG. 25 is a diagram illustrating a specific example of an individual detection image Q (t) -SM displayed when the modification example illustrated in FIG. 24 is performed. It is a figure which shows the modification of individual detection image Q (t) -SM shown in FIG. It is a figure explaining the handling of the rectangular data in embodiment which introduced the simple determination method of this invention. 10 is a flowchart showing a procedure for creating a stationary individual index by a cooperative operation of the duplication presence / absence determination unit 150, the stationary individual recognition unit 161, and the detected individual display unit 170 in the embodiment in which the simple determination method of the present invention is introduced. It is a figure which shows the specific example of the process shown to the flowchart of FIG. 10 is a flowchart showing a procedure for creating an action individual index by a cooperative operation of the duplication presence / absence determination unit 150, the action individual recognition unit 162, and the detected object display unit 170 in the embodiment in which the simple determination method of the present invention is introduced. It is a figure which shows the specific example of the process shown to the flowchart of FIG. 10 is a flowchart showing a modified example of a procedure for creating an action individual index by a cooperative operation of the duplication presence / absence determination unit 150, the action individual recognition unit 162, and the detected object display unit 170 in the embodiment in which the simple determination method of the present invention is introduced. FIG. 35 is a first diagram showing a specific example of the process shown in the flowchart of FIG. 34. FIG. 35 is a second diagram showing a specific example of the process shown in the flowchart of FIG. 34. It is a block diagram which shows an example of the monitoring system using the individual detection apparatus which concerns on this invention.

  Hereinafter, the present invention will be described based on the illustrated embodiments.

<<< §1. Configuration of individual detection apparatus according to basic embodiment >>
First, the overall configuration of the individual detection apparatus according to the present invention will be described. FIG. 1 is a block diagram showing a configuration of an individual detection device 100 in a captured image according to a basic embodiment of the present invention. This detection apparatus 100 is an individual detection apparatus that detects an individual existing in a predetermined imaging target region as a stationary individual that is stationary or an operating individual that is operating, and as illustrated, an original image input unit 110, A background image storage unit 120, a background difference image creation unit 130, an inter-frame difference image creation unit 140, an overlap presence / absence determination unit 150, an individual recognition unit 160, and a detected individual display unit 170 are provided.

  The original image input unit 110 is a component that inputs each original image P (t) obtained as a frame-by-frame image arranged in time series. In the illustrated embodiment, each input original image P (t ) Is temporarily stored. Here, the variable t indicates the time. For example, the original image P (t0) taken at time t0, the original image P (t1) taken at time t1, and the original image P (taken at time t2 ( As in t2),..., original images in units of frames are sequentially input. Each original image P (t) is an image obtained by photographing a predetermined photographing target region, for example, a digital image obtained by photographing by a fixed point camera installed at a predetermined monitoring place.

  On the other hand, the background image storage unit 120 is a component that stores the background image BG for the shooting target region. Here, the background image BG is an image obtained by shooting the shooting target region in a reference state where there is no individual to be detected by the detection device 100. For example, if the detection apparatus 100 is an apparatus that inputs a captured image of a fixed point camera installed in an indoor place where the lighting environment does not change, such as an underground parking lot, as an original image P (t), The original image P (t) taken when no vehicle is present may be stored in the background image storage unit 120 as the background image BG. The arrows from the original image input unit 110 to the background image storage unit 120 in FIG. 1 indicate the flow of signals for storing such an original image P (t) as the background image BG.

  On the other hand, when the detection apparatus 100 is an apparatus that inputs, as an original image P (t), a captured image of a fixed-point camera installed in an outdoor place where the lighting environment changes, for example, a background image for daytime, A plurality of background images such as night background images may be prepared in the background image storage unit 120 and used separately. However, practically, the background image storage unit 120 has a function of creating and storing a background image BG based on a plurality of frame-unit images input by the original image input unit 110 in the past, and the background difference image. It is preferable to provide the latest background image BG to the creation unit 130.

  For example, a large number of original images P (t) input by the original image input unit 110 in the past predetermined time are taken into the background image storage unit 120, and an average image for these many original images P (t) is created. The created average image may be handled as the background image BG at that time. Specifically, for example, if an average image is created for a long time such as one hour, the created average image can be used as a background image unless there is a detection target that remains in the same position for a long time. .

  The background difference image creation unit 130 obtains the difference between the original image P (t) sequentially input by the original image input unit 110 and the background image BG given from the background image storage unit 120, thereby obtaining the background difference image D (t ). As described above, when the background image storage unit 120 has a function of generating an average image of many original images P (t), the background difference image generation unit 130 is the latest stored in the background image storage unit 120. The background difference image D (t) is created using the background image BG.

  On the other hand, the inter-frame difference image creation unit 140 creates an inter-frame difference image indicating a difference between different frames in time series for each original image P (t) sequentially input by the original image input unit 110. Is an element. For example, for the original image P (t1) given at time t1, a difference from the original image P (t0) which is the previous frame is obtained, and the obtained difference image is set as an inter-frame difference image F (t1). Perform output processing.

  The duplication presence / absence determination unit 150 determines the inter-frame difference image F (t) created by the inter-frame difference image creation unit 140 for each background difference region d in the background difference image D (t) created by the background difference image creation unit 130. ) Is a component that determines whether or not there is an overlapping portion with respect to the inter-frame difference region f. In the embodiment shown here, the duplication presence / absence determination unit 150 has a function of outputting the determination result in the form of an overlap confirmation image C (t). A method of creating the duplication confirmation image C (t) will be described later with a specific example.

  The individual recognition unit 160 is a component that recognizes an individual included in the original image P (t) as a stationary individual or a moving individual based on the determination result by the duplication presence / absence determination unit 150. In short, the original image P (t) It plays a role of recognizing whether each individual appearing above is a stationary individual or a moving individual. More specifically, the individual recognition unit 160 uses the duplication presence / absence determination unit 150 for each background difference region d included in the background difference image D (t) created by the background difference image creation unit 130. Based on the determination result (in the case of the illustrated example, duplication confirmation image C (t)), this is a component that performs processing for recognizing an individual occupying the background difference region d as a stationary individual or a moving individual. And a stationary individual recognition unit 161 and a moving individual recognition unit 162. The stationary individual recognition unit 161 performs recognition processing of the stationary individual, and the moving individual recognition unit 162 performs recognition processing of the moving individual.

  The term “individual” as used in the present application refers to a detection object that can be moved independently, such as one person or one vehicle, and a part such as a human foot or a vehicle tire. Is only a part of an “individual” and is not itself an “individual”. In addition, the “moving individual” in the present application refers to “an individual that is performing some kind of motion”, and only refers to an individual that is moving (a walking person or a running vehicle). is not. For example, a person waving while staying at a fixed position, or a vehicle blinking a blinker while stopping, is an “operating object” in the present application.

  On the other hand, the “stationary individual” referred to in the present application refers to an “individual that has not performed any motion”, and refers to an individual other than the “moving individual” among the individuals present in the imaging target region. . For example, a person who does not move his / her limbs while sitting, a parked vehicle, and the like are “moving individuals” as used herein. However, “whether or not an operation is being performed” is a matter to be determined based on whether or not there is a portion recognized as an inter-frame difference area f on the inter-frame difference image F (t), as will be described later. An individual who performs only a fine motion (for example, a person who performs only a breathing motion) whose difference between frames is less than or equal to a predetermined reference is determined as a “stationary individual”.

  The detected individual display unit 170 is a component that displays an individual index for identifying an individual recognized as a stationary individual or a moving individual by the individual recognition unit 160 on the display screen. That is, a process for displaying a stationary individual index s for an individual recognized as a stationary individual by the stationary individual recognition unit 161 and a moving individual index m for an individual recognized as a moving individual by the moving individual recognition unit 162 is performed. . The stationary individual index s is an index indicating an individual recognized as a stationary individual among the individuals displayed on the display screen, and the moving individual index m is an action among the individuals displayed on the display screen. It is an index indicating an individual recognized as an individual.

  In the example shown here, the detected individual display unit 170 is sequentially given the original image P (t) input by the original image input unit 110. Therefore, the detected individual display unit 170 creates an individual detected image Q (t) by superimposing the stationary individual index s or the moving individual index m on the original image P (t), and this individual detected image Q (t) Is displayed on the display screen. The detected individual display unit 170 displays both a stationary individual index s indicating an individual recognized as a stationary individual by the individual recognition unit 160 and a moving individual index m indicating an individual recognized as a moving individual on a display screen. When displaying on the screen, it has a function of displaying both in a visually distinguishable manner.

  Therefore, when this detection apparatus 100 is used as a monitoring system, a monitoring person who monitors each individual while looking at the display screen has a stationary individual index s and a moving individual index included in the individual detection image Q (t). By distinguishing and recognizing m, it is possible to visually grasp whether an individual on the screen is a stationary individual or an operating individual. Specific modes of the stationary individual index s and the moving individual index m will be described later.

  The basic configuration of the individual detection apparatus 100 according to the basic embodiment of the present invention has been described above as a collection of functional elements indicated by individual blocks with reference to FIG. 100 can be configured by incorporating a dedicated application program in the computer. Therefore, in practice, the components shown in each block in FIG. 1 are realized by a combination of computer hardware and software by a program.

  In the embodiment shown in FIG. 1, both the stationary individual recognition unit 161 and the moving individual recognition unit 162 are provided in the individual recognition unit 160, but it is not always necessary to provide both. For example, if the device is used only for recognition of a stationary individual, it is sufficient to provide only the stationary individual recognition unit 161, and the detected individual display unit 170 performs processing for displaying the stationary individual index s. Good. Similarly, if the apparatus is used only for recognition of the moving object, it is sufficient to provide only the moving object recognition unit 162, and the detected object display unit 170 performs processing for displaying the moving object index m. Just do it.

  Further, the detected individual display unit 170 has a function of creating an individual detected image Q (t) by superimposing the stationary individual index s or the moving individual index m on the original image P (t) and displaying it on the display screen. In the case of a device that is a component but is used for an application that does not need to display the individual detection image Q (t), it is not necessary to provide the detected individual display unit 170. For example, in the case of a device that is not used for monitoring work but counts the number of stationary and moving individuals recognized by the individual recognition unit 160 and uses the statistics based on the count value, the individual recognition unit It suffices if 160 can recognize and count individual stationary individuals and moving individuals. In such an application, it is not necessary to display the individual detection image Q (t) on the display screen for the monitor, and the detected individual display unit 170 is not necessary.

<<< §2. Stationary and moving individuals in the present invention >>>
As described in §1, the basic processing function of the individual detection apparatus 100 shown in FIG. 1 sequentially inputs an original image P (t) obtained by photographing with a fixed point camera installed at a predetermined monitoring place. (Input to the original image input unit 110) and sequentially output the individual detection image Q (t) on the display screen (output from the detected individual display unit 170).

  For example, when processing an original image P (t) captured at a frame rate of 30 frames / second by a fixed point camera, the original images P (t0), P (t1), P () given at intervals of 1/30 seconds While sequentially inputting t2), P (t3),..., the individual detection images Q (t1), Q (t2), Q (t3),. Processing to output sequentially is performed. In addition, since the inter-frame difference image F (t) is required to create the individual detection image Q (t), the individual detection image Q (t0) at the time t0 is not generated in the above example.

  The important point here is that the individual detection image Q (t) created in this way includes not only information indicating the detected individual individuals but also whether the individual is a stationary individual (stationary individual) or a moving individual. This also includes information indicating whether the object is an (operation individual). Hereinafter, the concept of “stationary object” and “moving object” according to the present invention will be described with reference to specific examples.

  FIG. 2 is a diagram showing a specific example of the correspondence between the original image P (t) input to the apparatus of FIG. 1 and the individual detection image Q (t) -S output from the apparatus of FIG. First, on the left side of FIG. 2, the original images P (t0), P (t1), P (t2), P (t3), and P (taken at the time points t0, t1, t2, t3, and t4 are displayed. t4) is illustrated. Each of these original images is taken by a fixed point camera installed to take a picture of a predetermined shooting target area. In the case of a general camera for moving image shooting, shooting is performed at a frame rate of about 30 frames / second. Here, for convenience, two persons are shot at a frame rate of about 1 frame / second. The following description will be given using the example described. Further, illustration of the background is omitted.

  The original image P (t0) is a frame image obtained by photographing at time t0, and a first individual A (indicated by reference numeral A0, hereinafter referred to as an adult) is located at the left end, and the first image at the right end. Two individuals B (indicated by the symbol B0, hereinafter referred to as children) are located. In this example, since adult A has moved to the right side of the figure and child B has moved to the left side of the figure, in the original image P (t1) taken at time t1, the position of adult A (A1) Is slightly shifted to the right, and the position of child B (B1) is slightly shifted to the left. Here, for convenience of explanation, it is assumed that the child B stops moving at time t1. Since adult A continues to move, in the original image P (t2) taken at time t2, the position of adult A (A2) is further shifted to the right and the position of child B (B2) remains at the original position. It is.

  Here, assuming that the adult A stops moving at time t2, in the original image P (t3) photographed at time t3, both the adult A and the child B remain in their original positions. Therefore, the illustrated original image P (t3) is the same as the original image P (t2). Thus, since the movement operation | movement of the adult A and the child B has stopped at the time of reaching the time t3, both are in the state of standing. However, after this, let's assume that adult A moves up one hand. In this case, in the original image P (t4) taken at time t4, the position of the adult A has not changed, but its form has changed to a state where one hand is raised (A4). For child B, neither the position nor the form has changed.

  As described above, the five original images P (t0) to P (t4) shown on the left side of FIG. 2 are frame-based images taken in time series from the top to the bottom, and two sets of individuals ( The image includes adult A and child B) as subjects. Hereinafter, for convenience of explanation, an individual corresponding to the adult A included in the original images P (t0) to P (t4) will be referred to as A0 to A4, and an individual corresponding to the child B will be referred to as B0 to B4. .

  Of course, the images of these individuals are actually arranged on the background image, but here, for the convenience of explanation, the background is simply indicated by a plain area. Further, if the adult A or the child B moves on foot, the form will change according to the movement of the limbs. Here, for convenience of explanation, the adults A0 to A3 are drawn as the same form (adults). Since A4 raises his hand, the form will be different), and children B0-B4 are also depicted as the same form.

  Now, paying attention to the movements of these two individuals, the adult A is in a walking motion from time t0 to t2, is stationary at time t3, and is moving up his arm at time t4. On the other hand, child B is in a walking motion from time t0 to t1, and is stationary from time t2 to t4. Of course, in reality, fine movement is caused by breathing or the like even when stationary, but in the present invention, when there is no significant change in position or form, it is determined that the object is stationary.

  In short, in the detection apparatus 100 according to the present invention, an individual having a significant change in either position or form is determined to be operating, and such an individual is referred to as an “operating individual”. Therefore, the “moving individual” referred to in the present invention includes not only a moving individual whose position changes but also an individual whose form is changed even at a fixed position. On the other hand, the “stationary individual” referred to in the present invention is an individual whose position and form have not changed. Of course, in practice, the meaning of “no change” means “no significant change exceeding a predetermined standard”, and when the position change or the shape change is below the reference, It will be recognized as a “stationary individual”.

  In the embodiment shown here, when determining whether an individual appearing on a single frame image is a moving individual or a stationary individual, the position and form are substantially substantial with reference to the previous frame image. It is determined whether or not a change has occurred.

  For example, in the case of the illustrated example, the adult A1 has changed its position with respect to the previous A0, so it is determined as an “operating individual”, and the adult A2 has changed its position with respect to the previous A1. Therefore, it is determined as an “operating individual”. On the other hand, the adult A3 is determined to be a “stationary individual” because neither the position nor the form has changed with respect to the previous A2. The adult A4 does not change the position with respect to the previous A3, but changes the form, so it is determined as the “moving individual”. Similarly, the child B1 is determined to be an “operating individual” because the position is changed with respect to the previous B0, but the children B2 to B4 are not changed in position or form with respect to the previous one. Therefore, it is determined as a “stationary individual”.

  An object of the present invention is to detect individual individuals by distinguishing whether they are operating or stationary based on images taken by a fixed point camera. In other words, the object is not only to detect individual individuals appearing in a captured image, but also to recognize whether each detected individual is an operating individual or a stationary individual. In particular, in the embodiment shown in FIG. 1, the individual recognition unit 160 recognizes whether each individual is an operating individual or a stationary individual, and further displays the recognition result on the display screen by the detected individual display unit 170. It has a function to do.

  Specifically, the detected individual display unit 170 creates an individual detected image Q (t) by superimposing the stationary individual index s or the moving individual index m on the original image P (t) and displays it on the display screen. It has a function. The individual detection image Q (t) -S shown on the right side of FIG. 2 shows a display example of the detection result of the stationary individual by the detected individual display unit 170, and the original images P (t1) to t4 at the times t1 to t4. This is an image in which a stationary individual index s indicating an individual recognized as a stationary individual by the stationary individual recognition unit 161 is superimposed on P (t4). In the figure, the stationary individual index s is shown as a hatched area by a fine rectangular dot pattern.

  First, let us look at the individual detection image Q (t1) -S obtained corresponding to the original image P (t1) at time t1. In the case of the illustrated example, the individual detection image Q (t1) -S is the same image as the original image P (t1), and both are images including an adult A1 and a child B1. In this individual detection image Q (t1) -S, the stationary individual index s is not displayed. This is because both the adult A1 and the child B1 are moving individuals (the positions are changed with respect to the previous A0 and B0), and there are no stationary individuals.

  Next, let us look at the individual detection image Q (t2) -S obtained corresponding to the original image P (t2) at time t2. In the illustrated example, the individual detection image Q (t2) -S is an image in which the stationary individual index s1 (t2) is superimposed on the original image P (t2). Specifically, the stationary individual index s1 (t2) is shown as a hatched area with fine rectangular dots overlapping the area of the child B2. This is because the child B2 is a stationary individual whose position and form have not changed with respect to the previous B1.

  Next, let us look at the individual detection image Q (t3) -S obtained corresponding to the original image P (t3) at time t3. In the illustrated example, the individual detection image Q (t3) -S is an image in which the stationary individual indices s1 (t3) and s2 (t3) are superimposed on the original image P (t3). Specifically, the stationary individual index s1 (t3) is shown as a hatched area by fine rectangular dots that overlap the area of the adult A3, and the stationary individual index s2 (t3) is a fine rectangular dot that overlaps the area of the child B3. Are shown as hatched areas. This is an adult A3 is a stationary individual whose position and form have not changed with respect to the previous A2, and a child B3 is a stationary individual whose position and form has not changed with respect to the previous B2. Because.

  Finally, let us look at the individual detection image Q (t4) -S obtained corresponding to the original image P (t4) at time t4. In the illustrated example, the individual detection image Q (t4) -S is an image in which the stationary individual index s1 (t4) is superimposed on the original image P (t4). Specifically, the stationary individual index s1 (t4) is shown as a hatched area with fine rectangular dots overlapping the area of the child B4. This is because the child B4 is a stationary individual whose position and form have not changed with respect to the previous B3. The adult A4 does not change its position relative to the previous A3, but changes its form (because one hand is raised), and thus becomes an operating individual.

  After all, if the individual detection apparatus 100 shown in FIG. 1 is used as a monitoring system and an individual detection image Q (t) -S as shown on the right side of FIG. 2 is displayed on the display screen, a fixed point camera is obtained. Together with the captured image (original image P (t)), the stationary individual index s indicating the individual determined to be a stationary individual at each time will be presented. It becomes possible to do.

  For example, in the case of the example shown in FIG. 2, since the stationary individual index s is superimposed on the display screen as a hatched area of fine rectangular dots on the display screen between times t2 and t4, It can be recognized that the child B is in a state of not moving all the time. That is, the stationary individual index s composed of the hatched area by the fine rectangular dots not only shows the detection result of the individual called the child B but also plays a role of indicating that the individual is a stationary individual. Therefore, if this apparatus is used as a pool monitoring system, it is possible to perform monitoring with attention paid to users who may have drowned.

  On the other hand, FIG. 3 is a diagram showing a specific example of the correspondence relationship between the original image P (t) input to the apparatus of FIG. 1 and the individual detection image Q (t) -M output from the apparatus of FIG. . The original images P (t0), P (t1), P (t2), P (t3), and P (t4) shown on the left side of FIG. 3 are time points t0, t1, t2, t3, and t4. 2 is a frame-by-frame image taken by a fixed point camera, which is exactly the same as the image shown on the left side of FIG. On the other hand, the individual detection image Q (t) -M shown on the right side of FIG. 3 shows a display example of the detection result of the operating individual by the detected individual display unit 170.

  As described above, the individual detection image Q (t) -S shown on the right side of FIG. 2 is an image showing the detection result of the stationary individual, and the original images P (t1) to P (t4) at the times t1 to t4. On the top, an image in which a stationary individual index s indicating an individual recognized as a stationary individual by the stationary individual recognition unit 161 is superimposed. On the other hand, the individual detection image Q (t) -M shown on the right side of FIG. 3 is an image showing the detection result of the moving individual, and the original images P (t1) to P (t4) at the times t1 to t4. On top of this, an action individual index m indicating an individual recognized as an action individual by the action individual recognition unit 162 is superimposed. In the figure, the moving individual index m is shown as a hatched area with a hatched mesh pattern.

  First, let's look at the individual detection image Q (t1) -M obtained corresponding to the original image P (t1) at time t1. In the case of the illustrated example, the individual detection image Q (t1) -M is an image in which the motion individual indexes m1 (t1) and m2 (t1) are superimposed on the original image P (t1). Specifically, the motion individual index m1 (t1) is shown as a hatched area with hatched meshes overlapping the area of the adult A1, and the motion individual index m2 (t1) is hatched with hatched meshes overlapping with the area of the child B1. Shown as a region. This is because the adult A1 is an operating individual whose position is changed with respect to the previous A0, and the child B1 is an operating individual whose position is changed with respect to the previous B0.

  Next, let us look at the individual detection image Q (t2) -M obtained corresponding to the original image P (t2) at time t2. In the case of the illustrated example, the individual detection image Q (t2) -M is an image in which the motion individual index m1 (t2) is superimposed on the original image P (t2). Specifically, the motion individual index m1 (t2) is shown as a hatched area with a hatched mesh overlapping the area of the adult A2. This is because the adult A2 is an operating individual whose position is changed with respect to the previous A1.

  Next, let us look at the individual detection image Q (t3) -M obtained corresponding to the original image P (t3) at time t3. In the case of the illustrated example, the individual detection image Q (t3) -M is the same image as the original image P (t3), and the motion individual index m is not displayed. This is because both the adult A3 and the child B3 are stationary individuals (the positions and forms are not changed with respect to the previous A2 and B2), and there are no moving individuals.

  Finally, let's look at the individual detection image Q (t4) -M obtained corresponding to the original image P (t4) at time t4. In the case of the illustrated example, the individual detection image Q (t4) -M is an image in which the motion individual index m1 (t4) is superimposed on the original image P (t4). Specifically, the motion individual index m1 (t4) is shown as a hatched area with a hatched mesh overlapping the area of the adult A4. This is because the adult A4 is an operating individual whose position is not changed with respect to the previous A3 but whose form is changed (with one hand raised).

  After all, if the individual detection apparatus 100 shown in FIG. 1 is used as a monitoring system and an individual detection image Q (t) -M as shown on the right side of FIG. 3 is displayed on the display screen, a fixed point camera is obtained. In addition to the captured image (original image P (t)), the motion individual index m indicating the individual determined to be the motion individual at each time is presented, so that the monitor monitors the motion individual with attention. It becomes possible to do.

  For example, in the case of the example shown in FIG. 3, at time t1, t2, t4, the motion individual indicator m is superimposed and displayed as a hatched area with hatched meshes on the image of the adult A on the display screen. It can be recognized that the adult A is performing some kind of operation. That is, the moving individual index m composed of the hatched area by the hatched mesh not only shows the detection result of the individual called adult A, but also plays a role of indicating that the individual is an operating individual. Therefore, if this device is used as a security system in a lecture hall or the like, it is possible to perform security that pays attention to an audience performing suspicious behavior.

  FIG. 4 is a diagram illustrating a specific example of a correspondence relationship between the original image P (t) input to the apparatus of FIG. 1 and the individual detection image Q (t) -SM output from the apparatus of FIG. The original images P (t0), P (t1), P (t2), P (t3), and P (t4) shown on the left side of FIG. 4 are time points t0, t1, t2, t3, and t4. 2 is a frame-by-frame image taken by a fixed-point camera, which is exactly the same as the image shown on the left side of FIGS. On the other hand, the individual detection image Q (t) -SM shown on the right side of FIG. 4 shows a display example of detection results of both stationary and moving individuals by the detected individual display unit 170.

  In short, the individual detection image Q (t) -SM shown on the right side of FIG. 4 is recognized as a stationary individual by the stationary individual recognition unit 161 on the original images P (t1) to P (t4) at each time t1 to t4. This is an image in which both the stationary individual index s indicating the individual and the moving individual index m indicating the individual recognized as the moving individual by the moving object recognition unit 162 are superimposed. In the figure, the stationary individual index s is shown as a hatched area with a fine rectangular dot pattern, and the moving individual index m is shown as a hatched area with a hatched mesh pattern.

  That is, on the individual detection image Q (t1) -SM obtained in correspondence with the original image P (t1) at time t1, the motion individual index m1 (t1) and the adult individual A1 and the child B1 that are motion individuals are displayed. m2 (t1) is superimposed. Further, on the individual detection image Q (t2) -SM obtained corresponding to the original image P (t2) at time t2, the motion individual index m1 (t2) is superimposed on the adult A2 that is the motion individual, A stationary individual index s1 (t2) is superimposed on the child B2, which is a stationary individual. On the individual detection image Q (t3) -SM obtained corresponding to the original image P (t3) at time t3, the stationary individual index s1 (t3) and the adult individual A3 and the child B3 that are stationary individuals are displayed. s2 (t3) is superimposed. Finally, on the individual detection image Q (t4) -SM obtained corresponding to the original image P (t4) at time t4, the motion individual index m1 (t4) is superimposed on the adult A4 that is the motion individual. The stationary individual index s1 (t4) is superimposed on the child B2, which is a stationary individual.

  After all, if the individual detection apparatus 100 shown in FIG. 1 is used as a monitoring system and an individual detection image Q (t) -SM as shown on the right side of FIG. 4 is displayed on the display screen, a fixed point camera is obtained. A stationary individual index s indicating an individual determined to be a stationary individual and a moving individual index m indicating an individual determined to be an operating individual are presented at each time, together with a captured image (original image P (t)). It will be. In addition, the stationary individual index s and the moving individual index m are displayed in a visually distinguishable manner (in the example shown in the figure, a hatching area with a fine rectangular dot pattern and a hatched area with a hatched mesh pattern), The monitor can distinguish and recognize a stationary individual and a moving individual.

  That is, the observer can intuitively grasp the hatched area as an individual individual (in this case, a person), and if it is hatched by a fine rectangular dot, it is a stationary individual or hatched by a hatched mesh. If there is, it is possible to distinguish and grasp each as an individual in operation. For this reason, in various environments, it is possible to perform monitoring with attention directed to a stationary individual, and conversely, monitoring with attention directed to a moving individual.

  As described above, the individual detection device 100 shown in FIG. 1 can also display the individual detection image Q (t) -S indicating the detection result of the stationary individual as shown on the right side of FIG. The individual detection image Q (t) -M showing the detection result of the moving individual can be displayed as shown on the right side of FIG. 4, and the detection result of both the stationary individual and the moving individual as shown on the right side of FIG. It is also possible to display an individual detection image Q (t) -SM indicating In the case of a device that only needs to display the individual detection image Q (t) -S, it is not necessary to provide the motion individual recognition unit 162 shown in FIG. 1, and the motion detection image Q (t) -M is displayed. In the case of a device that is sufficient, it is not necessary to provide the stationary individual recognition unit 161 shown in FIG.

  When the individual detection device 100 shown in FIG. 1 is regarded as a device that displays an individual detection image Q (t) -SM showing detection results of both stationary and moving individuals, as shown on the right side of FIG. The device is an individual detection device for detecting an individual existing in a predetermined imaging target area as a stationary stationary object or an operating individual in motion, and frames arranged in time series by imaging with respect to the predetermined imaging target area An original image input unit 110 that inputs an original image P (t) obtained as a unit image, and an individual recognition unit that recognizes each individual appearing on the original image P (t) as a stationary individual or a moving individual 160, based on the recognition result by the individual recognition unit 160, on the original image P (t), a stationary individual index s indicating an individual recognized as a stationary individual, and an operating individual indicating an individual recognized as an operating individual index If, by superimposing, and a detecting individual display unit 170 and the stationary individual indices s and operating individual indices m may be displayed in a visually distinguishable and become such embodiments respectively, it can be that the apparatus comprising a.

<<< §3. Operation of the individual detection apparatus according to the basic embodiment >>
In §2 described above, when the original images P (t0) to P (t4) are input to the individual detection apparatus 100 shown in FIG. 1, what kind of individual detection images Q (t1) to Q (t4) are input. Is output with reference to the specific examples of FIGS. 2 to 4, and the individual detection images Q (t 1) to Q (t 4) displayed on the display screen are used to add additional information to the monitor. He also added that there is a merit that can provide information. After all, the individual detection apparatus 100 shown in FIG. 1 creates and outputs individual detection images Q (t1) to Q (t4) based on the given original images P (t0) to P (t4). It will be a device that performs. Here, specific processing operations of each part of the individual detection apparatus 100 will be described using the specific images shown in FIGS. 2 to 4 as examples.

<3-1. Operation of Original Image Input Unit 110>
The original image input unit 110 performs a process of inputting an image captured by a fixed point camera or the like as a frame-unit original image P (t) arranged in time series, and temporarily stores these original images P (t) as necessary. To save automatically. 2 to 4 show actual examples of original images P (t0) to P (t4) input at each time point t = t0, t1, t2, t3, and t4.

  As described above, a general video camera performs shooting at a frame rate of about 30 frames / second. For convenience of explanation, FIGS. 2 to 4 show a frame rate of about 1 frame / second. Original images P (t0), P (t1), P (t2), P (t3),. As will be described later, the inter-frame difference image creation unit 140 performs a process of obtaining an inter-frame difference from the previous original image for each original image continuously input in time series. .

  Therefore, when the fixed-point camera captures images at a frame rate of 30 frames / second, when the original image input unit 110 directly inputs these captured images as the original image P (t), the background difference image creation unit 130 The background difference image D (t) is created at a period of 1/30 seconds, and the inter-frame difference image creation unit 140 takes a difference from the original image 1/30 seconds before at a period of 1/30 seconds. Thus, the inter-frame difference image F (t) is created.

  However, in practice, the period for creating the background difference image D (t) and the inter-frame difference image F (t) does not necessarily have to be set to a short period of about 1/30 seconds, and is set to a longer period. It doesn't matter. In that case, the original image input unit 110 may be provided with a function of sampling and inputting images in frame units arranged in time series from the outside at a predetermined sampling period. In this case, the background difference image creation unit 130 suffices to create a background difference image D (t) for each sampled (temporarily thinned) original image P (t). For the sampled image of each frame, it is sufficient to create an inter-frame difference image F (t) by performing a difference operation with the immediately preceding image in time series. For example, setting the sampling period to 1 second is the same as using an original image with a frame rate of about 1 frame / second.

<3-2. Operation of Background Image Storage Unit 120>
The background image storage unit 120 is a component that stores the background image BG for the shooting target area of the fixed point camera. As described in §1, when using a fixed-point camera installed in an indoor place where there is no change in the lighting environment, such as an underground parking lot, a standard in which an individual to be detected does not exist as the background image BG In the state, an image captured in advance can be used.

  However, when using a fixed-point camera installed in an outdoor place where the lighting environment changes, an average image of a plurality of frame-unit images previously input by the original image input unit 110 to the background image storage unit 120. And a function for automatically creating the image, and the created average image may be switched as the latest background image BG at a predetermined timing. Since such a technique for automatically creating a background image is a known technique, a detailed description thereof will be omitted in the present application.

<3-3. Operation of Background Difference Image Creating Unit 130>
The background difference image creation unit 130 creates a background difference image D (t) indicating a difference from the background image BG for each original image P (t) input by the original image input unit 110. FIG. 5 is a diagram showing a specific example of the background difference image D (t) created in this way, and the background is shown on the right side of the drawing with respect to the original images P (t1) to P (t4) shown on the left side of the drawing. An example in which difference images D (t1) to D (t4) are created is shown. An image BG shown in the upper right of the figure is a background image.

  Here, for convenience of explanation, the background image BG is illustrated as a plain image, but the actual background image BG is an image in which only the background of the shooting target region is shown, and the original images P (t0) to P ( t4) is an image in which each individual (adult A and child B) is shown in the foreground of the background image BG. Note that the background difference image D (t0) corresponding to the original image P (t0) can also be created. However, since it is not necessary for the following procedure, the creation of the background difference image D (t0) is omitted here. Has been.

  The background difference image D (t) is an image created by a difference calculation between the original image P (t) and the background image BG. Although the specific procedure of the difference calculation will be described later, the difference image in the embodiment described here is an image made up of a collection of pixels having a pixel value of “0” or “1”. If the pixel values of the corresponding pixels of the two sets of images are within the predetermined approximate range, the pixel is set to the pixel value “0”. If the pixel values are outside the predetermined approximate range, the pixel is set to the pixel value “1”. It is obtained by the operation to do. In the background difference image D (t) in FIG. 5, the portion with the pixel value “1” is shown in black, and the portion with the pixel value “0” is shown in white.

  The background difference image D (t1) shown in FIG. 5 is an image created by the difference calculation between the original image P (t1) and the background image BG, and the area portion occupied by the adult A1 and the child B1 is shown in black. Yes. Here, the region portion occupied by the adult A1 is referred to as a background difference region d1 (t1), and the region portion occupied by the child B1 is referred to as a background difference region d2 (t1). Therefore, the background difference image D (t1) is an image including the background difference area d1 (t1) and the background difference area d2 (t1), and the background difference area d1 (t1) and the background difference area d2 (t1). Is constituted by an aggregate region of pixels having a pixel value “1”.

  Similarly, the background difference image D (t2) is an image created by the difference calculation between the original image P (t2) and the background image BG, and the background difference area d1 (t2) corresponding to the adult A2 and the child B2 The image includes the corresponding background difference area d2 (t2). The background difference image D (t3) is an image created by the difference calculation between the original image P (t3) and the background image BG, and corresponds to the background difference area d1 (t3) corresponding to the adult A3 and the child B3. The background difference image D (t4) is an image created by the difference calculation between the original image P (t4) and the background image BG, and corresponds to the adult A4. The image includes a background difference area d1 (t4) and a background difference area d2 (t4) corresponding to the child B4.

  As described above, the background difference image creation unit 130 includes the background difference image D including the background difference region di (t) indicating the difference from the background image BG for each original image P (t) input by the original image input unit 110. This is a component for creating (t). Here, i is a parameter indicating the number of the background difference area (substantially corresponding to each individual) included in the image, and is hereinafter referred to as “background difference area number”. In the case of the example shown in FIG. 5, i = 1 or 2, i = 1 is a number given to adult A, and i = 2 is a number given to child B.

  Each background difference area di (t) obtained in this way eventually corresponds to each individual existing in the original image P (t) photographed at time t. For example, the background difference area d1 (t1) in the background difference image D (t1) corresponds to the adult A1, and the background difference area d2 (t1) corresponds to the child B1. Therefore, if only individual individuals are detected at each time t, it is sufficient to create the background difference image D (t) by the background difference image creation unit 130, and each background difference image D (( Black human figures (background difference areas) included in t1) to D (t4) indicate individual individuals (persons). However, in the present invention, it is necessary to detect each individual by distinguishing whether it is in motion or at rest, so the processing described below is required.

<3-4. Operation of inter-frame difference image creation unit 140>
The inter-frame difference image creation unit 140, for each original image P (t) input by the original image input unit 110, an inter-frame difference image F () indicating a difference from a different frame (temporarily previous frame) in time series. t). FIG. 6 is a diagram showing a specific example of the inter-frame difference image F (t) created in this way. Based on the original images P (t0) to P (t4) shown on the left side of the figure, each is shown on the right side of the figure. The example in which the inter-frame difference images F (t1) to F (t4) shown are created is shown. That is, the nth inter-frame difference image is a difference image between the original image constituting the nth frame and the original image constituting the (n−1) th frame.

  Again, this inter-frame difference image is also an image made up of a collection of pixels having pixel values of either “0” or “1”, and the pixel values of the corresponding pixels of the two sets of target images are predetermined. If the pixel is within the approximate range, the pixel is set to a pixel value “0”, and if the pixel is outside the predetermined approximate range, the pixel is determined to be the pixel value “1”. In the inter-frame difference image F (t) in FIG. 6, the portion with the pixel value “1” is shown in black, and the portion with the pixel value “0” is shown in white.

  Specifically, the inter-frame difference image F (t1) shown in FIG. 6 is an image created by the difference calculation between the original image P (t0) and the original image P (t1). The change area of the image caused by the movement is shown in black. Here, the change area generated when the adult A moves from A0 to A1 is referred to as an interframe difference area f1 (t1), and the change area generated when the child B moves from B0 to B1 is referred to as the interframe difference area. Call it f2 (t1). Therefore, the inter-frame difference image F (t1) is an image including the inter-frame difference area f1 (t1) and the inter-frame difference area f2 (t1), and the inter-frame difference area f1 (t1) and the inter-frame difference. The region f2 (t1) is configured by an aggregate region of pixels having a pixel value “1”.

  If you look closely at the example shown in the figure, the inter-frame difference area f1 (t1) is an area in which the shadow of adult A is slightly shifted left and right, but it is white on part of the arm and part of the foot. There are unplugged parts. This white portion corresponds to an overlapping portion of the left and right adults A, and is a region where a sufficient difference was not obtained during the difference calculation. In other words, the black part is an area where a sufficient difference is obtained because the difference between the background part and the person part is obtained, whereas the white part is the difference between the person parts. This is a region where a sufficient difference could not be obtained. Similarly, in the inter-frame difference region f2 (t1), white portions are generated in a part of the arms and a part of the legs.

  As described above, here, for convenience of explanation, an example in which two persons are photographed at a frame rate of about 1 frame / second is shown, so the inter-frame difference area f1 (t1) and the inter-frame difference area Although f2 (t1) is an exaggerated example in which two shadows of a person are shifted and overlapped to the left and right, in the case of a general video camera, shooting at a frame rate of about 30 frames / second Therefore, this deviation amount becomes smaller.

  Similarly, the inter-frame difference image F (t2) is an image created by the difference calculation between the original image P (t1) and the original image P (t2). The inter-frame difference image F (t2) includes a change area generated when the adult A moves from A1 to A2 as an inter-frame difference area f1 (t2). Since child B is not moving, there is no inter-frame difference area for child B.

  Further, the inter-frame difference image F (t3) is an image created by a difference calculation between the original image P (t2) and the original image P (t3). This inter-frame difference image F (t3) is actually an empty image as shown in the figure. This is because during the time t2 to t3, neither the adult A nor the child B moves, and there is no inter-frame difference area.

  The last inter-frame difference image F (t4) is an image created by calculating a difference between the original image P (t3) and the original image P (t4). This inter-frame difference image F (t4) includes changed areas f1 (t4) and f2 (t4) that are generated by changing the form of the adult A from A3 to A4 (raising hands). It is. Since child B has not moved or changed its shape, there is no inter-frame difference area for child B. In this example, it is assumed that the areas f1 (t4) and f2 (t4) are not connected to each other, but are slightly separated from each other, and are recognized as two different areas.

  As described above, the inter-frame difference image creating unit 140 creates an inter-frame difference region fj (t) indicating a difference from the previous frame in the time series for each original image P (t) input by the original image input unit 110. This is a constituent element for creating the inter-frame difference image F (t). Here, j is a parameter indicating the inter-frame difference area included in the image, and is hereinafter referred to as “inter-frame difference area number”. This inter-frame difference area number j does not necessarily correspond to each individual.

  For example, in the illustrated inter-frame difference image F (t1), j = 1 is set as the first inter-frame difference area f1 (t1), and j = 2 is set as the second inter-frame difference area f2 (t1). , J = 1 happens to correspond to the first individual (adult A), and j = 2 happens to correspond to the second individual (child B). However, in the illustrated inter-frame difference image F (t4), j = 1 is set as the first inter-frame difference area f1 (t4), and j = 2 is set as the second inter-frame difference area f2 (t4). ), Both j = 1 and j = 2 correspond to the first individual (adult A).

  In the embodiment described here, a typical example in which the inter-frame difference image creation unit 140 creates a difference image between two frames that are continuous in time series given from a fixed point camera will be described. However, the inter-frame difference image is not necessarily a difference image between two frames that are continuous in time series. For example, as described above, when the original image input unit 110 has a sampling function, the two frame images to be subjected to the difference calculation are not continuous frame images obtained by photographing with a fixed point camera, but at a sampling period. That is, two frame images with a corresponding time interval.

  Recently, in order to improve moving object extraction accuracy, a technique for creating a difference image for three or more frame images, a technique for creating a difference image using a motion vector, and the like have been proposed. The inter-frame difference image creation unit 140 according to the present invention may create an inter-frame difference image using such a technique. In short, the inter-frame difference image creation unit 140 according to the present invention, for each frame image input in time series, is another frame that precedes in time (not necessarily the previous frame, or one frame) A process of creating an inter-frame difference image including an inter-frame difference area indicating a difference with (not limited to) may be performed.

<3-5. Operation of Duplication Presence Judgment Unit 150>
The duplication presence / absence determination unit 150 determines whether or not there is an overlapping portion of each background difference area di (t) in the background difference image D (t) with respect to the inter-frame difference area fj (t) in the inter-frame difference image F (t). Is a component for determining In particular, in the case of the embodiment shown here, the duplication presence / absence determination unit 150 determines the background difference area di (t) and the inter-frame difference area based on the background difference image D (t) and the inter-frame difference image F (t). It has a function of creating an overlapping confirmation image C (t) including an overlapping partial region ci (t) indicating an overlapping portion with fj (t). 7-10 is a figure which shows the specific example of the duplication confirmation image C (t) produced in this way. In each overlap confirmation image C (t), the overlapping portion between the background difference region di (t) and the inter-frame difference region fj (t) is shown as a black region, and the non-overlapping portion is shown as a white region. .

  First, FIG. 7 shows an overlap confirmation image C (t1) based on the background difference image D (t1) shown at time t1 in FIG. 5 and the inter-frame difference image F (t1) shown at time t1 in FIG. ) Is created. As illustrated, the background difference image D (t1) includes background difference areas d1 (t1) and d2 (t1), and the interframe difference image F (t1) includes an interframe difference area f1 (t1). ), F2 (t1). For this reason, on the overlap confirmation image C (t1), an overlapping partial region c1 (t1) corresponding to the overlapping portion of the background difference region d1 (t1) and the inter-frame difference region f1 (t1), and the background difference region d2 An overlapping partial region c2 (t1) corresponding to an overlapping portion between (t1) and the inter-frame difference region f2 (t1) is shown as a black region.

  As shown in the figure, the human shape of the overlapping partial region c1 (t1) lacks a part of the humanoid limb of the background difference region d1 (t1), and the human shape of the overlapping partial region c2 (t1) Is a part of the background difference region d2 (t1) lacking a part of the humanoid limb. This is because the double humanoid of the interframe difference region f1 (t1) and the double humanoid of the interframe difference region f2 (t1) lack a part of the limbs.

  As described above, in the embodiment described here, the background difference image D (t) created by the background difference image creation unit 130 is a binary consisting of a collection of pixels having pixel values “1” or “0”. Each background difference area di (t) is represented as a continuous area (black area in the figure) of pixels having a pixel value “1”, which is an image. Further, the inter-frame difference image F (t) created by the inter-frame difference image creation unit 140 is also a binary image composed of a collection of pixels having the pixel value “1” or “0”, and the pixel value “1”. The inter-frame difference area fj (t) is expressed as a continuous area (black area in the figure) of pixels having "".

  Therefore, the duplication presence / absence determination unit 150 performs a logical product operation on the corresponding pixels on the background difference image D (t) and the inter-frame difference image F (t), thereby obtaining a pixel value “1” or “0”. The duplication confirmation image C (t), which is a binary image made up of an aggregate of pixels having the above, can be created. In this overlap confirmation image C (t), an area having a pixel value “1” (black area in the figure) is an overlapped partial area ci (t).

  Similarly, FIG. 8 shows an overlap confirmation image C ((2) based on the background difference image D (t2) shown at time t2 in FIG. 5 and the inter-frame difference image F (t2) shown at time t2 in FIG. The state where t2) has been created is shown. As illustrated, the background difference image D (t2) includes background difference regions d1 (t2) and d2 (t2), and the inter-frame difference image F (t2) includes an inter-frame difference region f1 (t2). )It is included. For this reason, on the overlap confirmation image C (t2), the overlapping partial region c1 (t2) corresponding to the overlapping portion of the background difference region d1 (t2) and the inter-frame difference region f1 (t2) is shown as a black region. Has been.

  As shown in the figure, the human part of the overlapping partial area c1 (t2) lacks a part of the humanoid limb of the background difference area d1 (t2), but this is different from the inter-frame difference area f1 ( This is because the double humanoid of t2) lacks a part of the limb. Since there is no overlapping inter-frame difference area for the background difference area d2 (t2), the overlapping partial area c2 (t2) is not actually formed, but in FIG. 8, for convenience of explanation. The overlapping partial area c2 (t2) is assumed to be a virtual area, and only its outline is drawn. In other words, the overlapped partial region c2 (t2) shown in the figure is a region that does not actually exist including its contour portion, and in fact, pixels having a pixel value “0” are arranged in this portion. Yes.

  On the other hand, FIG. 9 shows an overlap confirmation image C (t3) based on the background difference image D (t3) shown at time t3 in FIG. 5 and the inter-frame difference image F (t3) shown at time t3 in FIG. ) Is created. As illustrated, the background difference image D (t3) includes background difference areas d1 (t3) and d2 (t3), but the interframe difference image F (t3) has no interframe difference area. Not included. For this reason, the overlap confirmation image C (t3) is an empty image that does not include any overlapping partial areas (all areas are non-overlapping parts). However, in FIG. 9, for convenience of explanation, the overlapping partial areas c1 (t3) and c2 (t3) are assumed to be virtual areas, and only their outlines are drawn. That is, the overlapping partial areas c1 (t3) and c2 (t3) shown in FIG. 9 are areas that do not actually exist including the outline portion, and actually, the pixel value “0” is set in this portion. Pixels having the same are arranged.

  Finally, in FIG. 10, based on the background difference image D (t4) shown at time t4 in FIG. 5 and the inter-frame difference image F (t4) shown at time t4 in FIG. The state where t4) has been created is shown. As illustrated, the background difference image D (t4) includes background difference areas d1 (t4) and d2 (t4), and the inter-frame difference image F (t4) includes an inter-frame difference area f1 (t4). ), F2 (t4). For this reason, on the overlap confirmation image C (t4), the overlapping partial region c1 (t4) corresponding to the overlapping portion of the background difference region d1 (t4) and the inter-frame difference region f1 (t4) is shown as a black region. Has been.

  In FIG. 10, for convenience of explanation, the outlines of the overlapping partial areas c1 (t4) and c2 (t4) are drawn as virtual area outlines. That is, the overlapping partial region c1 (t4) shown in FIG. 10 is originally a region constituted only by a black region, and the white portion is a region that does not actually exist including its contour line portion. Similarly, the overlapping partial region c2 (t4) is also a non-existent region including the contour portion. Eventually, in this overlap confirmation image C (t4), only the pixels in the area corresponding to the arm portion painted in black have the pixel value “1”, and all the other pixels have the pixel value “0”. .

  In the example shown in FIG. 10, the two sets of inter-frame difference areas f1 (t4) and f2 (t4) do not correspond to the two sets of background difference areas d1 (t4) and d2 (t4), respectively. The correspondence between the two is not a one-to-one relationship. In other words, the two sets of inter-frame difference areas f1 (t4) and f2 (t4) are both areas corresponding to the arm of the adult A, and both are areas corresponding to the background difference area d1 (t4). It will be. As described above, the background difference area di (t) on the background difference image D (t) and the inter-frame difference area fj (t) on the inter-frame difference image F (t) are not necessarily in a one-to-one correspondence. However, there is no problem in obtaining the duplication confirmation image C (t).

<3-6. Operation of individual recognition unit 160>
As described above, the individual recognition unit 160 determines the background difference area di (t) based on the determination result by the duplication presence / absence determination unit 150 for each background difference area di (t) included in the background difference image D (t). It is a component that performs processing for recognizing an individual occupying (t) as a stationary individual or a moving individual. In the case of the embodiment described here, the duplication presence / absence determination unit 150 has a function of creating duplication confirmation images C (t1) to C (t4) as shown in FIGS. Therefore, the individual recognition unit 160 can confirm the presence or absence of an overlapping portion by referring to these overlap confirmation images C (t1) to C (t4), and can recognize each individual as a stationary individual or a moving individual. it can.

  FIG. 11 shows specific original images P (t0) to P (t4) that have been exemplified so far, and specific duplication confirmation images C (t1) to C (t4) created corresponding to these. It is a figure which shows a correspondence. That is, the illustrated overlap confirmation images C (t1) to C (t4) are the same as the overlap confirmation images C (t1) to C (t4) illustrated in FIGS.

  Individuals (adult A and child B) on the original images P (t1) to P (t4) can be easily recognized by using the background difference images D (t1) to D (t4) shown in FIG. Can do. For example, in the example shown in FIG. 5, there are two sets of individuals A1 and B1 on the original image P (t1) at time t1, but the individual A1 is the second one on the background difference image D (t1). It can be recognized as an individual occupying one background difference area d1 (t1), and the individual B1 can be recognized as an individual occupying the second background difference area d2 (t1) on the background difference image D (t1).

  Therefore, in the original images P (t1) to P (t4) shown on the left in FIG. 11, the background differences corresponding to the individual in parentheses after the codes A1 to A4 and B1 to B4 indicating the individual individuals. The area code is added. For example, in the case of the original image P (t1) at the time t1, the individual A1 can be recognized as an individual occupying the first background difference area d1 (t1), and therefore “d1 ( t1) ”indicating the background difference region is added, and the individual B1 can be recognized as an individual occupying the second background difference region d2 (t1). Therefore,“ d2 ( The reference numeral indicating the background difference area “t1)” is added.

  Thus, the individual recognition unit 160 recognizes each individual on the original image P (t) by using each background difference area di (t) on the background difference image D (t), and then further By using the overlap confirmation image C (t), it is possible to determine whether each recognized individual is a stationary individual or a moving individual.

  Specifically, the stationary individual recognizing unit 161 responsible for recognizing a stationary individual has obtained a determination result that there is no substantial overlapping portion with respect to the inter-frame difference region fj (t) by the duplication presence / absence determining unit 150. An individual occupying the background difference area di (t) may be recognized as a stationary individual. Similarly, the motion individual recognition unit 162 responsible for recognition of the motion individual determines by the duplication presence / absence determination unit 150 that there is a substantial overlap with the inter-frame difference region fj (t) for at least a part of the region. The individual that occupies the background difference area di (t) obtained from the above may be recognized as an operating individual.

  In other words, the individual recognition unit 160 refers to the corresponding region on the overlap confirmation image C (t) for the specific background difference region di (t) included in the background difference image D (t), and When the corresponding overlapping area does not include a substantial overlapping partial area ci (t), the individual occupying the specific background difference area di (t) is recognized as a stationary individual, and is substantially included in the corresponding area. If the overlapping partial area ci (t) is included, a process of recognizing an individual occupying the specific background difference area as an operating individual may be performed.

  Note that the terms “substantial overlapping part” and “substantial overlapping part region” are used here because a part of a stationary individual flutters under the influence of wind or the like, and noise is mixed in the image. This is because, even if a minute overlapping portion is generated due to a factor, the minute overlapping portion is not handled as an overlapping portion in the processing of the present invention. For example, it is preferable to determine that there is no overlapping part even if there is an overlapping part region consisting of only a few pixels. Also in each of the embodiments and modifications described below, when determining the presence or absence of an overlapping portion, it is preferable to determine the presence or absence of a substantial overlapping portion.

  Here, according to the specific example shown in FIG. 11, the recognition processing by the individual recognition unit 160 will be described in more detail. First, at time t1, the individual recognition unit 160 performs the following recognition process. First, on the original image P (t1), an adult A1 is recognized as an individual occupying the background difference area d1 (t1), and a child B1 is recognized as an individual occupying the background difference area d2 (t1).

  Therefore, for the adult A1, on the overlap confirmation image C (t1), a corresponding region that occupies the same position as the background difference region d1 (t1) and has the same shape is defined, and any of the overlapping partial regions in the corresponding region It is checked whether or not even a part is included. In the example shown in FIG. 11, on the overlap confirmation image C (t1), the corresponding area of the background difference area d1 (t1) and the overlap partial area c1 (t1) do not completely match (the overlap partial area c1 ( Since a part of the limb of t1) is missing), an overlapping partial area c1 (t1) is included as a part in the corresponding area. In other words, an overlapping portion painted in black exists in at least a part of the corresponding region. Therefore, the individual recognition unit 160 recognizes the adult A1 as an operating individual.

  Subsequently, with respect to the child B1, a corresponding region having the same position and occupying the same position as the background difference region d2 (t1) is defined on the overlap confirmation image C (t1), and any overlapping portion is included in the corresponding region. Check whether or not the area is partially included. In the example shown in FIG. 11, on the overlap confirmation image C (t1), the corresponding region of the background difference region d2 (t1) and the overlapped partial region c2 (t1) do not completely match (the overlapped partial region c2 ( Since a part of the limb of t1) is missing), an overlapping partial area c2 (t1) is included as a part in the corresponding area. In other words, an overlapping portion painted in black exists in at least a part of the corresponding region. Therefore, the individual recognition unit 160 recognizes the child B1 as an operating individual.

  Thus, at the time t1, the two sets of individuals A1 and B1 included in the original image P (t1) are both recognized as operating individuals. The detected individual display unit 170 can display the individual detected image Q (t1) -SM as shown in the column of time t1 in FIG. 4 based on such a recognition result.

  At time t2, the individual recognition unit 160 performs the following recognition process. First, on the original image P (t2), the adult A2 is recognized as an individual occupying the background difference area d1 (t2), and the child B2 is recognized as an individual occupying the background difference area d2 (t2).

  Therefore, for the adult A2, on the overlap confirmation image C (t2), a corresponding region that occupies the same position as the background difference region d1 (t2) and has the same shape is defined, and any overlapping partial region is included in the corresponding region. It is checked whether or not even a part is included. In the example shown in FIG. 11, on the overlap confirmation image C (t2), the corresponding region of the background difference region d1 (t2) and the overlap partial region c1 (t2) do not completely match (the overlap partial region c1 ( Since a part of the limb of t2) is missing), an overlapping partial area c1 (t2) is included as a part in the corresponding area. In other words, an overlapping portion painted in black exists in at least a part of the corresponding region. Therefore, the individual recognition unit 160 recognizes the adult A2 as an operating individual.

  Subsequently, with regard to the child B2, a corresponding region having the same shape and occupying the same position as the background difference region d2 (t2) is defined on the overlap confirmation image C (t2), and any overlapping portion is defined in the corresponding region. Check whether or not the area is partially included. In the case of the example shown in FIG. 11, on the overlap confirmation image C (t2), there is no overlapping portion painted in black in the corresponding region of the background difference region d2 (t2). As described above, for the sake of convenience, the outline of the overlapping partial region c2 (t2) is drawn in the figure, but actually, this overlapping partial region c2 (t2) does not exist. Therefore, the overlapping partial area is not included at all in the corresponding area of the background difference area d2 (t2). Therefore, the individual recognition unit 160 recognizes the child B2 as a stationary individual.

  Thus, at time t2, adult A2 is recognized as a moving individual and child B2 is recognized as a stationary individual among the two sets of individuals A2 and B2 included in original image P (t2). . The detected individual display unit 170 can display the individual detected image Q (t2) -SM as shown in the column of time t2 in FIG. 4 based on such a recognition result.

  At time t3, the individual recognition unit 160 performs the following recognition process. First, on the original image P (t3), an adult A3 is recognized as an individual occupying the background difference area d1 (t3), and a child B3 is recognized as an individual occupying the background difference area d2 (t3).

  Therefore, for the adult A3, on the overlap confirmation image C (t3), a corresponding region that occupies the same position as the background difference region d1 (t3) and has the same shape is defined, and any overlapping partial region is included in the corresponding region. It is checked whether or not even a part is included. In the case of the example shown in FIG. 11, on the overlap confirmation image C (t3), there is no overlapping portion painted in black in the corresponding region of the background difference region d1 (t3). As described above, for the sake of convenience, the outlines of the overlapping partial areas c1 (t3) and c2 (t3) are drawn in the figure, but actually these overlapping partial areas c1 (t3) and c2 (t3) exist. The overlap confirmation image C (t3) is an empty image in which substantially all the pixels have a pixel value “0”. Therefore, the overlapping partial area is not included at all in the corresponding area of the background difference area d1 (t3). Therefore, the individual recognition unit 160 recognizes the adult A3 as a stationary individual.

  For exactly the same reason, the individual recognition unit 160 recognizes the child B3 as a stationary individual. Thus, at time t3, the two sets of individuals A3 and B3 included in the original image P (t3) are both recognized as stationary individuals. The detected individual display unit 170 can display the individual detected image Q (t3) -SM as shown in the column of time t3 in FIG. 4 based on such a recognition result.

  Finally, at time t4, the individual recognition unit 160 performs the following recognition process. First, on the original image P (t4), an adult A4 is recognized as an individual occupying the background difference area d1 (t4), and a child B4 is recognized as an individual occupying the background difference area d2 (t4).

  Therefore, for the adult A4, a corresponding region that occupies the same position and has the same shape as the background difference region d1 (t4) is defined on the overlap confirmation image C (t4), and any overlapping partial region is included in the corresponding region. It is checked whether or not even a part is included. In the example shown in FIG. 11, on the overlap confirmation image C (t4), the corresponding area of the background difference area d1 (t4) and the overlap partial area c1 (t4) do not completely match (the overlap partial area c1 ( t4) is actually only the part of the arm painted in black), and the overlapping partial area c1 (t4) is included as a part in the corresponding area. In other words, an overlapping portion painted in black exists in at least a part of the corresponding region. Therefore, the individual recognition unit 160 recognizes the adult A4 as an operating individual.

  Subsequently, with respect to the child B4, a corresponding region having the same position and occupying the same position as the background difference region d2 (t4) is defined on the overlap confirmation image C (t4), and any overlapping portion is included in the corresponding region. Check whether or not the area is partially included. In the case of the example shown in FIG. 11, on the overlap confirmation image C (t4), there is no overlapping portion painted in black in the corresponding region of the background difference region d2 (t4). As described above, for the sake of convenience, the outline of the overlapping partial region c2 (t4) is drawn in the figure, but actually, this overlapping partial region c2 (t4) does not exist. Therefore, the overlapping partial area is not included at all in the corresponding area of the background difference area d2 (t4). Therefore, the individual recognition unit 160 recognizes the child B4 as a stationary individual.

  Thus, at time t4, among the two sets of individuals A4 and B4 included in the original image P (t4), the adult A4 is recognized as a moving individual and the child B4 is recognized as a stationary individual. . The detected individual display unit 170 can display an individual detected image Q (t4) -SM as shown in the column of time t4 in FIG. 4 based on such a recognition result.

  Thus, if the duplication confirmation image C (t) including the duplication partial area indicating the duplication part of the background difference area di (t) and the inter-frame difference area fj (t) is created by the duplication presence / absence determination unit 150, The individual recognizing unit 160 determines whether each individual is a stationary individual or a moving individual based on whether or not there is an overlapping portion in the corresponding region on the overlap confirmation image C (t) for the background difference region corresponding to each individual. Can be judged.

  However, in practice, it is preferable to provide some reference in order to determine whether or not there is an overlapping portion. For example, in the case of the overlap confirmation image C (t4) shown in the lower right of FIG. 11, the overlapped partial region c1 (t4) shown in black is a portion corresponding to one arm of the adult A4. The individual recognition unit 160 recognizes the adult A4 as an operating individual because the overlapping partial region c1 (t4) is included in the corresponding region of the background difference region d1 (t4) indicating the adult A4.

  As in the above example, when an individual is performing an action of raising his hand, the determination that the individual is an operating individual will be appropriate. However, for example, since hair or clothes fluttered in the wind, an inter-frame difference area is generated in the inter-frame difference image F (t). As a result, a small overlapping partial area is present in the overlap confirmation image C (t). What if it happens? In such a case, it is not always appropriate to recognize the individual as a moving individual due to the presence of a minute overlapping portion.

  Therefore, in practice, when the individual recognition unit 160 determines whether or not an overlap portion is included in the corresponding region of the background difference region for a specific individual, the overlap included in the corresponding region is included. When the area of the partial region does not satisfy a predetermined standard, it is preferable to perform processing for determining that the overlapping partial region is not included in the corresponding region. Then, as shown in the overlapping confirmation image C (t4) shown in the lower right of FIG. 11, an overlapping portion (an overlapping portion region c1 (t4) corresponding to the arm) having a certain area is included in the corresponding region. The adult A4 will be recognized as the moving individual, but if the overlapping portion having a minute area that the hair flutters is included in the corresponding region, the overlapping portion region is It is determined that it is not included, and adult A4 is recognized as a stationary individual.

  Actually, as described above, the overlapping partial area in the overlapping confirmation image C (t) is an aggregate of pixels having the pixel value “1”. Therefore, the individual recognition unit 160 refers to the corresponding region on the duplication confirmation image C (t) for a specific background difference region included in the background difference image D (t), and among the pixels in the corresponding region. When the total number of pixels having the pixel value “1” does not satisfy the predetermined standard, the individual occupying the specific background difference area is recognized as a stationary individual, and the pixel value among the pixels in the corresponding area When the total number of pixels having “1” is equal to or greater than a predetermined reference, a process of recognizing an individual occupying the specific background difference area as an operating individual may be performed.

  The total number of pixels serving as a determination criterion is a detection sensitivity parameter that determines whether an individual is detected as a stationary individual or an operating individual. The value of the parameter may be set to an appropriate value as appropriate according to the installation location and application of the apparatus.

<3-7. Operation of Detected Object Display Unit 170>
In the embodiment shown here, the detected individual display unit 170 creates an individual detected image Q (t) by superimposing the stationary individual index s and the moving individual index m on the original image P (t), and this individual detection. It plays the role of displaying the image Q (t) on the display screen. In other words, a process of displaying an individual index indicating that it is a stationary individual or a moving individual on an individual recognized as a stationary individual or a moving individual by the individual recognition unit 160 is executed. For example, in the case of the embodiment shown in FIG. 4, the stationary individual index s is shown as a hatched area with a fine rectangular dot pattern, and the moving individual index m is shown as a hatched area with a hatched mesh pattern.

  As described above, when the detection results of both the stationary individual and the moving individual are displayed on the same display screen, the stationary individual index s and the moving individual index m may be displayed in a visually distinguishable manner. . As in the example shown in FIG. 4, the stationary individual index s consisting of hatching with a fine rectangular dot pattern and the moving individual index m consisting of hatching with a hatched mesh pattern are indices that allow the background to be partially seen through. Even when displayed over P (t), it is possible to visually confirm the contents of the individual located below it.

  Of course, if it is not necessary to visually confirm each individual on the original image P (t), an opaque index may be used. For example, a graphic composed of a solid blue closed region may be used as the stationary individual index s, and a graphic composed of a solid red closed region may be used as the motion individual index m. In this case, the stationary individual index s and the moving individual index m are human-shaped figures whose contours match the contours of the individual individuals, but are located below them because they are composed of opaque solid-coated closed regions. The contents of the individual (for example, a person's face and clothes) cannot be visually confirmed.

  In short, the detected individual display unit 170 is configured by a closed region index in which each individual index is arranged in the vicinity of a region occupied by the target individual and the inside is painted with a predetermined color or a predetermined pattern is formed inside. Then, the individual detection image Q (t) may be created by superimposing the closed region index on the original image.

  If the stationary individual index s and the moving individual index m are displayed simultaneously on the same screen as in the example shown in FIG. 4, the stationary individual index s is arranged in the vicinity of the area occupied by the target individual. , Constituted by a first closed region index in which the inside is painted with a predetermined color or a predetermined pattern is formed inside, and the motion individual index m is arranged in the vicinity of the region occupied by the target individual, What is necessary is just to comprise by the 2nd closed area | region parameter | index with which the predetermined color was painted or the predetermined pattern was formed in the inside.

  At this time, the first closed region index and the second closed region index are closed regions in which different colors are applied or different patterns are formed, and the first image P (t) is displayed in the first region. The individual detection image Q (t) -SM may be created by superimposing the closed region index and the second closed region index. As described above, as the first closed region index and the second closed region index, an opaque solid-colored closed region may be used, or a closed region in which an opaque solid-painted pattern is embedded is used. May be. In this case, it is not possible to visually confirm the contents of the individual located below it (for example, a person's face and clothes).

  However, in practice, it is preferable that the contents (for example, a person's face and clothes) of an individual located on the original image P (t) can be visually confirmed. In that case, the first closed region index and the second closed region index are constituted by closed regions that are painted in a translucent color or partially formed with a gap pattern, and the original image P (t ), When each closed region index is superimposed, the original image P (t) may be displayed through each closed region index. Then, it is possible to intuitively grasp whether each individual is a stationary individual or a moving individual while visually confirming the contents of each individual on the display screen.

  In the individual detection image Q (t) -SM shown in FIG. 4, the stationary individual index s and the moving individual index m are partially formed with a gap pattern (a hatching pattern with fine rectangular dots and a hatched pattern with hatched mesh). This is an example constituted by a closed region index. When the stationary individual index s and the moving individual index m are configured by a closed region index painted in a translucent color, for example, the stationary individual index s is a translucent blue colored region, and the moving individual index m is translucent. A red colored region. In this case, the stationary individual is displayed as if it was covered with blue cellophane, and the moving individual is displayed as if it was covered with red cellophane.

  However, the stationary individual index s and the moving individual index m are used to inform the observer who is observing the display screen of the position of each individual and whether the individual is a stationary individual or a moving individual. In addition to this, various forms of indicators can be used. Hereinafter, modified examples using forms different from the index described so far as the stationary individual index s and the moving individual index m are illustrated in FIGS.

  FIG. 12 is a diagram showing a first modification of the individual detection image Q (t) -SM shown in FIG. The original image P (t) shown on the left side of the figure is exactly the same as that described so far, but the individual detection image Q (t) -SM shown on the right side of the figure has a rectangular shape surrounding the target individual. A stationary individual index s and a moving individual index m are constituted by the frame-shaped index, and these are superimposed on the original image P (t).

  Specifically, the stationary individual indices s1 (t2), s1 (t3), s2 (t3), and s1 (t4) are all in the first frame shape surrounding the target stationary individuals B2, A3, B3, and B4. Each of the motion individual indices m1 (t1), m2 (t1), m1 (t2), and m1 (t4) is a target motion individual. It is configured by a second frame-like index (shown as a solid line frame in the figure) surrounding A1, B1, A2, and A4.

  After all, the individual detection image Q (t) -SM according to the first modification shown in the figure is obtained by adding two sets of rectangular frames on the original image P (t). Will represent individual individuals (adult A and child B). Since the individual rectangular frames move with the movement of the individual individuals, the observer can intuitively identify the individual individuals by looking at the individual detection image Q (t) -SM displayed on the display screen. I can grasp it. Moreover, since the rectangular frame constituting the stationary individual index s is indicated by a broken line and the rectangular frame constituting the moving individual index m is indicated by a solid line, each individual individual enclosed by the rectangular frame is a stationary individual. It is possible to visually recognize whether there is an individual or a moving individual.

  The first modified example shown in FIG. 12 is an example in which each individual index is configured by a rectangular frame, in particular, a circumscribed rectangular frame of the target individual. The circumscribed rectangular frame of each individual can be created as a circumscribed rectangular frame of the background difference area di (t) included in the background difference image D (t) shown on the right in FIG.

  However, the frame index need not necessarily be a rectangular frame. FIG. 13 is a diagram showing a second modification of the individual detection image Q (t) -SM shown in FIG. The original image P (t) shown on the left side of the figure is exactly the same as that described so far, but the individual detection image Q (t) -SM shown on the right side of the figure is an ellipse surrounding the target individual. The stationary individual index s and the moving individual index m are composed of the frame-shaped indices, and these are superimposed on the original image P (t). Since each frame index is an elliptical frame, it is not necessarily in contact with the target individual, but the oval frame constituting the stationary individual index s is indicated by a broken line and the ellipse constituting the moving individual index m The frame is indicated by a solid line, and it is possible to visually recognize whether the individual is surrounded by each elliptical frame, whether it is a stationary individual or a moving individual.

  In the example shown in FIGS. 12 and 13, the first frame index constituting the stationary individual index s and the second frame index constituting the motion individual index m are mutually indicated by a broken line and a solid line. Although it can be distinguished, it may be distinguished by another method as long as it can be visually distinguished. For example, even when a method of distinguishing by line type is adopted, it is also possible to distinguish by a broken line and an alternate long and short dash line. Also, the blue frame and the red frame may be distinguished by color, or the thick line and the thin line may be distinguished by thickness. Alternatively, the stationary individual indicator s may be a rectangular frame, and the moving individual indicator m may be an elliptical frame.

  In short, when the first frame-shaped index is used as the stationary individual index s and the second frame-shaped index is used as the moving individual index m, the first frame-shaped index and the second frame-shaped index are: What is necessary is just to make it comprise at least 1 among a color, a shape, and thickness by a mutually different frame line. The example shown in FIG. 12 and FIG. 13 is an example in which the individual detection image Q (t) -SM is created by superimposing such two types of frame-shaped indexes on the original image P (t). Note that the frame index does not necessarily need to be a figure that encloses the entire target individual, and may be a graphic that surrounds only a part of the target individual (for example, the upper body part). Further, as in the example shown in the figure, adjacent frame indexes may be partially overlapped and displayed.

  On the other hand, FIG. 14 is a diagram showing a third modification of the individual detection image Q (t) -SM shown in FIG. The original image P (t) shown on the left side of the figure is exactly the same as that described so far, but the individual detection image Q (t) -SM shown on the right side of the figure shows the contour of the target individual. A stationary individual index s and a moving individual index m are constituted by the contour index shown, and these are superimposed on the original image P (t).

  Specifically, the stationary individual indices s1 (t2), s1 (t3), s2 (t3), and s1 (t4) are all contour indices indicating the contour lines of the stationary individuals B2, A3, B3, and B4. (Shown as a dashed outline in the figure), and the motion individual indices m1 (t1), m2 (t1), m1 (t2), and m1 (t4) are all target motion individuals It is comprised by the outline parameter | index (it shows as a solid outline in the figure) which shows the outline of A1, B1, A2, A4.

  Eventually, the individual detection image Q (t) -SM according to the third modified example shown in FIG. 14 adds contour lines to each of the two sets of individuals appearing on the original image P (t). Individual contour lines will indicate individual individuals (adult A and child B). Since the individual contour lines move with the movement of the individual individuals, the observer can intuitively identify the individual individuals by looking at the individual detection image Q (t) -SM displayed on the display screen. I can grasp it. Moreover, since the contour line constituting the stationary individual index s is indicated by a broken line and the contour line constituting the motion individual index m is indicated by a solid line, the individual individual surrounded by each contour line is a stationary individual. It is possible to visually recognize whether there is an individual or a moving individual.

  In the example shown in FIG. 14, the first contour index constituting the stationary individual index s and the second contour index constituting the moving individual index m can be distinguished from each other by a broken line and a solid line. However, as long as it is visually distinguishable, it may be distinguished by another method. Each contour index can be created by extracting the contour line of the background difference region di (t) included in the background difference image D (t) shown on the right in FIG. It is determined by the contour shape. Therefore, unlike the frame-shaped index described above, the first contour index constituting the stationary individual index s and the second contour index constituting the moving individual index m cannot be distinguished by changing the shape. It is possible to distinguish between the two by changing the line type, color, and thickness of the outline.

  Eventually, the detected individual display unit 170 configures the stationary individual index s by the first contour index indicating the contour of the target stationary individual, and the moving individual index m indicates the contour of the target moving individual. The first contour index and the second contour index are configured by a contour line that is different from each other in at least one of line type, color, and thickness. The individual detection image Q (t) -SM may be created by superimposing the first contour index and the second contour index on (t).

  FIG. 15 is a diagram showing a modification of the individual detection image Q (t) -S shown in FIG. The original image P (t) shown on the left of the figure is exactly the same as that described so far, but the individual detection image Q (t) -S shown on the right of the figure shows a stationary individual to be detected. The image is shown using a mask image having an aperture window. On the other hand, FIG. 16 is a diagram showing a modification of the individual detection image Q (t) -M shown in FIG. The original image P (t) shown on the left side of the figure is exactly the same as that described so far, but the individual detection image Q (t) -M shown on the right side of the figure shows the motion individual to be detected. The image is shown using a mask image having an aperture window.

  In the modification shown in FIGS. 15 and 16, the detected individual display unit 170 has a stationary individual index s by a mask image in which an opening window is arranged in an individual part to be detected and a part other than the opening window is covered. Alternatively, the motion individual index m is configured, and the individual detection image Q (t) -S or Q (t) -M is created by superimposing the mask image on the original image P (t).

  For example, FIG. 15 is an example of an individual detection image Q (t) -S showing only the detection result of a stationary individual. The individual detection image Q (t1) -S shown in the column of time t1 is a mask image (shown as a mesh pattern in the figure, but actually, for example, it may be displayed as a gray region). It is in a covered state. In this case, the mask image covering the entire surface is the stationary individual index s. Since the original image P (t1) is entirely covered with the mask image constituting the stationary individual index s, the entire display screen is displayed in gray. This indicates that no stationary individual is detected at present.

  On the other hand, the individual detection image Q (t2) -S shown in the column of time t2 is a stationary individual consisting of a mask image in which an opening window is arranged in the region of the child B2 that is a stationary individual and the other region is covered. This is an image obtained by superimposing the index s on the original image P (t2). Since the underlying original image P (t2) can be observed only in the area where the opening window is formed, when the display screen is observed, it appears that only the child B2 that is a stationary individual is arranged on a gray background.

  Further, the individual detection image Q (t3) -S shown in the column of time t3 is based on a mask image in which aperture windows are arranged in the areas of adults A3 and children B3, which are stationary individuals, and the other areas are covered. This is an image obtained by superimposing a stationary individual index s on the original image P (t3). Since the underlying original image P (t3) can be observed only in the area where the opening window is formed, when the display screen is observed, it appears that the adult A3 and the child B3, which are stationary individuals, are arranged on a gray background. .

  The individual detection image Q (t4) -S shown in the column of time t4 is a stationary individual consisting of a mask image in which an opening window is arranged in the region of the child B4 that is a stationary individual and the other region is covered. This is an image obtained by superimposing the index s on the original image P (t4). Since the underlying original image P (t4) can be observed only in the area where the opening window is formed, when the display screen is observed, it appears that only the child B4, which is a stationary individual, is arranged on a gray background.

  As described above, in the individual detection image Q (t) -S shown in FIG. 15, only the stationary individual is displayed on the gray background, so that it is most suitable for the purpose of clearly grasping only the stationary individual. .

  On the other hand, FIG. 16 is an example of the individual detection image Q (t) -M showing only the detection result of the moving individual. The individual detection image Q (t1) -M shown in the column of time t1 is an operation composed of a mask image in which aperture windows are arranged in the areas of the adult A1 and the child B1 that are the operating individuals and the other areas are covered. This is an image obtained by superimposing the individual index m on the original image P (t1). Since the underlying original image P (t1) can be observed only in the area where the opening window is formed, when the display screen is observed, it appears that the adult A1 and the child B1 that are the moving individuals are arranged on the gray background. .

  On the other hand, the individual detection image Q (t2) -M shown in the column of time t2 is an operation individual consisting of a mask image in which an opening window is arranged in the region of the adult A2 that is the operation individual and the other region is covered. This is an image obtained by superimposing the index m on the original image P (t2). Since the underlying original image P (t2) can be observed only in the area where the opening window is formed, when the display screen is observed, it appears that only the adult A2 that is an operating individual is arranged on a gray background.

  The individual detection image Q (t3) -M shown in the column of time t3 is in a state where the entire surface is covered with the mask image. In this case, the mask image covering the entire surface is the operating body index m. Since the original image P (t3) is entirely covered by the mask image constituting the motion individual index m, the entire display screen is displayed in gray. This indicates that no moving object is detected at present.

  The individual detection image Q (t4) -M shown in the column of time t4 is an operation individual composed of a mask image in which an aperture window is arranged in the region of the adult A4 that is the operation individual and the other region is covered. This is an image obtained by superimposing the index m on the original image P (t4). Since the underlying original image P (t4) can be observed only in the area where the opening window is formed, when the display screen is observed, it appears that only the adult A4, which is an operating individual, is arranged on a gray background.

  As described above, in the individual detection image Q (t) -M shown in FIG. 16, only the motion individual is displayed on the gray background, so that it is most suitable for the purpose of clearly grasping only the motion individual. .

  It should be noted that the mask image constituting the stationary individual index s shown in FIG. 15 and the mask image constituting the motion individual index m shown in FIG. 16 are predetermined ones included in the background difference image D (t) shown on the right in FIG. It can be created by forming an opening window using the contour line of the background difference area di (t).

  As shown in the block diagram of FIG. 1, the detected individual display unit 170 in this embodiment is given the original image P (t) input by the original image input unit 110. Therefore, the detected individual display unit 170 can display the original image P (t) as it is on the display screen. Further, since the individual recognition unit 160 includes both the stationary individual recognition unit 161 and the moving individual recognition unit 162, the detected individual display unit 170 displays the original image P (t) as illustrated in FIG. It is possible to create an individual detection image Q (t) -S by superimposing a stationary individual index s and display it on the display screen, and as illustrated in FIG. It is also possible to create an individual detection image Q (t) -M by superimposing the motion individual index m on t) and display it on the display screen. Furthermore, it is also possible to create an individual detection image Q (t) -SM as illustrated in FIG. 4 using both the stationary individual index s and the moving individual index m and display it on the display screen. .

  Therefore, in practice, the four display modes described above are set to separate display modes, and the display modes can be switched as appropriate according to the request of the user (for example, a monitor). Is preferred. FIG. 17 is a diagram showing four display modes by the detected individual display unit 170. In FIG. 17, for convenience of explanation, an image obtained at time t <b> 2 is illustrated by using the individual indicator composed of the rectangular frame shown in FIG. 12 as the stationary individual indicator s and the moving individual indicator m.

  In the case of the embodiment shown in FIG. 17, the detected individual display unit 170 has a display function based on four display modes: an original image display mode, an all-individual display mode, a stationary individual display mode, and an operating individual display mode. In response to an instruction from the user, the display can be switched to an arbitrary display mode.

  As shown in FIG. 17A, the original image display mode is a mode in which only the original image P (t) is displayed on the display screen, and is a mode in which an image captured by the fixed point camera is displayed as it is. In the illustrated example, an original image P (t2) including an adult A2 and a child B2 as individuals is displayed. Of course, the adult A2 and the child B2 are displayed in a state of being melted in the background image, and the observer recognizes them as individual individuals in the brain, but as visual information, The adult A2 and the child B2 are not presented as individual individuals, and of course, information regarding whether they are stationary individuals or moving individuals is not presented.

  On the other hand, in the all-individual display mode, as shown in FIG. 17B, the individual detection image Q (t) in which the stationary individual index s, the motion individual index m, and the original image P (t) are superimposed on the display screen. -Mode for displaying SM. In the illustrated example, an original image P (t2) including an adult A2 and a child B2 as individuals is displayed, and a moving individual index m1 (t2) composed of a solid rectangular frame and a stationary individual index s1 composed of a dashed rectangular frame (T2) is displayed. The monitor can visually recognize the adult A2 as a moving individual and the child B2 as a stationary individual.

  In the stationary individual display mode, as shown in FIG. 17C, an individual detection image Q (t) -S in which the stationary individual index s and the original image P (t) are superimposed on the display screen is displayed. As shown in FIG. 17 (d), the motion individual display mode is an individual detection image Q (t) -M obtained by superimposing the motion individual index m and the original image P (t) on the display screen. This is the display mode. In the former mode, the monitor can visually recognize the child B2 as a stationary individual, and in the latter mode, the monitor can visually recognize the adult A2 as an operating individual. When it is desired to intuitively recognize only a stationary individual, it is preferable to switch to the former stationary individual display mode, and when it is desired to intuitively recognize only a moving individual, it is preferable to switch to the latter moving individual display mode.

  In this way, if the mode switching function as shown in FIG. 17 is provided, the user can select and use the optimum display form according to the monitoring application, which can improve convenience. it can.

<<< §4. Specific method of image calculation processing >>>
In the individual detection apparatus 100 shown in FIG. 1, a background difference image D (t) is created by the background difference image creation unit 130, and an inter-frame difference image F (t) is created by the inter-frame difference image creation unit 140. The duplication confirmation image C (t) is created by the presence / absence determination unit 150. The process of creating each image with each of these components is actually executed by image calculation processing by a computer. Here, a specific method of the image calculation process will be described.

  FIG. 18 is a diagram illustrating a general procedure of processing for creating a binary difference image W for two color images U and V and creating labeling information for the difference region. Here, a case is considered where two color images U and V as shown are given. Each color image is composed of an array of a large number of pixels having pixel values of the three primary colors R, G, and B.

  In the procedure shown here, these color images U and V are first converted into monochrome images and converted into luminance images U ′ and V ′. Specifically, as shown in the drawing, the pixel value of the pixel of interest arranged at an arbitrary position on the color image U is (Ru, Gu, Bu), and the pixel value of the pixel at the same position on the luminance image U ′ ( (Luminance value) Yu is calculated | required by the type | formula "Yu = 0.299Ru + 0.587Gu + 0.114Bu". Similarly, assuming that the pixel value of the pixel of interest arranged at an arbitrary position on the color image V is (Rv, Gv, Bv), the pixel value (luminance value) Yv of the pixel at the same position on the luminance image V ′ is It calculates | requires by the type | formula "Yv = 0.299Rv + 0.587Gv + 0.114Bv".

  When the luminance image U ′ composed of a set of pixels having the luminance value Yu and the luminance image V ′ composed of a set of pixels having the luminance value Yv are thus obtained, the luminance images U ′ and V ′ are identical. The difference image W ′ is created by taking the difference in the luminance values of the pixels at the position. That is, the pixel value (difference value) ΔY of an arbitrary pixel constituting the difference image W ′ is equal to the luminance value Yu of the pixel at the same position on the luminance image U ′ and the pixel at the same position on the luminance image V ′. Is obtained as the absolute value of the difference from the luminance value Yv. In other words, ΔY is obtained by an expression “ΔY = | Yu−Yv |”.

  Subsequently, the binarized difference image W is created by binarizing the difference image W ′. Specifically, a process is performed in which a predetermined threshold value is set in advance, a pixel value “1” is given to a pixel in which ΔY is equal to or greater than the threshold value, and a pixel value “0” is given to a pixel in which ΔY is less than the threshold value. Just do it. The binarized difference image W thus obtained is an aggregate of pixels having the pixel value “1” or “0”, and the continuous region of the pixels having the pixel value “1” is the original color image U, When V is compared, it is an area where the difference in pixel values is equal to or greater than a predetermined reference, that is, a difference area where a significant difference is recognized.

  The diagram shown in the lower left of FIG. 18 is a diagram illustrating an example of the pixel configuration of the binarized difference image W obtained in this way. In this figure, hatched pixels are pixels having a pixel value “1”, and the remaining white pixels are pixels having a pixel value “0”. As indicated by a bold line in the figure, two sets of difference areas w1 and w2 are formed by continuous areas of pixels having a pixel value “1”. These difference areas w1 and w2 are a background difference area di (t) shown as a black area in the background difference image D (t) in FIG. 5 and a black area in the inter-frame difference image F (t) in FIG. Corresponds to the inter-frame difference region fj (t) shown as.

  However, the difference areas w1 and w2 shown in the figure can be grasped as a group area of a group of pixels each having a pixel value “1” when observed by a human, but as data handled by a computer, There is still no concept of difference areas w1 and w2. Therefore, in order to handle the difference areas w1 and w2 on the computer, it is necessary to execute labeling processing to create labeling information L as shown in the lower right of FIG. The labeling information L is obtained by assigning a predetermined difference area code to each individual difference area on the difference image, and then specifying each difference area code, pixel specifying information for specifying a pixel included in the corresponding difference area, That is, information indicating the relationship.

  In the case of the illustrated example, the labeling information L is created in the form of a table, and two sets of codes w1 and w2 are accommodated in the difference area code column. These codes are given to the two sets of difference areas w1 and w2 indicated by bold lines in the difference image shown in the lower left of FIG. On the other hand, in the column of pixel specifying information, the coordinate value of each pixel is stored as information for specifying each pixel constituting the difference area. For example, the coordinate values (1, 3), (2, 2), (2, 3), (3, 1), (3, 2) stored in the pixel specifying information column corresponding to the difference area code w1 , (3, 3), (4, 2), (4, 3), (4, 4), (4, 5) are the coordinates of 10 pixels constituting the difference area w1 shown in the lower left of FIG. Value.

  As described above, the labeling information L is information obtained by labeling a specific pixel with a specific difference area code, and the individual difference areas formed on the difference image can be specified by the labeling information L. . Such labeling information L focuses on one pixel having a pixel value “1” on the difference image, identifies an adjacent pixel having the same pixel value “1” for the target pixel, and further It can be created by a method of repeatedly executing the same processing with the adjacent pixel as the target pixel. Since the labeling information L creation processing algorithm is a known technique, a detailed description thereof is omitted here.

  As described above, the basic method of creating the binarized difference image W for the general color images U and V shown in FIG. The background difference image creation unit 130 and the inter-frame difference image creation unit 140 in the apparatus of FIG. 1 create the background difference image D (t) and the inter-frame difference image F (t) based on such a basic method. Further, by performing the above-described labeling process, it is possible to create the labeling information L (D (t)) for the background difference area and the labeling information L (F (t)) for the inter-frame difference area.

  After all, when the procedure shown in FIG. 18 is introduced into the image processing performed by the individual detection apparatus 100 shown in FIG. 1, the following processing may be performed in each component. First, the original image input unit 110 inputs a color image made up of a set of pixels having pixel values of the three primary colors RGB as an original image P (t), and this is input to a pixel having a pixel value indicating luminance Y. A process of converting to a luminance image composed of an aggregate is performed. That is, when the original image P (t) obtained from the fixed point camera is a color image, the original image input unit 110 converts the image into a monochrome image and generates a luminance image. The background difference image creation unit 130 and the inter-frame difference image creation The luminance image is given to the unit 140 as the original image P (t). In this case, the background image storage unit 120 stores a monochrome background image BG made up of a collection of pixels having pixel values indicating the luminance Y.

  The background difference image creation unit 130 then converts the monochrome original image P (t) (luminance image) given from the original image input unit 110 and the monochrome background image stored in the background image storage unit 120. For BG, a process for obtaining an absolute value ΔY of a difference between pixel values (luminance values Y) of the corresponding pixels is performed. A set of pixels having ΔY as a pixel value thus obtained becomes a difference image corresponding to the difference image W ′ shown in FIG. Therefore, the background difference image creation unit 130 temporarily stores the difference image and binarizes it using a predetermined threshold value, thereby binarizing the difference image (the binarized difference image W shown in FIG. 18). And the binarized differential image is output as a background differential image D (t).

  More specifically, the background difference image creation unit 130 gives a pixel value “1” to a pixel whose absolute value ΔY of the luminance value difference is equal to or larger than the threshold value, and the absolute value ΔY of the luminance value difference is the threshold value. By giving a pixel value “0” to the pixels that are less than this, the background difference image D (t) is created. Here, the continuous area of pixels having the pixel value “1” is the background difference area di (t). Therefore, the background difference image creation unit 130 assigns a background difference area code to each background difference area di (t), and specifies a pixel included in the background difference area corresponding to each background difference area code. Labeling information L (D (t)) for the background difference area indicating information is created.

  On the other hand, the inter-frame difference image creation unit 140 corresponds to each of the two monochrome original images P (t) (two luminance images in units of frames) given from the original image input unit 110 with a time difference. The process of obtaining the absolute value ΔY of the difference between the pixel values (luminance value Y) of the. A set of pixels having ΔY as a pixel value thus obtained becomes a difference image corresponding to the difference image W ′ shown in FIG. Therefore, the inter-frame difference image creation unit 140 temporarily stores the difference image and binarizes it using a predetermined threshold value, thereby binarizing the difference image (the binary difference image shown in FIG. 18). Image corresponding to W) is generated, and the binarized difference image is output as an inter-frame difference image F (t).

  More specifically, the inter-frame difference image creation unit 140 gives a pixel value “1” for pixels whose absolute value ΔY of luminance value difference is equal to or greater than a threshold value, and the absolute value ΔY of luminance value difference is An inter-frame difference image F (t) is created by giving a pixel value “0” to pixels that are less than the threshold. Here, the continuous region of pixels having the pixel value “1” is the inter-frame difference region fj (t). Therefore, the inter-frame difference image creating unit 140 assigns an inter-frame difference area code for each inter-frame difference area fj (t), and the pixels included in the inter-frame difference area corresponding to the individual inter-frame difference area code. The labeling information L (F (t)) of the inter-frame difference area indicating the pixel specifying information for specifying is generated.

  The overlap determination unit 150 uses the background difference area labeling information L (D (t)) and the inter-frame difference area labeling information L (F (t)) created in this way to generate a background difference image D ( For each background difference area di (t) in t), it is possible to perform overlap presence / absence determination processing for determining whether or not there is an overlapping portion with respect to the inter-frame difference area fj (t) in the inter-frame difference image F (t). .

  FIG. 19 is a diagram illustrating the principle of the overlap presence / absence determination process by the overlap presence / absence determination unit 150. Here, the basic principle of the process of creating the duplication confirmation image C (t) based on the background difference image D (t) and the inter-frame difference image F (t) created at time t will be described.

  The background difference image D (t) shown in the upper left of FIG. 19 is an image created by the background difference image creation unit 130, and the difference calculation is performed between the original image P (t) and the background image BG. To obtain. According to the above-described procedure, the background difference image D (t) is a binary difference image composed of a set of pixels having the pixel value “0” or “1”. The illustrated background difference image D (t) includes two sets of background difference areas d1 (t) and d2 (t) as shown by hatching. These areas are composed of a collection of pixels having a pixel value “1”. In the figure, for convenience of explanation, two pixels a (xa, ya) and b (xb, yb) are illustrated as pixels constituting the background difference area d1 (t). One pixel c (xc, yc) is illustrated as a constituent pixel (the coordinate value of each pixel is in parentheses).

  As described above, the background difference image creation unit 130, for each background difference area d1 (t), d2 (t), labeling information L (D (t)) including pixel specifying information for specifying pixels included in the area. Create The background difference area labeling information L (D (t)) shown in the upper right of FIG. 19 is the labeling information thus created. For example, for the background difference area d1 (t), coordinate values of the pixel a (xa, ya), the pixel b (xb, yb), etc. are accommodated as pixel specifying information, and for the background difference area d2 (t) , A coordinate value such as a pixel c (xc, yc) is stored as pixel specifying information.

  On the other hand, the inter-frame difference image F (t) shown in the middle left of FIG. 19 is an image created by the inter-frame difference image creation unit 140, and the original image P (t) and the immediately preceding original image P It is obtained by performing a difference calculation with (t-1). According to the above-described procedure, this inter-frame difference image F (t) is also a binarized difference image composed of a set of pixels having the pixel value “0” or “1”. The illustrated inter-frame difference image F (t) includes three sets of inter-frame difference areas f1 (t), f2 (t), and f3 (t) as shown by hatching. These areas are composed of a collection of pixels having a pixel value “1”.

  Here, for convenience of explanation, the same positions as the three pixels a (xa, ya), b (xb, yb), and c (xc, yc) illustrated in the background difference image D (t) shown in the upper part of the figure. A certain pixel is illustrated using the same symbol in the inter-frame difference image F (t) shown in the middle of the figure. However, in the case of the inter-frame difference image F (t), the pixel a (xa, ya) has a pixel value “1” because it is a pixel in the inter-frame difference area f1 (t) shown by hatching. The pixels b (xb, yb) and c (xc, yc) are pixels outside the inter-frame difference area, and thus have a pixel value “0”.

  As described above, the inter-frame difference image creation unit 140 performs labeling information including pixel specifying information that specifies pixels included in the inter-frame difference areas f1 (t), f2 (t), and f3 (t). L (F (t)) is created. The labeling information L (F (t)) for the inter-frame difference area shown in the middle right of FIG. 19 is the labeling information created in this way. For example, for the inter-frame difference area f1 (t), coordinate values including the pixel a (xa, ya) are accommodated as pixel specifying information.

  The basic role of the duplication presence / absence determination unit 150 is to determine the inter-frame difference in the inter-frame difference image F (t) for the individual background difference areas d1 (t) and d2 (t) in the background difference image D (t). This is to determine whether or not there is an overlapping portion with respect to the regions f1 (t), f2 (t), and f3 (t). Such a determination can be made using the labeling information L (D (t)), L (F (t)).

  For example, for the background difference area d1 (t), when it is determined whether or not there is an overlapping portion with respect to the inter-frame difference areas f1 (t), f2 (t), and f3 (t), the labeling information L (D (t)) A coordinate value that is the same as the coordinate value stored as the pixel specifying information of d1 (t) is stored as the pixel specifying information of f1 (t) to f3 (t) in the labeling information L (F (t)). What is necessary is just to determine whether it exists. In the case of the illustrated example, the same coordinate value (xa, ya) as the coordinate value (xa, ya) for d1 (t) is accommodated as pixel specifying information for f1 (t). For this reason, it is determined that the background difference area d1 (t) has an overlapping portion with respect to the inter-frame difference area, and the individual occupying the background difference area d1 (t) is recognized as an operating individual.

  In short, as a basic principle, when the overlapping presence / absence determining unit 150 determines the presence / absence of an overlapping portion for a predetermined target background difference region d1 (t), pixel identification in the labeling information of the target background difference region d1 (t) is specified. For each pixel of interest specified by the information, it is checked whether there is a pixel at the same position in the pixel specifying information in the labeling information of the inter-frame difference areas f1 (t) to f3 (t). It may be determined that there is an overlapping portion.

  The duplication presence / absence judging unit 150 shown in FIG. 1 has a function of creating a duplication confirmation image C (t) as exemplified in FIGS. 7 to 10 in order to make a judgment based on such a basic principle. . Specifically, the duplication presence / absence determination unit 150 performs a logical product operation on the corresponding pixels on the background difference image D (t) and the inter-frame difference image F (t), thereby obtaining a pixel value “1” or A duplication confirmation image C (t), which is a binary image composed of a collection of pixels having “0”, is created.

  For example, in the example shown in FIG. 19, the pixel a (xa, ya) on the upper left background difference image D (t) and the pixel a (xa, ya) on the middle left frame difference image F (t) As a result of the logical product operation, the pixel a (xa, ya) on the left overlapping confirmation image C (t) is obtained. In this case, since the pixel values of the two sets of pixels to be calculated are both “1” (in the hatching area), the pixel value of the pixel a (xa, ya) on the overlap confirmation image C (t) is The logical product is “1”.

  On the other hand, for the pixels b (xb, yb) and c (xc, yc), the pixels on the background difference image D (t) have the pixel value “1” (in the hatching area), but the interframe difference Since the pixels on the image F (t) have the pixel value “0” (in the white background region), the pixel values of the pixels b (xb, yb) and c (xc, yc) on the overlap confirmation image C (t) are The logical product is “0”.

  In the overlap confirmation image C (t) on the lower left of FIG. 19, the pixel a (xa, ya) to which the pixel value “1” is given is shown as a black pixel, and the pixel b (xb) to which the pixel value “0” is given. , Yb), c (xc, yc) are shown as white pixels. Also, the corresponding areas corresponding to the background difference areas d1 (t) and d2 (t) on the background difference image D (t) are indicated by broken lines. In the figure, only three pixels a (xa, ya), b (xb, yb), and c (xc, yc) are shown. However, in reality, the pixel value “ Either “0” or “1” is given.

  The individual recognition unit 160 recognizes whether the individual occupying each of the background difference areas d1 (t) and d2 (t) is an operating individual or a stationary individual by using the duplication confirmation image C (t). can do. That is, a corresponding area of each background difference area d1 (t), d2 (t) is defined on the overlap confirmation image C (t) (area indicated by a broken line in the drawing), and the pixel value “1” is included in this corresponding area. When there is only a pixel having a pixel value (black pixel), and a pixel having a pixel value “1” (black pixel) and a pixel having a pixel value “0” (white pixel) are mixed If an individual occupying the background difference area is recognized as an operating individual and only a pixel (white pixel) having a pixel value “0” exists, the individual occupying the background difference area is recognized as a stationary individual. Good.

  In the case of the illustrated example, black pixels and white pixels are mixed in a corresponding region (region indicated by a broken line in the drawing) of the background difference region d1 (t) defined on the overlap confirmation image C (t). Therefore, an individual occupying the background difference area d1 (t) is recognized as an operating individual. On the other hand, since only a white pixel exists in the corresponding area (area indicated by a broken line in the drawing) of the background difference area d2 (t) defined on the overlap confirmation image C (t), the background difference area d2 ( Individuals occupying t) will be recognized as stationary individuals.

  Individual regions having a human-shaped outline in each of the overlap confirmation images C (t1) to C (t4) shown on the right in FIG. 11 are the background difference images D (t1) to D (t4) shown on the right in FIG. ) Included in the background difference region d1 (t1), d2 (t1),..., And is indicated by a broken line in the overlap confirmation image C (t) shown on the lower left of FIG. It corresponds to an area. In each of the duplication confirmation images C (t1) to C (t4) shown on the right side of FIG. 11, when a black part (overlapping part) is partially included in the outline of the humanoid (that is, the outline of the humanoid) When the inside is only a black part, and when a black part and a white part are mixed in the outline of a human figure), the individual corresponding to the human figure is recognized as an action individual, and the outline of the human figure is As described above, in the case of only the white part (non-overlapping part), the individual corresponding to the human type is recognized as a stationary individual.

  However, as described in §3-6, since the hair and clothes fluttered in the wind, an inter-frame difference area is generated in the inter-frame difference image F (t). There may be a case where a minute overlapping partial region (black portion) is generated in the image C (t). Assuming such a case, in the overlap confirmation image C (t) on the lower left of FIG. 19, a black pixel (pixel) in the corresponding region (region indicated by a broken line in the drawing) of the background difference region d1 (t). Even if there is a pixel having the value “1”, if the number of black pixels is very small, it is not preferable to treat this as a significant overlapping portion. In other words, in the example shown in the figure, even when black pixels exist in the corresponding region of the background difference region d1 (t) on the overlap confirmation image C (t), the number does not satisfy a predetermined criterion. In this case, the individual indicating the background difference area d1 (t) should not be a moving individual but a stationary individual.

  In consideration of such points, in practice, the individual recognition unit 160 performs a specific background difference area d1 (t) included in the background difference image D (t) on the overlap confirmation image C (t). When the total number of pixels (black pixels) having the pixel value “1” among the pixels in the corresponding area is not equal to a predetermined standard with reference to the corresponding area, the specific background difference area d1 (t ) Is recognized as a stationary individual, and when the total number of pixels (black pixels) having a pixel value “1” among the pixels in the corresponding region is equal to or greater than a predetermined reference, the specific background It is preferable to perform processing for recognizing an individual occupying the difference area d1 (t) as an operating individual.

<<< §5. Practical processing procedure by computer >>>
In the individual detection apparatus 100 shown in FIG. 1, after the background difference image creation unit 130 creates the background difference image D (t) and the interframe difference image creation unit 140 creates the interframe difference image F (t). Then, a process for determining the presence or absence of an overlapping part (a process for creating an overlapping confirmation image C (t)) is executed in the duplication presence / absence determination unit 150, and then, in the individual recognition unit 160, based on the determination result, a stationary individual Further, a process for recognizing a moving individual is executed, and further, a process for displaying a stationary individual index s indicating a stationary individual and a moving individual index m indicating a moving individual on the display screen is executed by the detected individual display unit 170. Here, a practical processing procedure suitable for executing such a series of processing by a computer program will be described.

<5-1. Procedure for creating stationary individual index s>
FIG. 20 is a flowchart showing a procedure for creating the stationary individual index s by the cooperative operation of the duplication presence / absence determination unit 150, stationary individual recognition unit 161, and detected individual display unit 170 of the apparatus of FIG. Here, the background difference image creation unit 130 creates background difference images D (t1) to D (t4) as shown in FIG. 5, and the interframe difference image creation unit 140 creates an interframe difference as shown in FIG. Assuming that the difference images F (t1) to F (t4) are created, the procedure executed by the duplication presence / absence determination unit 150, the stationary individual recognition unit 161, and the detected individual display unit 170 based on these two difference images will be described. To do.

  First, in step S11, the stationary individual index number k is set to an initial value 0. The stationary individual index number k is a serial number assigned to each recognized stationary individual. Actually, the stationary individual first recognized is k = 1 in step S18, so the stationary individual index number k = 1 is assigned.

  The subsequent step S12 is a loop process in which the procedure up to step S20 is repeatedly executed from i = 1 to Imax. Here, i indicates the background difference area number, the initial value is 1, and the maximum value is Imax. Therefore, the loop process from step S12 to step S20 is a loop process for the background difference area. For example, in the case of the example shown in FIG. 19, since the background difference image D (t) includes two sets of background difference areas d1 (t) and d2 (t), Imax = 2. Therefore, a process that focuses on the background difference area d1 (t) (a process that sets i = 1) and a process that focuses on the background difference area d2 (t) (a process that sets i = 2) are executed.

  The next step S13 is a loop process in which the procedure up to step S17 is repeatedly executed from j = 1 to Jmax. Here, j indicates the inter-frame difference area number, the initial value is 1, and the maximum value is Jmax. Therefore, the loop process from step S13 to step S17 is a loop process for the inter-frame difference area. For example, in the example shown in FIG. 19, the inter-frame difference image F (t) includes three sets of inter-frame difference areas f1 (t), f2 (t), and f3 (t). = 3. Therefore, a process that focuses on the inter-frame difference area f1 (t) (a process that sets j = 1), a process that focuses on the inter-frame difference area f2 (t) (a process that sets j = 2), and the inter-frame difference area A process focusing on f3 (t) (a process of setting j = 3) is executed.

  In step S14, the AND image of each pixel in the i-th background difference area di (t) and the pixel in the same position in the j-th inter-frame difference area fj (t) is obtained. Processing for creating IMG (AND) is executed. For example, in the first process executed when i = 1 and j = 1, for each pixel in the first background difference area d1 (t), in the first inter-frame difference area f1 (t) A process of creating an AND image IMG (AND) obtained by ANDing the pixels at the same position of is performed. In the case of the example shown in FIG. 19, this is a process for obtaining a logical product image of the area d1 (t) and the area f1 (t), and is located in an overlapping portion of the areas d1 (t) and f1 (t). An AND image IMG (AND) is obtained in which the pixel to be processed is a black pixel (pixel value “1”) and the other pixels are white pixels (pixel value “0”).

  In the subsequent step S15, a total logical sum VAL (OR) obtained by OR is obtained for all the pixels in the logical product image IMG (AND) obtained in step S14. That is, the logical product image IMG (AND) obtained in step S14 is an aggregate of pixels having either a pixel value “0” (for white pixels) or a pixel value “1” (for black pixels). The total logical sum VAL (OR) is calculated as the logical sum of the pixel values of all the pixels constituting the binary image. This total logical sum VAL (OR) is a logical value taking either “0” or “1”.

  In other words, the value of the total logical sum VAL (OR) is set to “0” only when all the pixels in the logical product image IMG (AND) are pixels (white pixels) having the pixel value “0”. Thus, when any one of the pixels (black pixels) having the pixel value “1” is included, “1” is obtained. Here, since the black pixel is a pixel indicating an overlapping portion of the regions d1 (t) and f1 (t), if the total logical sum VAL (OR) = 0, the regions d1 (t) and f1 ( t) indicates that there is no overlapping portion. If the total logical sum VAL (OR) = 1, it indicates that there is an overlapping portion between the regions d1 (t) and f1 (t).

  In practice, as described in §3-6, VAL (OR) is used as a VAL (OR) in order to avoid that a stationary person is recognized as an operating individual because of hair or clothes flying in the wind. If the number of black pixels is not a mathematically complete logical sum and the number of black pixels is less than the predetermined reference, VAL (OR) = 0 and the number of black pixels is equal to or greater than the predetermined reference. Preferably defines a function such that VAL (OR) = 1.

  In subsequent step S16, it is determined whether or not there is an overlapping portion based on the total logical sum VAL (OR). First, when the total logical sum VAL (OR) = 1, it is determined that “there is an overlapping portion” between the areas d1 (t) and f1 (t), and the process exits the loop of j and branches to step S20. Is done. This is because in the case of “there is an overlapping portion”, the individual occupying the background difference area d1 (t) is recognized as a moving individual, and thus it is not necessary to create the stationary individual index s. When such a recognition is made, the process for the background difference area d1 (t) performed with i = 1 ends. That is, for i = 1, it is not necessary to perform processing for j = 2, 3 (presence / absence of overlapping portions for regions d1 (t) and f2 (t), and regions d1 (t) and f3 (t)). The individual that occupies the background difference area d1 (t) can be recognized as an operating individual without having to check for the presence or absence of an overlapping portion. Therefore, in step S20, the update to i = 2 is performed, the process returns to step S12, and the process for the next background difference region d2 (t) is similarly performed with i = 2.

  On the other hand, if it is determined in step S16 that the total logical sum VAL (OR) = 0, it is determined that “there is no overlapping portion” between the regions d1 (t) and f1 (t). . In this case, the process branches to step S17, the update to j = 2 is performed, the process returns to step S13, i = 1, j = 2, and the overlapping portions for the regions d1 (t) and f2 (t) Processing for checking the presence / absence (steps S14 and S15) is executed. Here again, if it is determined that the total logical sum VAL (OR) = 0, the process branches again to step S17, the update to j = 3 is performed, the process returns to step S13, and i = 1, j = 3. As a result, the process (steps S14 and S15) for checking the presence / absence of overlapping portions in the areas d1 (t) and f3 (t) is executed.

  Thus, if it is determined that the total logical sum VAL (OR) = 0 for all combinations of i = 1 and j = 1 to 3, the background difference area d1 (t) is between the first to third frames. Since there is no overlapping part for any of the difference areas f1 (t), f2 (t), and f3 (t), an individual occupying the background difference area d1 (t) can be recognized as a stationary individual. Therefore, the process proceeds from step S17 to step S18, and the stationary individual index number k = 1 is updated. Further, the process proceeds to step S19, where the kth stationary individual index sk (t) is converted into the i-th background difference region. A process of creating based on di (t) is performed.

  In the case of the example shown in FIG. 19, in the procedure for the background difference area d1 (t) executed with i = 1, the overlapping part in the overlap confirmation process with the inter-frame difference area f1 (t) executed with j = 1. Since it is determined to be present, the process branches from step S16 to step S20. Subsequently, in the procedure for the background difference area d2 (t) executed as i = 2, j = 1 to 3 is executed. In the overlap confirmation process with the inter-frame difference areas f1 (t), f2 (t), and f3 (t), it is determined that there is no overlapping portion, so step S19 is executed. The stationary individual index s1 (t) is generated based on the second background difference area d2 (t). For example, in the case of the example shown in FIG. 2, a stationary individual index s composed of a hatched area with fine rectangular dots is created, and in the example shown in FIG. 12, a stationary individual index s composed of a dashed rectangular frame is created. .

  Note that the process of creating the logical product image IMG (AND) in step S14 includes the first binary image including the i-th background difference area di (t) and the j-th inter-frame difference area fj (t). Are generated as images of the same size using the same coordinate system, and an AND operation of the images is performed between the first binary image and the second binary image. Can be executed.

  Here, the first binary image and the second binary image may be created using labeling information L as shown in FIG. As described in §4, the computer recognizes the background difference areas d1 (t) and d2 (t) on the background difference image D (t) shown in FIG. )), And the computer recognizes the inter-frame difference areas f1 (t), f2 (t), and f3 (t) on the inter-frame difference image F (t). This is performed based on L (F (t)).

  Therefore, in step S14 of FIG. 20, i = 1 and j = 1, and the actual on the computer for creating the logical product image IMG (AND) for the two sets of regions d1 (t) and f1 (t). This processing can be performed by processing using the labeling information. Specifically, a sky image of a predetermined size (a set of pixels having a pixel value “0”) is prepared, and pixels for d1 (t) in the labeling information L (D (t)) shown in FIG. If the pixel value of only the pixel located at the coordinates listed as the specific information is changed to “1”, the first binary image described above can be created. Similarly, another sky image of the same size is prepared, and pixel values of only pixels located at coordinates listed as pixel specifying information for f1 (t) in labeling information L (F (t)) are obtained. If it is changed to “1”, the above-described second binary image can be created. Therefore, a logical product image IMG (AND) is obtained by performing a logical product operation on these binary images.

  However, as a specific calculation method, a method of directly comparing the labeling information L (D (t)) and L (F (t)) may be employed. For example, for each of the coordinate values of each pixel listed as pixel specifying information for d1 (t) in the labeling information L (D (t)) shown in FIG. 19, the labeling information L (F (t)) Whether there is a match with the coordinate value of each pixel listed as pixel specifying information for f1 (t) of the pixel. If there is a match, a black pixel (pixel If the pixel having the value “1” is arranged and there is no match, a white pixel (pixel having the pixel value “0”) is arranged at the position of the pixel. IMG (AND) may be created.

  Next, each procedure of the processing shown in the flowchart of FIG. 20 will be specifically described by taking as an example the case of executing using the images D (t2) and F (t2) at time t2 shown in FIG.

  The left side of FIG. 21 shows the processing contents when i = 1 and j = 1 in the flowchart of FIG. That is, FIG. 21 (a) shows a first binary image including the i-th (i = 1) background difference area d1 (t2), and FIG. 21 (b) shows the j-th (j = 1) shows a second binary image including the inter-frame difference area f1 (t2). Here, the background difference area d1 (t2) shown in FIG. 21 (a) is one of the two sets of background difference areas included in the background difference image D (t2) shown in FIG. Is composed of a collection of pixels listed as pixel specifying information for d1 (t2) in labeling information L (D (t2)). Similarly, the inter-frame difference area f1 (t2) shown in FIG. 21B is an inter-frame difference area included in the inter-frame difference image F (t2) shown in FIG. It is composed of a collection of pixels listed as pixel specifying information for f1 (t2) in L (F (t2)).

  The first binary image shown in FIG. 21 (a) and the second binary image shown in FIG. 21 (b) are binary images of the same size having the rectangular outer frame shown in the figure, and are black in the figure. The portion is composed of pixels having a pixel value “1”, and the white portion in the figure is composed of pixels having a pixel value “0”. As described above, the black portion in the figure is an aggregate of pixels specified by the pixel specifying information (data in which the coordinate values of the constituent pixels are listed) in the labeling information L. The same applies to FIGS. 23, 25, and 26 described later.

  In step S14 in the flowchart of FIG. 20, a logical product image IMG (AND) is obtained as a logical product of the two sets of regions d1 (t2) and f1 (t2). Actually, the logical product operation of the first binary image (FIG. 21 (a)) including the region d1 (t2) and the second binary image (FIG. 21 (b)) including the region f1 (t2). Thus, the logical product image IMG (AND) can be obtained. FIG. 21 (c) shows the logical product image IMG (AND) obtained in this way.

  On the other hand, in step S15 in the flowchart of FIG. 20, the total logical sum VAL (OR) for the pixel values of all the pixels in the logical product image IMG (AND) is obtained. That is, if there is at least one black pixel (pixel value “1”) in the logical product image IMG (AND) shown as an image in a rectangular frame in the figure, VAL (OR) = 1. If there is no black pixel, VAL (OR) = 0.

  In practice, as described above, when the number of black pixels is less than the predetermined reference, VAL (OR) = 0, and when the number of black pixels exceeds the predetermined reference, VAL ( OR) = 1. In short, it is preferable to determine that there is no overlapping portion when the overlapping portions of the two sets of regions d1 (t2) and f1 (t2) are less than a predetermined area.

  In the case of the logical product image IMG (AND) shown in FIG. 21 (c), VAL (OR) = 1 and it is determined that there is an overlapping portion. Therefore, in the flowchart of FIG. The process branches to S20, and no stationary individual index is created for i = 1 (for the background difference area d1 (t2)) (step S19 is not executed). FIG. 21 (d) shows that a stationary individual index for i = 1 is not created for convenience of explanation.

  On the other hand, the right side of FIG. 21 shows the processing contents when i = 2 and j = 1 in the flowchart of FIG. That is, FIG. 21 (e) shows a first binary image including the i-th (i = 2) background difference region d2 (t2), and FIG. 21 (f) shows the j-th (j = 1) shows a second binary image including the inter-frame difference area f1 (t2). The background difference area d2 (t2) shown in FIG. 21 (e) is one of the two sets of background difference areas included in the background difference image D (t2) shown in FIG. It is comprised by the aggregate | assembly of each pixel enumerated as pixel specific information about d2 (t2) in labeling information L (D (t2)). Similarly, the inter-frame difference area f1 (t2) shown in FIG. 21 (f) is the inter-frame difference area included in the inter-frame difference image F (t2) shown in FIG. 8, and actually the labeling information It is composed of a collection of pixels listed as pixel specifying information for f1 (t2) in L (F (t2)).

  In step S14 in the flowchart of FIG. 20, a logical product image IMG (AND) is obtained as the logical product of the two sets of regions d2 (t2) and f1 (t2). Actually, the logical product image IMG (AND) can be obtained by the logical product operation of the first binary image (FIG. 21 (e)) and the second binary image (FIG. 21 (f)). . FIG. 21 (g) shows the logical product image IMG (AND) thus obtained. In this example, since there is no overlapping portion between the areas d2 (t2) and f1 (t2), the logical product image IMG (AND) shown in FIG. 21 (g) is an empty image.

  Therefore, since VAL (OR) = 0 and it is determined that there is no overlapping portion, the process proceeds from step S16 to steps S17, S18, and S19 in the flowchart of FIG. The stationary individual index s1 (t2) for the region d2 (t2) is created. Here, “1” of “s1” corresponds to the stationary individual number k = 1. FIG. 21 (h) shows the stationary individual index s1 (t2) for i = 2 created in this way. Here, as in the example shown in FIG. 2, an example is shown in which a stationary individual index s1 (t2) composed of a hatched area with fine rectangular dots is created based on the background difference area d2 (t2).

<5-2. Procedure for creating moving individual index m>
FIG. 22 is a flowchart showing a procedure for creating the motion individual index m by the cooperative work of the duplication presence / absence determination unit 150, the motion individual recognition unit 162, and the detected object display unit 170 of the apparatus of FIG. Again, the background difference image creation unit 130 creates background difference images D (t1) to D (t4) as shown in FIG. 5, and the interframe difference image creation unit 140 creates an interframe difference as shown in FIG. Assuming that the difference images F (t1) to F (t4) are created, the procedure executed by the duplication presence / absence determination unit 150, the moving individual recognition unit 162, and the detected individual display unit 170 based on both the difference images will be described. To do.

  First, in step S21, the motion individual index number h is set to an initial value 0. The motion individual index number h is a serial number assigned to each recognized motion individual. Actually, the motion object first recognized is h = 1 in step S28, and therefore the motion object index number h = 1 is assigned.

  The subsequent step S22 is a loop process in which the procedure up to step S30 is repeatedly executed from i = 1 to Imax. As in the flowchart of FIG. 20, i indicates the background difference area number, the initial value is 1, and the maximum value is Imax. Therefore, the loop process from step S22 to step S30 is a loop process for the background difference area.

  The next step S23 is a loop process in which the procedure up to step S27 is repeatedly executed from j = 1 to Jmax. As in the flowchart of FIG. 20, j indicates the inter-frame difference area number, the initial value is 1, and the maximum value is Jmax. Therefore, the loop process from step S23 to step S27 is a loop process for the inter-frame difference area.

  The process of step S24 is exactly the same as the process of step S14 in the flowchart of FIG. 20, and for each pixel in the i-th background difference area di (t), the j-th inter-frame difference area fj (t). A logical product image IMG (AND) obtained by ANDing the pixels at the same position is created. Since the specific process has already been described as the process of step S14, the description thereof is omitted here.

  Further, the processing in step S25 is exactly the same as the processing in step S15 in the flowchart of FIG. 20, and the total logical sum VAL obtained by ORing all the pixels in the logical product image IMG (AND) obtained in step S24. (OR) is required. Since the specific processing has already been described as the processing in step S15, description thereof is omitted here.

  The processing in step S26 corresponds to the processing in step S16 in the flowchart of FIG. 20, but the processing after branching is different. This is because the flowchart of FIG. 20 shows the procedure for creating the stationary individual index s, whereas the flowchart of FIG. 22 described here shows the procedure for creating the moving individual index m. is there.

  In step S26, it is determined whether or not there is an overlapping portion based on the total logical sum VAL (OR). First, when the total logical sum VAL (OR) = 1, it is determined that “there is an overlapping portion” between the regions d1 (t) and f1 (t). In this case, since the individual occupying the background difference area d1 (t) can be recognized as an action individual, the process exits the loop j, branches to step S28, updates the action individual index number h, and further performs step S29. , The process of creating the h-th motion individual index mh (t) based on the i-th background difference area di (t) is performed.

  In the case of the example shown in FIG. 19, in the procedure for the background difference area d1 (t) executed with i = 1, in the overlap confirmation process with the inter-frame difference area f1 (t) executed with j = 1. Since it is determined that there is, the process proceeds from step S26 to steps S28 and S29, and the first motion individual index m1 (t) is created based on the first background difference area d1 (t). For example, in the case of the example shown in FIG. 3, the motion individual index m composed of a hatched area with a hatched mesh is created, and in the example shown in FIG. 12, the motion individual index m composed of a solid rectangular frame is created.

  Thus, when the first motion individual index m1 (t) is created in step S29, the process for the background difference area d1 (t) performed with i = 1 is completed. Therefore, in step S30, the update to i = 2 is performed, the process returns to step S22, and the process for the next background difference region d2 (t) is similarly performed with i = 2.

  On the other hand, if it is determined in step S26 that the total logical sum VAL (OR) = 0, it is determined that “there is no overlapping portion” between the areas d1 (t) and f1 (t). . In this case, the process branches to step S27, the update to j = 2 is performed, the process returns to step S23, i = 1, j = 2, and the overlapping portions for the regions d1 (t) and f2 (t) Processing for checking the presence or absence (steps S24 and S25) is executed. Again, if it is determined that the total logical sum VAL (OR) = 0, the process branches back to step S27, the update to j = 3 is performed, the process returns to step S23, and i = 1, j = 3. As a result, processing (steps S24 and S25) for checking the presence / absence of overlapping portions for the regions d1 (t) and f3 (t) is executed.

  Thus, if it is determined that the total logical sum VAL (OR) = 0 for all combinations of i = 1 and j = 1 to 3, the background difference area d1 (t) is between the first to third frames. Since there is no overlapping part for any of the difference areas f1 (t), f2 (t), and f3 (t), an individual occupying the background difference area d1 (t) can be recognized as a stationary individual. For this reason, for the background difference area d1 (t), the process of creating the motion individual index m is not executed, and the process of i = 1 is completed. Therefore, in step S30, the update to i = 2 is performed, the process returns to step S22, and the process for the next background difference region d2 (t) is similarly performed with i = 2.

  In the case of the example shown in FIG. 19, in the procedure for the background difference area d1 (t) executed with i = 1, as described above, the first motion individual index m1 (t) is created. In the procedure for the background difference area d2 (t) executed as i = 2, overlap with the inter-frame difference areas f1 (t), f2 (t), and f3 (t) executed as j = 1 to 3 Since it is determined in the confirmation process that there is no overlapping part, the process of creating the motion individual index m is not executed, and the process of i = 2 is finished.

  Here, each procedure of the process shown in the flowchart of FIG. 22 will be specifically described by taking as an example the case of executing each procedure D (t2) and F (t2) at time t2 shown in FIG.

  The left side of FIG. 23 shows the processing contents when i = 1 and j = 1 in the flowchart of FIG. 23A shows the first binary image including the i-th (i = 1) background difference area d1 (t2), and FIG. 23B shows the j-th (j = 1) shows a second binary image including the inter-frame difference area f1 (t2). FIG. 23 (c) shows a logical product image IMG (AND) created in step S24 in the flowchart of FIG. The contents of FIGS. 23 (a), (b), and (c) are the same as those of FIGS. 21 (a), (b), and (c), and detailed description thereof is omitted.

  For the logical product image IMG (AND) shown in FIG. 23 (c), VAL (OR) = 1 and it is determined that there is an overlapping portion. For this reason, in the flowchart of FIG. 22, the process branches from step S26 to step S28, and the motion individual index m1 (t2) is created for i = 1 (for the background difference area d1 (t2)). Here, “1” of “m1” corresponds to the operation individual number h = 1. FIG. 23 (d) shows the motion individual index m1 (t2) for i = 1 created in this way. Here, as in the example shown in FIG. 3, an example is shown in which an action individual index m1 (t2) including a hatched area with hatched meshes is created based on the background difference area d1 (t2).

  On the other hand, the right side of FIG. 23 shows the processing contents when i = 2 and j = 1 in the flowchart of FIG. 23 (e) shows a first binary image including the i-th (i = 2) background difference area d2 (t2), and FIG. 23 (f) shows the j-th (j = 1) shows a second binary image including the inter-frame difference area f1 (t2). FIG. 23 (g) shows the logical product image IMG (AND) created in step S24 in the flowchart of FIG. The contents of FIGS. 23 (e), (f), and (g) are the same as those of FIGS. 21 (e), (f), and (g), and detailed description thereof is omitted.

  The logical product image IMG (AND) shown in FIG. 23 (e) is actually an empty image, and VAL (OR) = 0. Therefore, finally, in the flowchart of FIG. 22, the process proceeds from step S26 to steps S27 and S30, and no motion individual index is created for i = 2 (for the background difference area d2 (t2)) (step S29 is executed). Not) FIG. 23 (h) shows that the motion individual index for i = 2 is not created for convenience of explanation.

<5-3. Modification of the procedure for creating the motion individual index m>
As described so far, the detected individual display unit 170 in the individual detection apparatus 100 shown in FIG. 1 includes the stationary individual index s indicating the individual recognized as the stationary individual by the stationary individual recognition unit 161, and the moving individual recognition unit. 162 has a function of displaying an action individual index m indicating an individual recognized as an action individual by 162. In the example described so far, the stationary individual index s and the moving individual index m are created based on the background differential image di (t) included in the background differential image D (t).

  For example, according to the procedure for creating a stationary individual index shown in FIG. 20, a stationary individual index s1 (t2) as shown in FIG. 21 (h) can be created. This stationary individual index s1 (t2) is created based on the background difference area d2 (t2) shown in FIG. Specifically, the stationary individual index s1 (t2) shown in FIG. 21 (h) is created by applying hatching with fine rectangular dots inside the background difference area d2 (t2) shown in FIG. 21 (e). Can do. Also, the stationary individual index s1 (t2) shown in the individual detection image Q (t2) -SM in FIG. 12 can be created as a circumscribed rectangle of the background difference area d2 (t2). The stationary individual index s1 (t2) related to various variations shown in FIG. 13 and subsequent figures can also be created based on the background difference area d2 (t2).

  Similarly, according to the procedure for creating the motion individual index shown in FIG. 22, the motion individual index m1 (t2) as shown in FIG. 23 (d) can be created. This motion individual index m1 (t2) is created based on the background difference area d1 (t2) shown in FIG. Specifically, the stationary individual index m1 (t2) shown in FIG. 23 (d) can be created by applying hatching with hatched mesh inside the background difference area d1 (t2) shown in FIG. 23 (a). it can. Also, the motion individual index m1 (t2) shown in the individual detection image Q (t2) -SM in FIG. 12 can be created as a circumscribed rectangle of the background difference area d1 (t2). The motion individual index m1 (t2) related to various variations shown in FIG. 13 and subsequent figures can also be created based on the background difference area d1 (t2).

  Thus, in the embodiments described so far, the detected individual display unit 170 creates the stationary individual index s indicating the stationary individual based on the background difference area di (t) occupied by the stationary individual, The motion individual index m indicating the motion individual is created based on the background difference area di (t) occupied by the motion individual.

  For this reason, in the procedure for creating a stationary individual index shown in FIG. 20, first, pay attention to the i-th background difference area di (t) (step S12), and that the background difference area di (t) has no overlapping portion. When the determination is made (step S16), the stationary individual index sk (t) is created based on the background difference area di (t) (step S19). Similarly, in the procedure for creating the motion individual index shown in FIG. 22, first, the i-th background difference area di (t) is focused (step S22), and the background difference area di (t) has an overlapping portion. When the determination is made (step S26), the motion individual index mh (t) is created based on the background difference area di (t) (step S29).

  The basic concept of the procedure described here is to create an index based on the inter-frame difference area fj (t), not the background difference area di (t). However, the background difference area di (t) is an area indicating a difference with respect to the background image as shown in the background difference image D (t) of FIG. ) Is a region showing temporal changes as shown in the inter-frame difference image F (t) in FIG. Therefore, when adopting a modified example in which an index is created based on the inter-frame difference area fj (t) instead of the background difference area di (t), it is necessary to pay attention to the following two points.

  The first point to note about this modification is that it can be applied to the creation of the moving individual indicator m, but cannot be applied to the creation of the stationary individual indicator s. For example, the inter-frame difference image F (t1) shown at time t1 in FIG. 6 includes inter-frame difference areas f1 (t1) and f2 (t1) corresponding to the adult A1 and the child B1, respectively. However, the inter-frame difference image F (t2) shown at time t2 includes only the inter-frame difference area f1 (t2) corresponding to the adult A2. This is because the child B2 is stationary at time t2. Also, the inter-frame difference area F is not included in the inter-frame difference image F (t3) shown at time t3. This is because both the adult A3 and the child B3 are stationary at time t3.

  Thus, the inter-frame difference area fj (t) included in the inter-frame difference image F (t) is an area formed by the motion of the moving individual, and does not include information on the stationary individual. For this reason, the modified example in which the index is created based on the inter-frame difference area fj (t) can be used to create the moving individual index m, but cannot be used to create the stationary individual index s.

  The second point to note about this modification is that the background difference area di (t) included in the background difference image D (t) in FIG. (In this example, the human type) is an area that is directly expressed, whereas the inter-frame difference area fj (t) is an example included in the inter-frame difference image F (t) in FIG. As you can see, it is not an area that directly represents the outline of an individual. Therefore, when adopting a modification in which the motion individual index m is created based on the inter-frame difference area fj (t), correction processing as necessary is required for the inter-frame difference area fj (t). .

  For example, the inter-frame difference image F (t1) shown at time t1 in FIG. 6 includes two sets of inter-frame difference areas f1 (t1) and f2 (t1), which are an adult A1 and a child B1. The outer shape of the limb is not shown as it is, but the shape is formed by overlapping double humanoids according to the movements, and there is a lack of white in a part of the limb. When the inter-frame difference image F (t4) shown at time t4 in FIG. 6 is reached, the whole image of the original individual is no longer completely lost, and two sets of inter-frame difference areas f1 (corresponding to some arm portions). Only t4) and f2 (t4) are left.

  Therefore, in the case of the modification described here, the detected individual display unit 170 uses the motion individual index m indicating the motion individual based on the background difference region occupied by the motion individual with respect to the inter-frame difference region occupied by the motion individual. A method of creating by performing correction is adopted. Specifically, the correction may be performed by performing a logical OR operation on the figures constituting the area for the inter-frame difference area occupied by the moving object and the background difference area occupied by the moving object.

  For example, when the motion individual index m for the adult A1 (motion individual) is created using the inter-frame difference region f1 (t1) included in the inter-frame difference image F (t1) in FIG. , Based on the background difference area d1 (t1) included in the background difference image D (t1) of FIG. 5 with respect to the interframe difference area f1 (t1) in which a part of the limb is white. The motion individual index m is created based on the corrected inter-frame difference region f1 (t1) in which correction of the missing limbs is performed. Actually, a logical OR operation is performed on the graphic forming the inter-frame difference area f1 (t1) and the graphic forming the background differential area d1 (t1) to create a logical OR image, and this logical OR image The motion individual index m may be created based on the above.

  Hereinafter, with reference to FIG. 24, a procedure for creating the motion individual index m according to this modification will be described. FIG. 24 is a flowchart showing a procedure for creating the motion individual index m by the cooperative work of the duplication presence / absence determination unit 150, the motion object recognition unit 162, and the detected object display unit 170 of the apparatus of FIG. Again, the background difference image creation unit 130 creates background difference images D (t1) to D (t4) as shown in FIG. 5, and the interframe difference image creation unit 140 creates an interframe difference as shown in FIG. Assuming that the difference images F (t1) to F (t4) are created, the procedure executed by the duplication presence / absence determination unit 150, the moving individual recognition unit 162, and the detected individual display unit 170 based on both the difference images will be described. To do.

  First, in step S31, the motion individual index number h is set to an initial value 0. The motion individual index number h is a serial number assigned to each recognized motion individual as described above. Actually, the first motion object recognized is h = 1 in step S33, and therefore the motion object index number h = 1 is assigned.

  The subsequent step S32 is a loop process in which the procedure up to step S42 is repeatedly executed from j = 1 to Jmax. Similar to the flowcharts of FIGS. 20 and 22, j indicates the inter-frame difference area number, the initial value is 1, and the maximum value is Jmax. Therefore, the loop process from step S32 to step S42 is a loop process for the inter-frame difference area. The procedure shown in FIGS. 20 and 22 creates the stationary individual index s and the moving individual index m based on the background differential image area di (t) included in the background differential image D (t) as described above. Therefore, the processing focusing on the individual background difference areas di (t) is performed, but in the procedure shown in FIG. 24, the inter-frame difference areas fj (t) included in the inter-frame difference image F (t). ) Is generated based on the individual inter-frame difference area fj (t).

  In the next step S33, the motion individual index number h is updated to h = h + 1. In step S34, the jth inter-frame difference area fj (t) is set as the hth temporary motion individual index m′h (t). ) Is set. For example, when the loop processing of j = 1 is started in step S32, h = 1 is set in step S33. In step S34, the first inter-frame difference area f1 (t) is changed to the first provisional motion. It is set as the individual index m′1 (t). Here, the “provisional motion individual index” means a “provisional” index that is not an official motion individual index m at this stage.

  Specifically, for example, the loop of the inter-frame difference area starting from step S32 is a loop that targets the inter-frame difference area f1 (t1) in the inter-frame difference image F (t1) in FIG. In this case, first, the inter-frame difference area f1 (t1) is set as the temporary motion individual index m′1 (t1). The temporary motion individual index m′1 (t1) is missing in a part of the limb, and in order to be used as the official motion individual index m, it is necessary to perform correction in step S39 through a subsequent procedure. There is.

  The subsequent step S35 is a loop process in which the procedure up to step S40 is repeatedly executed from i = 1 to Imax. As in the flowcharts of FIGS. 20 and 22, i indicates the background difference area number, the initial value is 1, and the maximum value is Imax. Therefore, the loop process from step S35 to step S40 is a loop process for the background difference area. The purpose of this loop processing is to perform correction for the temporary motion individual index m′h (t) using the overlapping background difference area di (t).

  First, in step S36, a logical product image obtained by ANDing each pixel in the jth inter-frame difference area fj (t) with a pixel at the same position in the i-th background difference area di (t). IMG (AND) is created. Actually, the logical product of the first binary image including the region fj (t) and the second binary image including the region di (t) is similar to step S14 in FIG. 20 and step S24 in FIG. A logical product image IMG (AND) can be obtained by calculation.

  Subsequently, in step S37, for all the pixels in the logical product image IMG (AND) obtained in step S36, a total logical sum VAL (OR) obtained by OR is obtained. As already described in step S15 in FIG. 20 and step S25 in FIG. 22, the value of this total logical sum VAL (OR) has the pixel value “0” in all the pixels in the logical product image IMG (AND). It is “0” only when it is a pixel (white pixel), and it is “1” when any one of the pixels (black pixels) having the pixel value “1” is included. Therefore, if the total logical sum VAL (OR) = 0, it indicates that there is no overlap between the regions fj (t) and di (t), and if the total logical sum VAL (OR) = 1, the region It shows that there is an overlapping portion between fj (t) and di (t).

  In practice, as described above, when the number of black pixels is less than the predetermined reference, VAL (OR) = 0, and when the number of black pixels exceeds the predetermined reference, VAL ( OR) = 1.

  Therefore, in the next step S38, a branch based on the total logical sum VAL (OR) is performed. First, when the total logical sum VAL (OR) = 1, it is determined that “there is an overlapping portion” between the regions fj (t) and di (t). In this case, since the background difference area di (t) can be determined as an area for the same individual as the inter-frame difference area fj (t), it can be used for correction of the provisional motion individual index m′h (t). Accordingly, the process proceeds to step S39, and the logical sum image IMG obtained by ORing each pixel in the temporary motion individual index m′h (t) with a pixel at the same position in the i-th background difference region di (t). (OR) is created, and this is used as a new temporary motion individual index m′h (t).

  Thus, when the loop processing (loop processing for i) in steps S35 to S40 is repeatedly executed from i = 1 to Imax, the background difference area determined to have an overlapping portion with respect to the interframe difference area fj (t). Each time di (t) is found, a logical sum image IMG (OR) is created by ORing the region di (t) and the temporary motion individual index m′h (t) at that time. Is updated to a new temporary motion individual index m′h (t). As a result, the initial provisional motion individual index m′h (t) including the missing portion is gradually repaired.

  Thus, when the loop processing for i is completed, the process proceeds to step S41, where the h-th motion individual index mh (t) is the h-th temporary motion individual index m′h (t) at that time, that is, It is created on the basis of a provisional index in which the repair of the chipped portion is completed. For example, when the motion individual index m shown in FIG. 3 is used, the motion individual index mh (t) is obtained by hatching the area constituting the temporary motion individual index m′h (t) with a hatched mesh. When the motion individual index m shown in FIG. 12 is used, the motion individual index mh (t) is obtained by obtaining a circumscribed rectangle of the area constituting the temporary motion individual index m′h (t). Can be created.

  Thus, when the h-th motion individual indicator mh (t) is created in step S41, the process returns from step S42 to step S32, and a new j is obtained based on the new inter-frame difference area by updating j. In order to create the motion individual index m, the loop processing (loop processing for j) in steps S32 to S42 is repeatedly executed from j = 1 to Jmax.

  Here, each procedure of the process shown in the flowchart of FIG. 24 will be described based on a specific example executed based on the images D (t1) and F (t1) at time t1 shown in FIG.

  The upper part of FIG. 25 shows the processing contents when j = 1 and i = 1 in the flowchart of FIG. First, FIG. 25A shows the first binary including the j-th (j = 1) inter-frame difference area f1 (t1) to be processed when j = 1 is set in step S32. An image is shown (same as shown in F (t1) of FIG. 7). In step S34, the inter-frame difference area f1 (t1) is set as the h-th (h = 1) temporary motion individual index m′1 (t1) as it is. FIG. 25 (b) shows a binary image including the temporary motion individual index m′1 (t1) set in this way (“1” of “m′1” indicates the motion individual number h = 1). Corresponding to). As shown in the figure, the temporary motion individual index m′1 (t1) is incomplete with a part of the limb missing and needs to be corrected.

  On the other hand, FIG. 25 (c) shows a second binary image including the i-th (i = 1) background difference region d1 (t1) to be processed when i = 1 is set in step S35. (Same as shown in D (t1) of FIG. 7). As described above, in step S36, the first binary image including the inter-frame difference area f1 (t1) shown in FIG. 25A and the second binary image including the background difference area d1 (t1) shown in FIG. A logical product image IMG (AND) obtained by ANDing the two binary images is created. The image shown in FIG. 25 (d) is the logical product image IMG (AND) created in this way.

  If the total logical sum VAL (OR) obtained by ORing all the pixels in the logical product image IMG (AND) is obtained in step S37, VAL (OR) = 1 is obtained for the illustrated IMG (AND). Therefore, it is determined that there is an overlapping part. This determination result is corrected for the current temporary motion individual index m′1 (t1) shown in FIG. 25B using the background difference area d1 (t1) shown in FIG. Means that you can. Therefore, the process proceeds from step S38 to step S39, and correction processing (processing for creating and replacing the logical sum image IMG (OR)) is executed. That is, an OR image IMG (OR) is created by taking an OR between the binary image including the temporary motion individual index m′1 (t1) and the binary image including the background difference region d1 (t1). Then, a process of setting this as a new temporary motion individual index m′1 (t1) is executed.

  FIG. 25 (e) shows a binary image including the new temporary motion individual index m′1 (t1) created in this way. The missing part of the limb that existed in the old temporary motion individual index m′1 (t1) shown in FIG. 25 (b) is repaired in the new temporary motion individual index m′1 (t1) shown in FIG. 25 (e). You can see that

  Subsequently, in step S40, updating to i = 2 is performed, and the processing from step S35 is executed again. The lower part of FIG. 25 shows the processing contents when j = 1 and i = 2 in the flowchart of FIG. First, FIG. 25 (f) shows a binary image including the i-th (i = 2) background difference region d2 (t1) to be processed when i = 2 is set (FIG. 25f). 5 (same as shown in D (t1)). In step S36, an AND image IMG (AND) obtained by ANDing between the inter-frame difference area f1 (t1) shown in FIG. 25A and the background difference area d2 (t1) shown in FIG. Created. The image shown in FIG. 25 (g) is the logical product image IMG (AND) created in this way. As shown in the figure, the regions f1 (t1) and d2 (t1) do not have any overlapping portions, so the logical product image IMG (AND) shown in FIG. 25 (g) is actually an empty image.

  Accordingly, in step S37, if a total logical sum VAL (OR) obtained by ORing all the pixels in the logical product image IMG (AND) is obtained, VAL (OR) = It becomes 0, and it is determined that there is no overlapping portion. This determination result is corrected for the temporary motion individual index m′1 (t1) at the present time shown in FIG. 25 (e) using the background difference area d2 (t1) shown in FIG. 25 (f). It means you can't. Therefore, the process proceeds from step S38 to step S40, and the correction processing (processing for creating and replacing the logical sum image IMG (OR)) in step S39 is not executed.

  Thus, when the loop processing from step S35 to step S40 is completed from i = 1 to Imax, in step S41, based on the current temporary motion individual index m′1 (t1) shown in FIG. A process of creating a formal motion individual index m1 (t1) is performed. FIG. 25 (h) shows the official motion individual index m1 (t1) created in this way. In the case of this example, the formal operation shown in FIG. 25 (h) is performed by applying hatching to the inside of the region constituting the temporary motion individual index m′1 (t1) shown in FIG. 25 (e). An individual index m1 (t1) is created.

  With the above processing, the first (h = 1) motion individual index m1 (t1) is created based on the inter-frame difference area f1 (t1). Therefore, in step S42, updating to j = 2 is performed, and the processing from step S32 is executed again.

  The upper part of FIG. 26 shows the processing contents when j = 2 and i = 1 in the flowchart of FIG. First, in FIG. 25 (a), the first binary including the j-th (j = 2) inter-frame difference region f2 (t1) to be processed when j = 2 is set in step S32. An image is shown (same as shown in F (t1) of FIG. 6). In step S34, the inter-frame difference area f2 (t1) is set as the h-th (h = 2) temporary motion individual index m′2 (t1) as it is. FIG. 26B shows the temporary motion individual index m′2 (t1) set in this way (“2” of “m′2” corresponds to the motion individual number h = 2). As shown in the figure, this temporary motion individual index m′2 (t1) is an incomplete one in which a part of the limb is missing and needs to be corrected.

  On the other hand, FIG. 26C shows a second binary image including the i-th (i = 1) background difference area d1 (t1) to be processed when i = 1 is set in step S35. (Same as shown in D (t1) of FIG. 5). As described above, in step S36, the AND image IMG obtained by ANDing between the inter-frame difference area f2 (t1) shown in FIG. 26 (a) and the background difference area d1 (t1) shown in FIG. 26 (c). (AND) is created. Actually, the logical product image IMG (AND) can be obtained by the logical product operation of the first binary image (FIG. 26 (a)) and the second binary image (FIG. 26 (c)). . The image shown in FIG. 26 (d) is a logical product image IMG (AND) created in this way. As shown in the figure, the regions f2 (t1) and d1 (t1) have no overlapping portion, so the logical product image IMG (AND) shown in FIG. 26 (d) is actually an empty image.

  Accordingly, in step S37, if a total logical sum VAL (OR) obtained by ORing all the pixels in the logical product image IMG (AND) is obtained, VAL (OR) = It becomes 0, and it is determined that there is no overlapping portion. This determination result is corrected using the background difference area d1 (t1) shown in FIG. 26 (c) with respect to the current temporary motion individual index m′2 (t1) shown in FIG. 26 (b). It means you can't. Therefore, the process proceeds from step S38 to step S40. At this time, the correction processing (processing for creating and replacing the logical sum image IMG (OR)) in step S39 is not executed.

  Subsequently, in step S40, updating to i = 2 is performed, and the processing from step S35 is executed again. The lower part of FIG. 26 shows the processing contents when j = 2 and i = 2 in the flowchart of FIG. First, FIG. 26 (e) shows a binary image including the i-th (i = 2) background difference region d2 (t1) to be processed when i = 2 is set (FIG. 26). 5 (same as shown in D (t1)). In step S36, an AND image IMG (AND) obtained by ANDing between the inter-frame difference area f2 (t1) shown in FIG. 26A and the background difference area d2 (t1) shown in FIG. Created. Actually, the logical product image IMG (AND) can be obtained by the logical product operation of the first binary image (FIG. 26 (a)) and the second binary image (FIG. 26 (e)). . The image shown in FIG. 26 (f) is the logical product image IMG (AND) created in this way.

  In the next step S37, if the total logical sum VAL (OR) obtained by ORing all the pixels in the logical product image IMG (AND) is obtained, VAL (OR) = 1 and it is determined that there is an overlapping portion. The determination result is corrected for the current temporary motion individual index m′2 (t1) shown in FIG. 26B by using the background difference area d2 (t1) shown in FIG. Means that you can. Therefore, the process proceeds from step S38 to step S39, and correction processing (processing for creating and replacing the logical sum image IMG (OR)) is executed. That is, the OR image IMG (OR) is created by performing OR between the binary image including the temporary motion individual index m′2 (t1) and the binary image including the background difference region d2 (t1). Then, a process of setting this as a new temporary motion individual index m′2 (t1) is executed.

  FIG. 26 (g) shows a binary image including the new temporary motion individual index m′2 (t1) created in this way. The missing part of the limb that existed in the old temporary motion individual index m′2 (t1) shown in FIG. 26 (b) is repaired in the new temporary motion individual index m′2 (t1) shown in FIG. 26 (g). You can see that

  Thus, when the loop processing from step S35 to step S40 is completed from i = 1 to Imax, in step S41, based on the current temporary motion individual index m′2 (t1) shown in FIG. A process of creating a formal motion individual index m2 (t1) is performed. Specifically, the formal motion individual index m2 (t1) is obtained by hatching the area constituting the temporary motion individual index m′2 (t1) shown in FIG. It is created (not shown).

  Through the above processing, the second (h = 2) motion individual indicator m2 (t1) is created based on the inter-frame difference region f2 (t1).

  In the case of the examples shown in FIGS. 25 and 26, the correction for the temporary motion individual index m′h (t) is completed by performing the correction process shown in step S39 only once. For example, in the case of the example of FIG. 25, the temporary motion individual index m′1 (t1) shown in FIG. 25 (b) is corrected to be ORed with the background difference area d1 (t1) shown in FIG. 25 (c). A new temporary motion individual index m′1 (t1) shown in FIG. 25 (e) can be obtained by performing the process only once, and a part of the missing limb is completely repaired. Similarly, in the case of the example of FIG. 26, the provisional motion individual index m′2 (t1) shown in FIG. 26B is ORed with the background difference area d2 (t1) shown in FIG. A new temporary motion individual index m′2 (t1) shown in FIG. 26 (g) can be obtained by performing the correction process only once, and a part of the missing limb is completely repaired.

  Therefore, with respect to these embodiments, there is no problem even if the process exits the loop of i and branches to step S41 when the correction process of step S39 is executed in the flowchart shown in FIG. However, in practice, there may be a case where the background difference area di (t) used for correction does not necessarily correspond to the entire individual. Therefore, in the flowchart shown in FIG. 24, after the correction processing in step S39 is executed. However, the process proceeds to step S40 without exiting the i loop.

  For example, when the background difference area d1 (t1) shown in FIG. 25 (c) has been divided into an upper body area d1 (t1) -1 and a lower body area d1 (t1) -2 for some reason (this individual Such a phenomenon occurs when the vicinity of the abdomen is approximated to the background), the region d1 (t1) -1 and the region d1 (t1) -2 are treated as separate regions in the i loop. In this case, since the complete restoration cannot be performed only by the correction process using the upper body region d1 (t1) -1, a correction process using the lower body region d1 (t1) -2 is required. According to the procedure shown in FIG. 24, since the loop of i is performed for all of i = 1 to Imax, sufficient repair can be performed even in the above example.

  In the above example, the procedure shown in the flowchart of FIG. 24 is executed based on the images D (t1) and F (t1) at time t1 shown in FIG. Although two motion individual indices mh (t) are created, there are cases where a plurality of motion individual indices mh (t) are created for the same individual.

  For example, when the procedure shown in the flowchart of FIG. 24 is executed based on the images D (t4) and F (t4) at time t4 shown in FIG. 10, two types of motion individual indices m1 (t4), m2 (t4) is created. This is because the inter-frame difference image F (t4) shown in FIG. 10 includes two sets of inter-frame difference areas f1 (t4) and f2 (t4), and these areas f1 (t4), This is because f2 (t4) is an area for the same individual (adult A).

  The two motion individual indices m1 (t4) and m2 (t4) created in this way are regions created based on the same individual (adult A), and thus have overlapping portions. Of course, when a plurality of motion individual indices m are created for the same individual, the detected individual display unit 170 may display them on the display screen in a duplicated manner, but in practice they overlap each other. When there are a plurality of indicators having a portion, it is preferable to select and display only one (for example, the indicator having the largest area).

  By the way, the motion individual index m created by the modification described in §5-3 has a slightly different form from the motion individual index m created by the embodiment described in §5-2. For example, the motion individual index m1 (t1) included in the individual detection image Q (t1) -M in FIG. 3 is created by the procedure of the embodiment described in §5-2, and FIG. It is created based on the background difference area d1 (t1) shown in c). On the other hand, the motion individual index m1 (t1) shown in FIG. 25 (h) is created by the modification described in §5-3, and the interframe interval shown in FIG. It is created based on the difference image f1 (t1). Each is composed of a closed region arranged in the vicinity of the region occupied by the individual A at time t1, whereas the former is an index indicating the position of the individual A at time t1. The latter is an index indicating the movement of the individual A from time t0 to t1.

  FIG. 4 shows the individual detection image Q displayed on the display screen when the procedure shown in FIG. 20 is used as the procedure for creating the stationary individual index s and the procedure shown in FIG. 22 is used as the procedure for creating the moving individual index m. It is a figure which shows the specific example of (t) -SM. On the other hand, FIG. 27 shows a display screen when the procedure shown in FIG. 20 is used as the procedure for creating the stationary individual index s and the procedure according to the modification shown in FIG. 24 is used as the procedure for creating the moving individual index m. It is a figure which shows the specific example of the individual detection image Q (t) -SM displayed on the top.

  Comparing FIG. 4 and FIG. 27, the original image P (t) shown on the left side is exactly the same, but the individual detected on the individual detection image Q (t) -SM shown on the right side. There is a difference for the index m. For example, comparing the individual detection image Q (t1) -SM with respect to time t1, in the case of FIG. 4, the motion individual indices m1 (t1) and m2 (t1) indicate the positions of the individual A1 and B1 at the time t1, respectively. On the other hand, in the case of FIG. 27, the moving individual indices m1 (t1) and m2 (t1) are indices indicating movement to the individuals A0 to A1 and movement to the individuals B0 to B1, respectively. ing

  As described above, the motion individual indices m1 (t1) and m2 (t1) are displayed so as to be superimposed on the original image P (t1). Therefore, in the individual detection image Q (t1) -SM shown in FIG. The indices m1 (t1) and m2 (t1) play a role of indicating the individuals A1 and B1 currently displayed on the original image P (t1). On the other hand, in the individual detection image Q (t1) -SM shown in FIG. 27, each index m1 (t1), m2 (t1) is the individual A1 currently displayed on the original image P (t1). , B1 as well as individuals A0 and B0 corresponding to the past positions.

  Of course, since the individuals A0 and B0 do not exist on the original image P (t1), the indices m1 (t1) and m2 (t1) displayed in the individual detection image Q (t1) -SM shown in FIG. ) Can represent a state in which the individual A1, B1 is moving by dragging a shadow. In addition, the size of the shadow to be dragged depends on the moving speed of the individual, so that a sense of speed is also expressed. Similarly, in the case of the individual detection image Q (t4) -SM shown in FIG. 27, the motion individual index m1 (t4) can represent the movement of the arm of the individual A4 (note that the motion individual index shown in FIG. 27). For convenience of explanation, m1 (t4) is not an example using the inter-frame difference areas f1 (t4) and f2 (t4) divided into two sets shown in the inter-frame difference image F (t4) in FIG. The motion individual index obtained when the regions f1 (t4) and f2 (t4) are merged to form one closed region).

  Such an effect is the same when another form of index is adopted. For example, the individual detection image Q (t) -SM shown in FIG. 12 uses the procedure shown in FIG. 20 as the procedure for creating the stationary individual index s and the procedure shown in FIG. 22 as the procedure for creating the moving individual index m. Is a diagram showing a specific example of the individual detection image Q (t) -SM displayed on the display screen, and is an example in which the circumscribed rectangle of the individual is used as the individual indices s and m. On the other hand, FIG. 28 shows a display screen when the procedure shown in FIG. 20 is used as the procedure for creating the stationary individual index s and the procedure according to the modification shown in FIG. 24 is used as the procedure for creating the moving individual index m. It is a figure which shows the specific example of the individual detection image Q (t) -SM displayed on the top. Here, the circumscribed rectangle of the stationary individual is used as the stationary individual index s, but the circumscribed rectangle of the final temporary moving individual index m′h (t) is used as the moving individual index m.

  When comparing the individual detection image Q (t) -SM shown in FIG. 12 and the individual detection image Q (t) -SM shown in FIG. 28, the latter motion individual index m also represents the motion of the motion individual. Can do. For example, looking at the individual detection image Q (t1) -SM shown in FIG. 28, the individual A1 is located at the right end of the motion individual indicator m1 (t1) made of a rectangle, so that it moves to the right side while dragging the shadow to the left side. You can feel that you are moving. Similarly, since the individual B1 is located at the left end of the action individual indicator m2 (t1) formed of a rectangle, it can be felt that the individual B1 is moving to the left side while dragging a shadow on the right side.

  As described above, when the modified example described in §5-3 is performed, the motion of the motion individual can be expressed by the motion individual index m on the individual detection image displayed by the detected individual display unit 170, Depending on the size of the shadow to be dragged, a special effect that the individual can feel the speed of movement is born.

  After all, when the embodiment described in §5-2 (an embodiment in which the motion individual index m is created based on the background difference image D (t)) is adopted, the motion individual index m as shown in FIG. 4 is displayed. Display using an index indicating an accurate individual without an afterimage of the previous time. Each action individual index m shown in FIG. 4 indicates a shape obtained by accurately cutting out each action individual at the time along the contour line.

  Therefore, in the embodiment described in §5-2, as a closed region index for forming a predetermined pattern inside a figure showing an accurate shape of each individual moving object or filling the inside of the figure with a predetermined color, This is suitable for expressing the motion individual index m. Further, when the motion individual index m is expressed as a contour index indicating the contour line of each motion individual as in the example shown in FIG. 14, or the individual motion individual is removed from the mask as in the example illustrated in FIG. 16. It is also suitable for expressing the motion individual index m as an index shown as an exposed portion.

  On the other hand, if the modified example described in §5-3 (an example in which the motion individual index m is created based on the inter-frame difference image F (t)) is employed, the motion individual index as shown in FIG. m can be displayed, and it is possible to indicate a moving motion object including an afterimage of the previous time. That is, such an action individual index m can be used to imply the movement direction and speed of each action individual to the observer, and is suitable for a case where it is desired to express the action individual movement more emphasized. is there. Therefore, in the modification described in §5-3, the motion individual index m is expressed as a frame-shaped index surrounding the motion individual, such as a rectangular frame (or an elliptical frame or the like) as shown in FIG. Suitable for cases.

<<< §6. Simple judgment method using circumscribed rectangle >>>
<6-0. Basic concept of simple judgment method using circumscribed rectangle>
In the individual detection device 100 shown in FIG. 1, the overlap presence / absence determination unit 150 performs inter-frame difference image F (t) between frames for each background difference region di (t) in background difference image D (t). The presence / absence of an overlapping portion with respect to the difference region fj (t) is determined, and the individual recognition unit 160 recognizes an individual occupying a specific background difference region di (t) as a stationary individual or a moving individual based on the determination result. become. Specifically, when a determination result of “no overlapping part” is obtained, it is recognized as a stationary individual, and when a determination result of “with overlapping part” is obtained, it is recognized as a moving individual. Is made.

  In the example shown in FIG. 21, for example, in the example shown in FIG. 21, the overlap presence / absence determination unit 150 performs the first binary image including the i-th background difference region di (t) and the j-th inter-frame difference. An AND operation between the images for the second binary image including the region fj (t) is performed to create an AND image IMG (AND), and all the pixels in the AND image IMG (AND) Is obtained by performing an operation for obtaining a total logical sum VAL (OR) obtained by ORing. Specifically, in this example, when the value of the total logical sum VAL (OR) is “0”, it is determined that “there is no overlapping portion”, and the value of the total logical sum VAL (OR) is “1”. In the case of, it is determined that “there is an overlapping portion”.

  However, in practice, such an image calculation is an operation that places a heavy processing burden on the computer. Of course, if the computer in charge of the processing as the duplication determination unit 150 is provided with a graphic chip dedicated to image processing, the image processing function can be improved as such, but for a general personal computer, The duplication presence / absence determination process is a very burdensome process. In particular, when a high-resolution camera is used as a fixed point camera, the number of pixels in one frame becomes enormous and the calculation burden becomes extremely heavy.

  Moreover, in the case of the individual detection apparatus 100 shown in FIG. 1, the detected individual display unit 170 displays an individual detection image Q (t) in which the stationary individual index s and the moving individual index m are superimposed on the original image P (t). The process of creating and displaying on the display screen is executed. Therefore, when using an image captured by a monitoring camera for a task of monitoring in real time, the duplication presence / absence determination unit 150 needs to execute duplication presence / absence determination processing on each input frame image in real time. It is necessary to complete the process.

  Considering such circumstances, here, a simple determination method for reducing the burden of the overlap determination processing by the overlap determination unit 150 will be described. In the case of a device that uses a computer such as a general personal computer and uses an image taken by a surveillance camera in real time, the simple determination method described below is used instead of the processing procedure described in §5. It is preferable to perform a processing procedure based on the above.

  A feature of the simple determination method described here is that, when determining the presence or absence of an overlapping portion between two regions, simple determination is performed using these circumscribed rectangles instead of the regions themselves. That is, the duplication presence / absence determination unit 150 determines whether or not there is an overlapping portion of the background difference area di (t) in the background difference image D (t) with respect to the inter-frame difference area fj (t) in the inter-frame difference image F (t). Is determined, a simple determination method for determining whether or not there is an overlapping portion between the circumscribed rectangle of the background difference area di (t) and the circumscribed rectangle of the inter-frame difference area fj (t) is executed.

  FIG. 29 is a diagram for explaining the handling of rectangular data in an embodiment in which this simple determination method is introduced. FIG. 29 (a) is a diagram showing the relationship between a figure of arbitrary shape and its circumscribed rectangle. The humanoid figure shown in the figure is an arbitrary figure, and corresponds to the background difference area di (t) and the inter-frame difference area fj (t) described in the above description. Such an arbitrarily-shaped figure must be expressed by specifying individual pixels located inside the figure by labeling information or the like, and the amount of information becomes enormous. On the other hand, the rectangle R shown in the figure is a rectangle circumscribing this arbitrarily shaped humanoid figure, and the humanoid figure has an origin O at the upper left position, the right direction is the X axis, and the lower direction is the lower direction. When arranged on the XY two-dimensional coordinate system as the Y-axis, it becomes a regular rectangle whose upper and lower sides are parallel to the X-axis and whose left and right sides are parallel to the Y-axis.

  Here, assuming that the four vertices of the circumscribed rectangle R are A, B, C, and D, the coordinate values are A (xmin, ymin), B (xmax, ymin) as shown on the right side of FIG. ), C (xmin, ymax), and D (xmax, ymax), and the circumscribed rectangle R can be defined only by the coordinate values of these four vertices (actually, the two diagonal vertices of the circumscribed rectangle R). Can be defined by coordinates only) Here, the coordinate value “xmin” is the minimum value of the X coordinate values of each pixel constituting the humanoid figure, and the coordinate value “xmax” is the maximum value thereof. Similarly, the coordinate value “ymin” is the minimum value of the Y coordinate value of each pixel constituting the humanoid figure, and the coordinate value “ymax” is the minimum value.

  When such a circumscribed rectangle R is used, the process for determining the presence or absence of an overlapping portion between two areas becomes very simple. FIG. 29 (b) is a diagram illustrating a method for determining the presence or absence of overlapping portions for two circumscribed rectangles. Here, a method for determining whether or not the first circumscribed rectangle Ra and the second circumscribed rectangle Rb have overlapping portions is shown. As illustrated, in this determination method, the upper left vertex coordinates (x1, y1) of the first circumscribed rectangle Ra, the lower right vertex coordinates (x2, y2) of the first circumscribed rectangle Ra, and the second circumscribed rectangle Rb. By using the upper left vertex coordinates (x3, y3) and the lower right vertex coordinates (x4, y4) of the second circumscribed rectangle Rb, the logical expression described on the right side of FIG. The determination result can be obtained.

  Specifically, as shown in the figure, a rectangular overlap determination value VAL defined by a logical operation expression of VAL (Overlap) = (x1 ≦ x4) AND (x3 ≦ x2) AND (y1 ≦ y4) AND (y3 ≦ y2) What is necessary is just to obtain the value of (Overlap). If VAL (Overlap) = 1, a determination result “overlapping portion” is obtained, and if VAL (Overlap) = 0, a determination result “no overlapping portion” is obtained. In the case of the illustrated example, since VAL (Overlap) = 1, the determination result is “There is an overlapping portion”.

  Therefore, in the procedure described in §5, the logical product image IMG (AND) is obtained for the background difference region di (t) and the inter-frame difference region fj (t), and the total logical sum VAL (OR) is calculated for this. Instead of processing, the coordinate values of the two vertices A and D of the circumscribed rectangle shown in FIG. 29 (a) are obtained for each area di (t) and fj (t), and a simple logical operation shown in FIG. 29 (b) is obtained. If the value of the rectangular overlap determination value VAL (Overlap) is obtained by an equation, a determination result indicating the presence or absence of an overlapping portion for two regions can be obtained. For this reason, the calculation burden is greatly reduced.

  Of course, the determination result indicating the presence / absence of overlapping portions for the regions di (t) and fj (t) does not exactly match the determination result indicating the presence / absence of overlapping portions for these circumscribed rectangles. When the simple determination method using is adopted, an error is included in the detection result, and there is a possibility that an error occurs in recognition of whether it is a stationary individual or a moving individual. However, the classification of a stationary individual or a moving individual is not strict in the first place, and even if some errors occur in the classification, there is no practical problem.

  Further, in step S39 in the flowchart of FIG. 24, it is necessary to create a logical sum image IMG (OR) for two images. When a circumscribed rectangle is used, this logical sum image IMG (OR) is created. Processing can also be simplified.

  FIG. 29 (c) is a diagram illustrating a method of creating a logical sum image IMG (OR) for an image including the rectangle Ra and an image including the rectangle Rb. Specifically, if an inclusion rectangle Rc that includes the rectangle Ra and the rectangle Rb is obtained, an image including the inclusion rectangle Rc is the logical sum image IMG (OR). The inclusion rectangle Rc corresponds to a circumscribed rectangle for the two rectangles Ra and Rb as shown by broken lines in the figure. Therefore, the coordinate values of the four vertices of the inclusion rectangle Rc can be easily determined based on the coordinate values of the four vertices of the rectangles Ra and Rb.

  Specifically, as described on the right side of FIG. 29 (c), the X coordinate of the upper left vertex of the inclusion rectangle Rc is the X coordinate value x1 of the upper left vertex of the rectangle Ra and the X coordinate value x3 of the upper left vertex of the rectangle Rb. The Y coordinate of the upper left vertex of the inclusion rectangle Rc is the smaller of the Y coordinate value y1 of the upper left vertex of the rectangle Ra and the Y coordinate value y3 of the upper left vertex of the rectangle Rb. . The X coordinate of the lower right vertex of the inclusion rectangle Rc is the larger of the X coordinate value x2 of the lower right vertex of the rectangle Ra and the X coordinate value x4 of the lower right vertex of the rectangle Rb, and the inclusion rectangle Rc The Y coordinate of the lower right vertex of Y is the larger one of the Y coordinate value y2 of the lower right vertex of the rectangle Ra and the Y coordinate value y4 of the lower right vertex of the rectangle Rb.

  In this way, in step S39 of FIG. 24, instead of creating the logical sum image IMG (OR) for the two images, the inclusion rectangle Rc is created based on the coordinate values of the two vertices of the circumscribed rectangle of each region. If the image including the inclusion rectangle Rc is used instead of the logical sum image IMG (OR), the calculation burden is greatly reduced.

  Of course, the inclusion rectangle Rc does not coincide with the logical sum area included in the logical sum image IMG (OR) for the two images. For example, the binary image shown in FIG. 25 (e) is an OR image IMG (OR) created based on the binary image shown in FIG. 25 (b) and the binary image shown in FIG. 25 (c). A region m′1 (t1) formed as a set of black pixels is a double humanoid region. On the other hand, when the inclusion rectangle Rc is created based on the circumscribed rectangle of the region m′1 (t1) shown in FIG. 25B and the circumscribed rectangle of the region d1 (t1) shown in FIG. The obtained inclusion rectangle Rc is merely a rectangular frame surrounding the region m′1 (t1) shown in FIG. 25 (e), and does not become a figure indicating a double humanoid region.

  However, the inclusion rectangle Rc can be used as it is as the rectangular motion individual index m1 (t1). In other words, in the case of the example shown in FIG. 25, the motion individual index m1 (t1) to be finally created is a hatched area with a hatched mesh as shown in FIG. When the simple determination method used is used, a rectangular motion individual index m1 (t1) (that is, an inclusion rectangle Rc) as shown by Q (t1) -SM in FIG. 28 is displayed. You can do it.

  Subsequently, what kind of processing is specifically performed when the simple determination method using the circumscribed rectangle is applied to each processing procedure described in §5-1, 5-2, and §5-3? Are briefly described in §6-1, 6-2, and 6-2, respectively.

<6-1. Procedure for creating stationary individual index s by simple judgment method>
In section 5-1, the procedure for creating the stationary individual index s has been described with reference to the flowchart of FIG. Here, a procedure when the simple determination method using the circumscribed rectangle described above is applied to the duplication presence / absence determination processing in the procedure shown in the flowchart of FIG. 20 will be described.

  FIG. 30 is a flowchart showing a procedure when this simple determination method is applied, and basically, the processing procedure is the same as the procedure shown in FIG. However, the content of the duplication determination process is replaced with a simple determination method. Therefore, only the parts different from the procedure shown in FIG. 20 (the procedure described in §5-1) will be described below. Specifically, the steps S11, S12, S13, S17, S18, and S20 are exactly the same as the procedure shown in FIG. 20, and the i loop for the background difference area and the inter-frame difference area. The loop of j is repeatedly executed.

  However, steps S14, S15, and S16 (duplication presence / absence determination processing and the like) shown in FIG. 20 are replaced with steps S14 ′ and S16 ′ in the flowchart of FIG. Step S19 is replaced with step S19 ′.

  First, in step S14 ′, a circumscribed rectangle ddi (t) of the i-th background difference area di (t) and a circumscribed rectangle ffj (t) of the j-th inter-frame difference area fj (t) are created. The Here, as data indicating each circumscribed rectangle, for example, the upper left vertex coordinate and the lower right vertex coordinate can be used, and one circumscribed rectangle can be specified only by the coordinate values of at least two vertices. In each figure used in the description of §6, for convenience, each circumscribed rectangle and inclusion rectangle is shown as a quadrangle on the image, but in reality, each of these rectangles is specified as image data including a set of pixels. Instead, it is specified by data of coordinate values of two vertices (X coordinate value and Y coordinate value of each vertex).

  Next, a rectangle overlap determination value VAL (Overlap) is obtained for both circumscribed rectangles ddi (t) and ffj (t). This rectangle overlap determination value VAL (Overlap) is a logical value calculated based on the logical operation expression shown on the right side of FIG. 29B, and as described above, both circumscribed rectangles ddi (t), ffj (t) If there is an overlapping part, VAL (Overlap) = 1, and if there is no overlapping part, VAL (Overlap) = 0.

  In the subsequent step S16 ′, a branch based on the rectangular overlap determination value VAL (Overlap) is performed. First, when the rectangle overlap determination value VAL (Overlap) = 1, it is determined that “there is an overlapping portion” in both circumscribed rectangles ddi (t) and ffj (t), so the loop of j is exited. Thus, branching to step S20 is performed. On the other hand, if the rectangle overlap determination value VAL (Overlap) = 0, it is determined that there is no overlapping portion in both circumscribed rectangles ddi (t) and ffj (t), and the process branches to step S17. Then, the update to j = 2 is performed, and the process returns to step S13, where i = 1, j = 2, and the process of checking the presence / absence of overlapping portions in the regions d1 (t) and f2 (t) The determination process using the simple determination method used is executed. Again, if it is determined that the rectangle overlap determination value VAL (Overlap) = 0, the process branches back to step S17, the update to j = 3 is performed, the process returns to step S13, and i = 1, j = 3, a process for determining whether or not there are overlapping portions in the regions d1 (t) and f3 (t) (a determination process using a simple determination method using a circumscribed rectangle) is executed.

  In this way, when the rectangle overlap determination value VAL (Overlap) = 0 is determined for all the combinations of i = 1 and j = 1 to 3, the circumscribed rectangle dd1 (t) of the background difference area d1 (t) is An overlapping portion is provided for any of the circumscribed rectangles ff1 (t), ff2 (t), and ff3 (t) of the first to third inter-frame difference areas f1 (t), f2 (t), and f3 (t). Therefore, an individual occupying the background difference area d1 (t) can be recognized as a stationary individual. Therefore, the process proceeds from step S17 to step S18, where the stationary individual index number k = 1 is updated. Further, the process proceeds to step S19 ′, where the kth stationary individual index ssk (t) is changed to the i-th background difference. A process of creating based on the circumscribed rectangle ddi (t) of the area di (t) is performed.

  In step S19 shown in FIG. 20, since the stationary individual index sk (t) is created based on the background difference area di (t), for example, a human-type stationary individual index s1 (see FIG. 21 (h)). t2) can be created. In step S19 ′ shown in FIG. 30, the stationary individual index ssk (t) is created based on the circumscribed rectangle ddi (t). ) Is not a human figure as shown in FIG. 21 (h), but becomes a rectangular index surrounding the figure (for example, a broken-line rectangle shown in the individual detection image Q (t2) -SM in FIG. 12). An index such as s1 (t2).

  However, if necessary, from among the background difference areas di (t) included in the background difference image D (t), the one located within the circumscribed rectangle ddi (t) is selected, and the selected identification is made. If the stationary individual index ssk (t) is created based on the background difference area di (t), a humanoid index as shown in FIG. 21 (h) can be created. Of course, it is possible to create an index related to variations as illustrated in FIGS.

  Next, each procedure of the process shown in the flowchart of FIG. 30 will be specifically described by taking as an example the case of executing using the images D (t2) and F (t2) at time t2 shown in FIG.

  The left side of FIG. 31 shows the processing contents when i = 1 and j = 1 in the flowchart of FIG. That is, FIG. 31A shows a first binary image including the i-th (i = 1) background difference region d1 (t2), and FIG. 31B shows the j-th (j = 1) shows a second binary image including the inter-frame difference area f1 (t2). However, in the simple determination method described here, as illustrated, a circumscribed rectangle dd1 (t2) is created for the background difference area d1 (t2), and a circumscribed rectangle ff1 (t2) for the inter-frame difference area f1 (t2). Is created. Actually, as described above, the upper left vertex coordinate and the lower right vertex coordinate of each circumscribed rectangle are determined and stored as data. The subsequent calculation is actually performed using the coordinate values of these two vertices.

  First, as shown in FIG. 31 (c), the circumscribed rectangles dd1 (t2) and ff1 (t2) are determined for overlap (step S14 ′). Specifically, based on the logical operation expression shown on the right side of FIG. 29 (b), the overlap existence determination operation for the circumscribed rectangles dd1 (t2) and ff1 (t2) may be performed. In the case of the example shown in FIG. 31 (c), since both rectangles overlap in the hatched portion, VAL (Overlap) = 1 is obtained, and a determination result that “there is an overlapping portion” is obtained. Therefore, in the flowchart of FIG. 30, the process branches from step S16 ′ to step S20, and no stationary individual index is created for i = 1 (for the background difference area d1 (t2)) (step S19 ′ is not executed). FIG. 31 (d) shows that a stationary individual index for i = 1 is not created for convenience of explanation.

  On the other hand, the right side of FIG. 31 shows the processing contents when i = 2 and j = 1 in the flowchart of FIG. That is, FIG. 31 (e) shows the first binary image including the i-th (i = 2) background difference region d2 (t2), and FIG. 31 (f) shows the j-th (j = 1) shows a second binary image including the inter-frame difference area f1 (t2). Here, as shown, a circumscribed rectangle dd2 (t2) is created for the background difference area d2 (t2), and a circumscribed rectangle ff1 (t2) is created for the inter-frame difference area f1 (t2).

  Then, as shown in FIG. 31 (g), it is determined whether or not these circumscribed rectangles dd2 (t2) and ff1 (t2) are overlapped (step S14 ′). Specifically, based on the logical operation expression shown on the right side of FIG. 29 (b), the overlap existence determination operation for the circumscribed rectangles dd2 (t2) and ff1 (t2) may be performed. In the case of the example shown in FIG. 31 (g), since both rectangles do not overlap at all, VAL (Overlap) = 0, and a determination result of “no overlapping portion” is obtained. Therefore, in the flowchart of FIG. 30, the process proceeds from step S16 ′ to steps S17, S18, and S19 ′, and a stationary individual index ss1 (t2) is created for i = 2 (for the background difference area d2 (t2)). become.

  FIG. 31 (h) shows the stationary individual index ss1 (t2) for i = 2 created in this way. According to the procedure shown in step S19 ′, the stationary individual index ss1 (t2) is created based on the circumscribed rectangle dd2 (t2) of the background difference area d2 (t2). The stationary individual index ss1 (t2) shown in FIG. 31 (h) is a representation of the circumscribed rectangle dd2 (t2) shown in FIG. 31 (e) with a broken line. This is the same index as the stationary individual index s1 (t2) shown in the individual detection image Q (t2) -SM in FIG.

  In short, the stationary individual recognizing unit 161 in the individual recognizing unit 160 has obtained a determination result that there is no overlapping portion of the circumscribed rectangle by the simple determination method performed by the overlapping presence determining unit 150. The individual B2 occupying (t2) is recognized as a stationary individual. Based on such recognition, the detected individual display unit 170, as shown on the right side of FIG. 31, shows a stationary individual index ss1 (t2) including a rectangular frame indicating the individual B2 recognized as a stationary individual (FIG. 31). (h)) is created based on the circumscribed rectangle dd2 (t2) of the background difference area d2 (t2) occupied by the stationary individual B2, and this is displayed on the individual detection image Q (t).

  However, if necessary, a collation operation based on the circumscribed rectangle dd2 (t2) and the background difference image D (t2) shown in FIG. 5 is performed, and the background difference area d1 ( d2 (t2) located in the circumscribed rectangle dd2 (t2) is selected from t2) and d2 (t2), and the stationary individual index ss1 (t2) is selected based on the selected background difference region d2 (t2). If created, a humanoid stationary individual index s1 (t2) as shown in FIG. 21 (h) can be created. Of course, it is possible to create an index related to variations as illustrated in FIGS.

<6-2. Procedure for creating the motion individual index m by the simple judgment method>
In §5-2, the procedure for creating the motion individual index m has been described with reference to the flowchart of FIG. Here, a procedure when the simple determination method using the circumscribed rectangle described above is applied to the duplication presence / absence determination processing in the procedure shown in the flowchart of FIG. 22 will be described.

  FIG. 32 is a flowchart showing a procedure when this simple determination method is applied. The processing procedure is basically the same as the procedure shown in FIG. However, the content of the duplication determination process is replaced with a simple determination method. Therefore, only the parts different from the procedure shown in FIG. 22 (the procedure described in §5-2) will be described below. Specifically, steps S21, S22, S23, S27, S28, and S30 are exactly the same as the procedure shown in FIG. 22, and the i loop for the background difference region and the inter-frame difference region are processed. The loop of j is repeatedly executed.

  However, steps S24, S25, and S26 (duplication presence / absence determination processing, etc.) shown in FIG. 22 are replaced with steps S24 ′ and S26 ′ in the flowchart of FIG. Further, step S29 is replaced with step S29 '.

  Here, the process of step S24 'is the same as the process of step S14' shown in FIG. That is, a circumscribed rectangle ddi (t) of the i-th background difference area di (t) and a circumscribed rectangle ffj (t) of the j-th inter-frame difference area fj (t) are created, and these circumscribed rectangles are created. A rectangular overlap determination value VAL (Overlap) is obtained for ddi (t) and ffj (t). For both circumscribed rectangles ddi (t) and ffj (t), if there is an overlapping portion, VAL (Overlap) = 1, and if there is no overlapping portion, VAL (Overlap) = 0.

  In the subsequent step S26 ′, a branch based on the rectangular overlap determination value VAL (Overlap) is performed. First, when the rectangle overlap determination value VAL (Overlap) = 1, it is determined that “there is an overlapping portion” in both circumscribed rectangles ddi (t) and ffj (t), so the loop of j is exited. Then, branching to step S28 is performed, the motion individual index number h is updated, and the process further proceeds to step S29 ′, where the h-th motion individual index mmh (t) is changed to the i-th background difference area di. A process of creating based on the circumscribed rectangle ddi (t) of (t) is performed.

  In this step S29 ′, the motion individual index mmh (t) is created based on the circumscribed rectangle ddi (t). Therefore, the motion individual index mmh (t) to be created is a person as shown in FIG. It is not a mold but a rectangular index surrounding it (for example, an index such as m1 (t2) consisting of a solid-line rectangle shown in the individual detection image Q (t2) -SM in FIG. 12).

  However, if necessary, from among the background difference areas di (t) included in the background difference image D (t), the one located within the circumscribed rectangle ddi (t) is selected, and the selected identification is made. If the stationary individual index mmh (t) is created based on the background difference area di (t), a human-type index as shown in FIG. 23 (d) can be created. Of course, it is also possible to create an index related to variations as exemplified in FIGS. 13, 14, and 16.

  Subsequently, each procedure of the process shown in the flowchart of FIG. 32 will be specifically described by taking as an example a case of executing using the images D (t2) and F (t2) at time t2 shown in FIG.

  The left side of FIG. 33 shows the processing contents when i = 1 and j = 1 in the flowchart of FIG. That is, FIG. 33 (a) shows a first binary image including the i-th (i = 1) background difference area d1 (t2), and FIG. 33 (b) shows the j-th (j = 1) shows a second binary image including the inter-frame difference area f1 (t2). However, in the simple determination method described here, as illustrated, a circumscribed rectangle dd1 (t2) is created for the background difference area d1 (t2), and a circumscribed rectangle ff1 (t2) for the inter-frame difference area f1 (t2). Is created.

  Then, as shown in FIG. 33 (c), it is determined whether or not these circumscribed rectangles dd1 (t2) and ff1 (t2) are overlapped (step S24 ′). Specifically, based on the logical operation expression shown on the right side of FIG. 29 (b), the overlap existence determination operation for the circumscribed rectangles dd1 (t2) and ff1 (t2) may be performed. In the example shown in FIG. 33 (c), since both rectangles overlap in the hatched portion, VAL (Overlap) = 1 is obtained, and a determination result that “there is an overlapping portion” is obtained. Therefore, in the flowchart of FIG. 32, the process proceeds from step S26 ′ to steps S28 and S29 ′, and the motion individual index mm1 (t2) for i = 1 (for the background difference area d1 (t2)) is created. .

  FIG. 33 (d) shows the motion individual index mm1 (t2) for i = 1 created in this way. According to the procedure shown in step S29 ′, the motion individual index mm1 (t2) is created based on the circumscribed rectangle dd1 (t2) of the background difference area d1 (t2). The motion individual index mm1 (t2) shown in FIG. 33 (d) is a representation of the circumscribed rectangle dd1 (t2) shown in FIG. 33 (a) as it is. This is the same index as the motion individual index m1 (t2) shown in the individual detection image Q (t2) -SM in FIG.

  In short, the motion individual recognition unit 162 in the individual recognition unit 160 has obtained a determination result that there is an overlapping portion for the circumscribed rectangle by the simple determination method performed by the duplication presence / absence determination unit 150. The individual A2 occupying (t2) is recognized as an operating individual. Then, based on such recognition, the detected individual display unit 170, as shown on the left side of FIG. 33, the moving individual indicator mm1 (t2) (FIG. 33) including a rectangular frame indicating the individual A2 recognized as the moving individual. (d)) is created based on the circumscribed rectangle dd1 (t2) of the background difference area d1 (t2) occupied by the moving individual A2, and this is displayed on the individual detection image Q (t).

  However, if necessary, a collation operation based on the circumscribed rectangle dd1 (t2) and the background difference image D (t2) shown in FIG. 5 is performed, and the background difference area d1 ( d1 (t2) located in the circumscribed rectangle dd1 (t2) is selected from t2) and d2 (t2), and the motion individual index mm1 (t2) is selected based on the selected background difference area d1 (t2). If created, a human-type motion individual index m1 (t2) as shown in FIG. 23 (d) can be created. Of course, it is also possible to create an index related to variations as exemplified in FIGS. 13, 14, and 16.

  On the other hand, the right side of FIG. 33 shows the processing contents when i = 2 and j = 1 in the flowchart of FIG. That is, FIG. 33 (e) shows the first binary image including the i-th (i = 2) background difference area d2 (t2), and FIG. 33 (f) shows the j-th (j = 1) shows a second binary image including the inter-frame difference area f1 (t2). Here, as shown, a circumscribed rectangle dd2 (t2) is created for the background difference area d2 (t2), and a circumscribed rectangle ff1 (t2) is created for the inter-frame difference area f1 (t2).

  Then, as shown in FIG. 33 (g), it is determined whether or not these circumscribed rectangles dd2 (t2) and ff1 (t2) are overlapped (step S24 '). Specifically, based on the logical operation expression shown on the right side of FIG. 29 (b), the overlap existence determination operation for the circumscribed rectangles dd2 (t2) and ff1 (t2) may be performed. In the case of the example shown in FIG. 33 (g), since both rectangles do not overlap at all, VAL (Overlap) = 0 and a determination result of “no overlapping portion” is obtained.

  Therefore, in the flowchart of FIG. 32, the process proceeds from step S26 ′ to steps S27 and S30, and no motion individual index is created for i = 2 (for the background difference region d2 (t2)) (step S29 ′ is not executed). FIG. 33 (h) shows that the motion individual index for i = 2 is not created for convenience of explanation.

<6-3. Modified example of the procedure for creating the motion individual index m by the simple judgment method>
In §5-3, the procedure according to the modified example of creating the motion individual index m has been described with reference to the flowchart of FIG. Here, a procedure when the simple determination method using the circumscribed rectangle described above is applied to the duplication presence / absence determination processing in the procedure shown in the flowchart of FIG. 24 will be described.

  FIG. 34 is a flowchart showing a procedure when this simple determination method is applied, and is basically the same processing procedure as the procedure shown in FIG. However, the content of the duplication determination process is replaced with a simple determination method. Therefore, only the parts different from the procedure shown in FIG. 24 (the procedure described in §5-3) will be described below. Specifically, the steps S31, S32, S33, S35, S40, and S42 are exactly the same as the procedure shown in FIG. 24, and the j loop for the interframe difference area and the background difference area. The i loop is repeatedly executed.

  However, steps S34, S36, S37, S38, S39, and S41 (duplication presence / absence determination processing, etc.) shown in FIG. Has been replaced.

  Here, in the process of step S34 ', the circumscribed rectangle ffj (t) of the jth inter-frame difference area fj (t) is set as the hth temporary motion individual index mm'h (t). Done. Here, the “provisional motion individual index” means a “provisional” index that is not an official motion individual index mm at the present stage. The temporary motion individual index mm′h (t) has an incomplete shape for use as the formal motion individual index mm, and in order to be used as the formal motion individual index mm, It is necessary to perform correction in step S39 '.

  The subsequent step S35 is a loop process in which the procedure up to step S40 is repeatedly executed from i = 1 to Imax. i indicates the background difference area number, the initial value is 1, and the maximum value is Imax. Therefore, the loop process from step S35 to step S40 is a loop process for the background difference area. The purpose of this loop processing is to perform correction for the temporary motion individual index mm′h (t) using the circumscribed rectangle ddi (t) of the overlapping background difference area di (t).

  First, in step S36 ′, a circumscribed rectangle ffj (t) of the j-th inter-frame difference region fj (t) and a circumscribed rectangle ddi (t) of the i-th background difference region di (t) are created. The rectangle overlap determination value VAL (Overlap) is obtained for both circumscribed rectangles ffj (t) and ddi (t). For both circumscribed rectangles ffj (t) and ddi (t), if there is an overlapping part, VAL (Overlap) = 1, and if there is no overlapping part, VAL (Overlap) = 0.

  In the subsequent step S38 ′, a branch based on the rectangular overlap determination value VAL (Overlap) is performed. First, when the rectangle overlap determination value VAL (Overlap) = 1, it is determined that “there is an overlapping portion” in both circumscribed rectangles ffj (t) and ddi (t). Branching is performed, an inclusion rectangle is obtained for the provisional motion individual index mm′h (t) and the circumscribed rectangle ddi (t) of the i-th background difference region di (t), and the inclusion rectangle is obtained as a new provisional rectangle. A process for setting the motion individual index mm′h (t) is performed. The method of obtaining the inclusion rectangle Rc for any two sets of rectangles Ra and Rb is as described in FIG. 29 (c).

  Thus, when the loop processing (loop processing for i) of steps S35 to S40 is repeatedly executed from i = 1 to Imax, there is an overlapping portion with respect to the circumscribed rectangle ffj (t) of the inter-frame difference area fj (t). Each time the circumscribed rectangle ddi (t) of the background difference area di (t) determined as is found, the circumscribed rectangle ddi (t) and the rectangle constituting the temporary motion individual index mm′h (t) at that time An inclusion rectangle is obtained, and update processing is performed using this as a new temporary motion individual index mm′h (t). As a result, the initial provisional motion individual index mm′h (t), which was an incomplete rectangle, is gradually restored.

  Thus, when the loop processing for i is completed, the process proceeds to step S41 ′, where the h-th motion individual index mmh (t) is the h-th temporary motion individual index mm′h (t) at that time, That is, the incomplete shape is created based on the temporary index that has been repaired. For example, the rectangular motion individual index m1 (t1) included in the individual detection image Q (t1) -SM in FIG. 28 is an index created in this way.

  Thus, when the h-th motion individual indicator mmh (t) is created in step S41 ′, the process returns from step S42 to step S32, and j is updated, so that a new one based on the new inter-frame difference area is obtained. In order to create a simple motion individual index mm, the loop processing of steps S32 to S42 (loop processing for j) is repeatedly executed from j = 1 to Jmax.

  Here, each procedure of the processing shown in the flowchart of FIG. 34 will be described based on a specific example executed based on the images D (t1) and F (t1) at time t1 shown in FIG.

  The upper part of FIG. 35 shows the processing contents when j = 1 and i = 1 in the flowchart of FIG. First, FIG. 35 (a) shows the first binary including the j-th (j = 1) inter-frame difference region f1 (t1) to be processed when j = 1 is set in step S32. An image is shown. In step S34 ′, a circumscribed rectangle ff1 (t1) of the inter-frame difference area f1 (t1) is created. The inter-frame difference area f1 (t1) is originally the same as that shown in F (t1) of FIG. 7, but here, for convenience, all the foot portions are missing as shown in the figure. The following explanation will be given as it has occurred. Therefore, the circumscribed rectangle ff1 (t1) for the inter-frame difference region f1 (t1) is a rectangle with a slightly reduced height that does not include the missing foot portion, as shown.

  In step S34 ′, the circumscribed rectangle ff1 (t1) is set as the h-th (h = 1) temporary motion individual index mm′1 (t1). FIG. 35 (b) shows the temporary motion individual index mm′1 (t1) set in this way (“1” of “mm′1” corresponds to the motion individual number h = 1). As shown in the figure, this temporary motion individual index mm′1 (t1) is an incomplete rectangle with a part missing, and needs to be corrected.

  On the other hand, FIG. 35C shows a second binary image including the i-th (i = 1) background difference region d1 (t1) to be processed when i = 1 is set in step S35. It is shown. As shown in the figure, a circumscribed rectangle dd1 (t1) is created for the background difference area d1 (t1). Since the background difference area d1 (t1) is a complete individual area in which there is no chipping in the foot portion, the circumscribed rectangle dd1 (t1) is also a complete rectangle in which no chipping occurs.

  Then, as shown in FIG. 35 (d), it is determined whether or not these circumscribed rectangles ff1 (t1) and dd1 (t1) are overlapped (step S36 '). Specifically, based on the logical operation expression shown on the right side of FIG. 29 (b), the overlap existence determination operation for the circumscribed rectangles ff1 (t1) and dd1 (t1) may be performed. In the case of the example shown in FIG. 35 (d), since both rectangles overlap in the hatched portion, VAL (Overlap) = 1 is obtained, and a determination result that “there is an overlapping portion” is obtained.

  This determination result is corrected using the circumscribed rectangle dd1 (t1) shown in FIG. 35 (c) for the current temporary motion individual index mm′1 (t1) shown in FIG. 35 (b). Means you can. Therefore, the process proceeds from step S38 ′ to step S39 ′, and correction processing (processing for creating and replacing an inclusion rectangle) is executed. That is, for the provisional motion individual index mm′1 (t1) and the circumscribed rectangle dd1 (t1), a process of obtaining an inclusion rectangle and setting the inclusion rectangle as a new provisional motion individual index mm′1 (t1) is performed.

  FIG. 35 (e) shows the new temporary motion individual index mm′1 (t1) created in this way. The broken lines in the new temporary motion individual index mm′1 (t1) indicate the outlines of two sets of rectangles that are the basis of the inclusion rectangle. The missing portion in the circumscribed rectangle constituting the old temporary motion individual indicator mm′1 (t1) shown in FIG. 35 (b) constitutes the new temporary motion individual indicator mm′1 (t1) shown in FIG. 35 (e). It can be seen that the inclusion rectangle is repaired.

  Subsequently, in step S40, updating to i = 2 is performed, and the processing from step S35 is executed again. The lower part of FIG. 35 shows the processing contents when j = 1 and i = 2 in the flowchart of FIG. First, FIG. 35 (f) shows a binary image including the i-th (i = 2) background difference area d2 (t1) to be processed when i = 2 is set (FIG. 35 (f)). 5 (same as shown in D (t1)). For this background difference area d2 (t1), a circumscribed rectangle dd2 (t1) is created as shown.

  Then, as shown in FIG. 35 (g), it is determined whether or not there are overlaps with respect to the circumscribed rectangles ff1 (t1) and dd2 (t1) (step S36 '). In the case of the example shown in FIG. 35 (g), since there is no overlapping portion in both rectangles, VAL (Overlap) = 0 is obtained, and a determination result of “no overlapping portion” is obtained.

  This determination result means that the circumscribed rectangle dd2 (t1) shown in FIG. 35 (g) cannot be used for correction of the current temporary motion individual index mm′1 (t1) shown in FIG. 35 (e). Therefore, the process proceeds from step S38 ′ to step S40, and the correction process (the process of creating and replacing the inclusion rectangle) in step S39 ′ is not executed.

  Thus, when the loop processing from step S35 to step S40 is completed from i = 1 to Imax, in step S41 ′, based on the current temporary motion individual index mm′1 (t1) shown in FIG. Then, a process of creating a formal motion individual index mm1 (t1) is performed. FIG. 35 (h) shows the official motion individual index mm1 (t1) created in this way. In this example, the formal motion individual index mm1 (t1) is composed of the same rectangular index as the provisional motion individual index mm′1 (t1), and the individual detection image Q (t1) − in FIG. It becomes the same as the motion individual index m1 (t1) shown in SM.

  In short, the motion individual recognition unit 162 in the individual recognition unit 160 uses the simple determination method performed by the duplication presence / absence determination unit 150 to obtain an inter-frame difference region from which a determination result is obtained that there is an overlapping portion for a circumscribed rectangle. The individual A1 occupying f1 (t1) is recognized as an operating individual. Then, based on such recognition, the detected individual display unit 170, as shown in FIG. 35, the moving individual indicator mm1 (t1) (FIG. 35 (h1)) formed of a rectangular frame indicating the individual A1 recognized as the moving individual. )) To the circumscribed rectangle dd1 (t1) of the background difference region d1 (t1) occupied by the motion individual A1 with respect to the circumscribed rectangle ff1 (t1) of the inter-frame difference region f1 (t1) occupied by the motion individual A1. It is created by performing a correction based on this, and this is displayed on the individual detection image Q (t).

  Specifically, the detected individual display unit 170 includes a circumscribed rectangle ff1 (t1) of the inter-frame difference area f1 (t1) occupied by the moving object A1 and a circumscribed rectangle dd1 of the background difference area d1 (t1) occupied by the moving object A1. For (t1), an inclusion rectangle that includes both circumscribed rectangles is obtained (FIG. 35 (e)), and an action individual index mm1 (t1) is created based on the obtained inclusion rectangle.

  Through the above processing, the first (h = 1) motion individual index mm1 (t1) is created as a rectangular figure. Therefore, in step S42, updating to j = 2 is performed, and the processing from step S32 is executed again.

  The upper part of FIG. 36 shows the processing contents when j = 2 and i = 1 in the flowchart of FIG. First, FIG. 36 (a) shows the first binary including the j-th (j = 2) inter-frame difference region f2 (t1) to be processed when j = 2 is set in step S32. An image is shown. In step S34 ′, a circumscribed rectangle ff2 (t1) of the inter-frame difference area f2 (t1) is created. The inter-frame difference area f2 (t1) is originally the same as that shown in F (t1) of FIG. 7, but here, for convenience, all the foot portions are missing as shown in the figure. The following explanation will be given as it has occurred. Therefore, the circumscribed rectangle ff2 (t1) for the inter-frame difference region f2 (t1) is a rectangle with a slightly reduced height that does not include the missing foot portion, as illustrated.

  In step S34 ′, the circumscribed rectangle ff2 (t1) is set as the h-th (h = 2) temporary motion individual index mm′2 (t1). FIG. 36B shows the provisional motion individual index mm′2 (t1) set in this way (“2” in “mm′2” corresponds to the motion individual number h = 2). As shown in the figure, this temporary motion individual index mm′2 (t1) is an incomplete rectangle with a part missing, and needs to be corrected.

  On the other hand, FIG. 36C shows a second binary image including the i-th (i = 1) background difference region d1 (t1) to be processed when i = 1 is set in step S35. It is shown. As shown in the figure, a circumscribed rectangle dd1 (t1) is created for the background difference area d1 (t1). Since the background difference area d1 (t1) is a complete individual area in which there is no chipping in the foot portion, the circumscribed rectangle dd1 (t1) is also a complete rectangle in which no chipping occurs.

  Then, as shown in FIG. 36 (d), it is determined whether or not these circumscribed rectangles ff2 (t1) and dd1 (t1) are overlapped (step S36 '). Specifically, based on the logical operation expression shown on the right side of FIG. 29 (b), the overlap existence determination operation for the circumscribed rectangles ff2 (t1) and dd1 (t1) may be performed. In the example shown in FIG. 36 (d), since there is no overlapping portion in both rectangles, VAL (Overlap) = 0 is obtained, and a determination result of “no overlapping portion” is obtained.

  This determination result means that the circumscribed rectangle dd1 (t1) shown in FIG. 36 (c) cannot be used for correction of the current temporary motion individual index mm′2 (t1) shown in FIG. 36 (b). Therefore, the process proceeds from step S38 ′ to step S40, and the correction process (the process of creating and replacing the inclusion rectangle) in step S39 ′ is not executed.

  Subsequently, in step S40, updating to i = 2 is performed, and the processing from step S35 is executed again. The lower part of FIG. 36 shows the processing contents when j = 2 and i = 2 in the flowchart of FIG. First, FIG. 36 (e) shows a binary image including the i-th (i = 2) background difference region d2 (t1) to be processed when i = 2 is set (FIG. 36). 5 (same as shown in D (t1)). For this background difference area d2 (t1), a circumscribed rectangle dd2 (t1) is created as shown.

  Then, as shown in FIG. 36 (f), it is determined whether or not there are overlaps with respect to the circumscribed rectangles ff2 (t1) and dd2 (t1) (step S36 ′). In the case of the example shown in FIG. 36 (f), since both rectangles overlap in the hatched portion, VAL (Overlap) = 1 is obtained, and a determination result that “there is an overlapping portion” is obtained.

  This determination result is corrected using the circumscribed rectangle dd2 (t1) shown in FIG. 36 (e) with respect to the current temporary motion individual index mm′2 (t1) shown in FIG. 36 (b). Means you can. Therefore, the process proceeds from step S38 ′ to step S39 ′, and correction processing (processing for creating and replacing an inclusion rectangle) is executed. That is, for the provisional motion individual index mm′2 (t1) and the circumscribed rectangle dd2 (t1), an inclusion rectangle is obtained, and the inclusion rectangle is used as a new provisional motion individual index mm′2 (t1).

  FIG. 36 (g) shows the new temporary motion individual index mm′2 (t1) created in this way. The broken lines in the new temporary motion individual indicator mm′2 (t1) indicate the outlines of two sets of rectangles that are the basis of the inclusion rectangle. The missing portion in the circumscribed rectangle constituting the old temporary motion individual indicator mm′2 (t1) shown in FIG. 36 (b) constitutes the new temporary motion individual indicator mm′2 (t1) shown in FIG. 36 (g). It can be seen that the inclusion rectangle is repaired.

  Thus, when the loop processing from step S35 to step S40 is completed from i = 1 to Imax, in step S41 ′, based on the current temporary motion individual index mm′2 (t1) shown in FIG. 36 (g). Then, a process of creating a formal motion individual index mm2 (t1) is performed. In this example, the formal motion individual index mm2 (t1) is composed of the same rectangular index as the provisional motion individual index mm′2 (t1), and the individual detection image Q (t1) − in FIG. This is the same as the motion individual index m2 (t1) shown in SM.

  In the case of the embodiment described in §6-3, if necessary, based on the inclusion rectangle mm1 (t1) shown in FIG. 35 (h) or the inclusion rectangle mm2 (t1) shown in FIG. 36 (g). It is possible to search for a humanoid region to be included therein and display a humanoid motion individual indicator m instead of a rectangle, but the humanoid motion individual indicator m displayed in this way is inappropriate. Since it may become an index, it is preferable to avoid it in practice.

  That is, in the embodiment described in §6-1 and 6-2, instead of displaying the circumscribed rectangle as an index as it is, the background difference area di (t2) included in the background difference image D (t) If an object located within the circumscribed rectangle is selected from among the images, and an index is created based on the selected background difference area di (t), the human-type stationary individual index s and the moving individual index m It is also possible to create an index related to variations as exemplified in FIGS.

  However, in the example described in §6-3, for example, the inclusion rectangle mm1 (t1) shown in FIG. 35 (h) is not a circumscribed rectangle of the specific background difference area di (t). The rectangle occupies a wider area. For this reason, the background difference area di (t) located in the inclusion rectangle mm1 (t1) cannot be accurately specified. Of course, the inclusion rectangle mm1 (t1) shown in FIG. 35 (h) is a rectangle including the inter-frame difference region f1 (t1) shown in FIG. 35 (a), but the inter-frame difference region f1 (t1) is missing. Therefore, it is not preferable to create the motion individual index m based on the incomplete individual shape including the portion that is not corrected.

  Therefore, practically, in the embodiment described in this §6-3, based on the inclusion rectangle finally obtained as the provisional motion individual index mm′h (t) as in the example shown in FIG. 35 (h). It is preferable to create and display a rectangular motion individual index mmh (t).

<<< §7. Monitoring system using an individual detection apparatus according to the present invention >>>
The embodiments described so far are examples in which a person is used as an individual to be detected. Of course, the present invention uses various objects such as animals, vehicles, ships, aircrafts, various mechanical devices, and toys. It can be used for the purpose of detecting as an individual.

  Further, in an environment where individuals to be detected are densely packed, a plurality of individuals overlap on the original image P (t), so that the background difference region di (t) and the inter-frame difference region fi (t) There is a possibility that the fusion area includes the individual, but in performing the above-described processing, for example, even if two persons holding hands are recognized as one fusion individual, no trouble occurs. In this case, the motion individual index m indicates two persons holding hands as one motion individual.

  Therefore, the individual detection apparatus 100 shown in FIG. 1 can be used as a monitoring system for monitoring various objects under various environments. FIG. 37 is a block diagram showing an example of a monitoring system using the individual detection apparatus according to the present invention. As illustrated, the monitoring system includes a client system 200 and a server system 400 connected to the client system 200 via a network 300 (for example, the Internet).

  As illustrated, the client system 200 includes a fixed point camera 210, a display device 220, and a client computer 230, and the server system 400 includes a server computer 410. The client computer 230 and the server computer 410 can communicate with each other via the network 300.

  The fixed point camera 210 has a function of acquiring images in units of frames arranged in time series by performing moving image shooting on a predetermined shooting target region, and the frame unit image acquired by the fixed point camera 210. Is given to the client computer 230 as the original image P (t) described in the above description. The client computer 230 inputs the original image P (t) given from the fixed point camera 210 and executes a predetermined process. The display device 220 displays an image based on an instruction from the client computer 230. On the other hand, the server computer 410 executes predetermined processing while communicating with the client computer 230 via the network 300.

  A dedicated client computer program is installed in the client computer 230, and a part of the processing as the individual detection apparatus 100 shown in FIG. 1 is executed by an operation based on the client computer program. On the other hand, a dedicated server computer program is installed in the server computer 410, and other processes (client computer) as the individual detection apparatus 100 shown in FIG. 1 are performed by operations based on the server computer program. Processing that is not performed on the 230 side is executed.

  Due to the function of the client computer 230, an image captured by the fixed point camera 210 (original image P (t)) is displayed in real time on the display screen of the display device 220, and information indicating the recognition result by the individual recognition unit 160 (for example, Then, a stationary individual index s indicating an individual recognized as a stationary individual and a moving individual index m) indicating an individual recognized as a moving individual are displayed. A monitoring person who uses this monitoring system visually identifies a stationary object and a moving object appearing on the captured image in real time based on the captured image displayed on the display screen of the display device 220 and information indicating the recognition result. Can be distinguished and understood.

  In addition, the monitor can give an instruction to the operation of the individual detection device 100 by giving some operation input to the client computer 230 as necessary. For example, the display mode can be switched as illustrated in FIG. As described above, for the monitor who uses this monitoring system, the client system 200 is a direct operation target. However, the server system 400 plays a role of providing a cloud environment, and a part of the client system 200 is replaced. Can be executed.

  The role of each component of the individual detection apparatus 100 shown in FIG. 1 is shared between the client system 200 and the server system 400 depending on the use of the monitoring system and information on the image captured by the fixed point camera 210. However, in general, the client system 200 is in charge of processing that requires high speed in order to perform real-time processing. Thus, it is preferable that the server system 400 is in charge of processing that is too computationally intensive to be processed by the client computer 230.

  In the monitoring system shown in FIG. 37, the client computer 230 has a function of issuing an alarm to the outside as shown. In the case of the example shown in the figure, the warning is given to the supervisor, and notification is made by lighting the lamp or sounding the buzzer. Of course, an alarm may be transmitted to the server system 400 and other places via the network 300.

  This alarm is issued based on the recognition result of the stationary individual or the moving individual by the individual recognition unit 160, and is given to the outside when a preset reporting condition is satisfied. For example, a notification condition such as “when the total number of stationary individuals exceeds a predetermined reference value” or “when the total number of moving individuals exceeds a predetermined reference value” is set, If a function for counting the total number of stationary and moving individuals recognized at the time is provided, an alarm can be issued to the outside when the above-described reporting conditions are satisfied. Of course, “when the total number of stationary individuals exceeds a predetermined reference value continues for a predetermined reference time” or “when the total number of moving individuals exceeds a predetermined reference value continues for a predetermined reference time” It is also possible to set a notification condition including time.

  The monitoring system shown in FIG. 37 is configured by the client system 200 and the server system 400, and is a system that executes the processing necessary for the individual detection device 100 shown in FIG. The client computer 230 shown in the figure may execute all necessary processes, and only the client computer 230 may function independently as the individual detection apparatus 100 shown in FIG.

  In this case, the client computer 230 is not necessarily connected to the network 300, and may be a so-called stand-alone computer. In this case, the monitoring system performs moving image shooting on a predetermined shooting target region, thereby obtaining a fixed point camera 210 that acquires images in frame units arranged in time series, and an image in frame units acquired by the fixed point camera. The computer 230 is input as an original image and executes predetermined processing, and the display device 220 that displays an image based on an instruction of the computer 230. The computer 230 is an installed program. By the operation based on, it functions as the individual detection apparatus 100 shown in FIG.

  A client system 200 shown in FIG. 37 is a monitoring system having a function of immediately processing an image photographed by the fixed point camera 210 by the client computer 230 and displaying the result on the display device 220 in real time. If such a monitoring system is used, the monitoring staff can perform monitoring in real time. However, in practice, this kind of real-time monitoring is not necessarily required, and a task of checking the presence or absence of a suspicious person while confirming past captured images later (here, for the sake of convenience In some cases.

  In order to perform such post-monitoring, for example, in the client system 200 shown in FIG. 37, instead of the fixed point camera 210, moving image data (moving image shooting is performed on a predetermined shooting target area). The monitoring system may be configured using the input image storage device that stores the images of the frame units arranged in time series obtained by the above. As an input image storage device, a video recording / playback device or a hard disk device for a computer may be used. The client computer 230 inputs the frame unit image stored in the input image storage device as the original image instead of inputting the frame unit image from the fixed point camera 210 as the original image and executing a predetermined process. Thus, a predetermined process is executed.

  For example, in the client system 200 shown in FIG. 37, if a post-mortem monitoring system is configured by replacing the fixed point camera 210 with an input image storage device (for example, a video recording / playback device), the video recording / playback device can By reproducing the captured moving image and processing it by the client computer 230, an individual detection image obtained by superimposing a stationary individual index or a moving individual index on the original image (reproduced moving image) is displayed on the display device 220. Since it can be displayed, it can be used for finding a suspicious person existing on a past photographed image, although it is not in real time.

  Of course, if both the fixed point camera 210 and the input image storage device are provided, the client computer 230 can perform real-time monitoring based on the image input from the fixed point camera 210, and the image storage device. It is also possible to perform post-mortem monitoring based on the image input from.

  Further, instead of the display device 220, an image storage device for output that stores an image given from the client computer 230 is provided, so that it can be used for post-monitoring. For example, in the client system 200 shown in FIG. 37, if the post-mortem monitoring system is configured by replacing the display device 220 with an output image storage device, the client computer 230 is based on the image taken by the fixed point camera 210. Thus, an individual detection image can be created, and a moving image composed of the individual detection image can be stored in an output image storage device instead of being displayed on the display device 220.

  For example, if a video recording / playback device is used as an output image storage device and a moving image composed of individual detection images output from the client computer 230 is stored in an optical information recording medium such as a DVD-R, the storage is performed. Since the moving image is obtained by superimposing the stationary individual index and the moving individual index on the video captured by the fixed point camera 210, it is monitored by the video recording / reproducing apparatus completely different from the system shown in FIG. It is also possible to do business.

  Of course, if both the display device 220 and the output image storage device are provided, the client computer 230 can display the output image on the display device 220 or can store the output image in the output image storage device. You can also.

  Further, both an input image storage device and an output image storage device may be provided. In this case, in practice, one image storage device can be shared as an input image storage device and an output image storage device. Alternatively, an image storage device for input and an image storage device for output are provided on the server system 400 side instead of the client system 200 side, and the client computer 230 inputs the original image via the network 300, and the processed individual The detected image may be output to the server system 400 via the network 300 and stored.

100: individual detection device 110: original image input unit 120: background image storage unit 130: background difference image creation unit 140: inter-frame difference image creation unit 150: overlap determination unit 160: individual recognition unit 161: stationary individual recognition unit 162: Motion individual recognition unit 170: Detected individual display unit 200: Client system 210: Fixed point camera 220: Display device 230: Client computer 300: Network (Internet)
400: Server system 410: Server computer A, A0 to A4: First individual (adult)
a (xa, ya): pixels B, B0 to B4: second individual (child)
BG: Background image Bu, Bv: Pixel value b (xb, yb) of blue pixel: Pixels C (t), C (t1) to C (t4): Overlapping confirmation images c1 (t1) to c2 (t4): Overlapping Partial region c (xc, yc): Pixels D (t), D (t1) to D (t4): Background difference images d1 (t), d2 (t): Background difference regions d1 (t1) to d2 (t4) : Background difference area di (t): Background difference area dd1 (t1), dd1 (t2), dd2 (t1), dd2 (t2): circumscribed rectangle of background difference area ddi (t): circumscribed rectangle F of background difference area (T), F (t1) to F (t4): Inter-frame difference images f1 (t), f2 (t), f3 (t): Inter-frame difference areas f1 (t1) to f2 (t4): Inter-frame differences Region fj (t): Inter-frame difference region ff1 (t1), ff1 (t2), ff2 (t1 : Circumscribing rectangle fi (t) of inter-frame difference area: circumscribing rectangle Gu, Gv: inter-frame difference area pixel value h: motion individual index number IMG (AND): logical product image IMG (OR): logical sum Image Imax: Maximum value of background difference area number i: Background difference area number Jmax: Maximum value of interframe difference area number j: Interframe difference area number k: Stationary individual index number L: Labeling information m, m1 (t1), m2 (t1), m1 (t2), m1 (t4): Motion individual index mh (t): Motion individual index m′1 (t1), m′2 (t1): Temporary motion individual index m′h (t) : Temporary motion individual index mm1 (t1), mm1 (t2), mm2 (t1): Motion individual index mmh (t): Motion individual index mm'1 (t1), mm'2 (t1): Temporary motion individual index mm ′ H (t): provisional motion individual index n: Frame number P (t), P (t0) ~P (t4): the original image (image of the frame unit lined up in chronological order)
Q (t): individual detection image Q (t) -M: individual detection image Q (t1) -M: individual detection image Q (t2) -M: individual detection image Q (t3) ) -M: individual detection image Q (t4) of the moving individual M: individual detection image Q (t) of the moving individual S: individual detection image Q (t1) of the stationary individual S: individual detection image Q of the stationary individual (T2) -S: individual detection image Q (t3) -S: stationary individual detection image Q (t4) -S: stationary individual detection image Q (t) -SM: stationary individual / moving individual Individual detection image Q (t1) -SM: stationary individual / moving individual detection image Q (t2) -SM: stationary individual / moving individual detection image Q (t3) -SM: stationary individual / moving individual Detected image Q (t4) -SM: Individual detected image R, Ra, Rb, Rc of stationary and moving individuals Rectangles Ru, Rv: Pixel values S11 to S42 of red pixels: Steps s, s1 (t2), s1 (t3), s2 (t3), s1 (t4) of the flowchart: Stationary individual index sk (t): Stationary individual Index ss1 (t2): Stationary individual index ssk (t): Stationary individual index t, t0 to t4: Time U: Color image U ′: Luminance image V: Color image V ′: Luminance image VAL (OR): Total OR VAL (Overlap): rectangle overlap determination value W: binarized difference image W ′: difference image w1, w2: difference area x1 to x4: X coordinate value xmax: maximum value of X coordinate xmin: minimum value Yu of X coordinate Yv: luminance values y1 to y4: Y coordinate value ymax: maximum value of Y coordinate ymin: minimum value of Y coordinate ΔY: absolute value of difference in luminance value

Claims (58)

An individual detection device for detecting an individual existing in a predetermined imaging target region as a stationary individual at rest or an operating individual at motion,
An original image input unit for inputting each original image obtained as an image in frame units arranged in time series by photographing with respect to a predetermined photographing target area;
A background image storage unit that stores a background image of the imaging target region;
For each original image input by the original image input unit, a background difference image creation unit that creates a background difference image including a background difference region indicating a difference from the background image;
For each original image input by the original image input unit, an inter-frame difference image creating unit that creates an inter-frame difference image including an inter-frame difference region indicating a difference from different frames in time series, and
For each background difference area in the background difference image, an overlap presence / absence determination unit that determines the presence / absence of an overlapping portion with respect to the inter-frame difference area in the inter-frame difference image;
Based on the determination result by the duplication presence / absence determination unit, an individual recognition unit that recognizes an individual included in the original image as a stationary individual or a moving individual;
A device for detecting an individual in a photographed image. In the detection apparatus of the individual | organism | solid in the picked-up image of Claim 1,
The individual recognition unit recognizes an individual occupying the background difference region, which is obtained as a result of the determination that there is no substantial overlap with the inter-frame difference region, by the duplication presence / absence determination unit as a stationary individual. An apparatus for detecting an individual in a photographed image, comprising: In the detection apparatus of the individual | organism | solid in the picked-up image of Claim 2,
An apparatus for detecting an individual in a photographed image, further comprising: a detected individual display unit that displays a stationary individual index indicating an individual recognized as a stationary individual by the stationary individual recognition unit. In the detection apparatus of the individual | organism | solid in the picked-up image of Claim 3,
An apparatus for detecting an individual in a photographed image, wherein the detected individual display unit creates a stationary individual index indicating a stationary individual based on a background difference area occupied by the stationary individual. In the detection apparatus of the individual | organism | solid in the picked-up image of Claim 1,
The individual recognition unit recognizes an individual that occupies a background difference region from which the determination result that at least a part of the region has a substantial overlap portion with respect to the inter-frame difference region is obtained by the overlap presence / absence determination unit as an operation individual An apparatus for detecting an individual in a photographed image, characterized by comprising an action individual recognition unit. In the detection apparatus of the individual | organism | solid in the picked-up image of Claim 5,
An apparatus for detecting an individual in a photographed image, further comprising a detected individual display unit that displays an operating individual index indicating an individual recognized as an operating individual by the operating individual recognition unit. In the detection apparatus of the individual | organism | solid in the picked-up image of Claim 6,
An apparatus for detecting an individual in a captured image, wherein the detected individual display unit creates an operating individual index indicating the operating individual based on a background difference area occupied by the operating individual. In the detection apparatus of the individual | organism | solid in the picked-up image of Claim 6,
The detected object display unit creates an action object index indicating an action object by performing correction based on a background difference area occupied by the action object on an inter-frame difference area occupied by the action object. A device for detecting an individual in a captured image. In the detection device of the individual in the photographed image according to claim 8,
The detected individual display unit performs correction by performing a logical OR operation on a figure constituting the region for the inter-frame difference region occupied by the moving individual and the background difference region occupied by the moving individual. A device for detecting an individual in an image. In the detection apparatus of the individual | organism | solid in the picked-up image of Claim 1,
The individual recognition unit
A stationary individual recognizing unit that recognizes an individual occupying a background differential region obtained as a result of determination that there is no substantial overlapping portion with respect to the inter-frame difference region by the duplication presence / absence determining unit,
The motion individual recognition unit that recognizes an individual occupying the background difference region from which the determination result that at least a part of the region has a substantial overlap with the inter-frame difference region is obtained by the overlap presence / absence determination unit as a motion individual When,
A device for detecting an individual in a photographed image. In the detection apparatus of the individual | organism | solid in the picked-up image of Claim 10,
A detected individual display unit for displaying a stationary individual index indicating an individual recognized as a stationary individual by a stationary individual recognition unit and a moving individual index indicating an individual recognized as a moving individual in a visually distinguishable manner. An apparatus for detecting an individual in a photographed image, further comprising: In the detection apparatus of the individual | organism | solid in the picked-up image of Claim 11,
The detected individual display unit creates a stationary individual index indicating a stationary individual based on the background difference area occupied by the stationary individual, and creates an operating individual index indicating the moving individual based on the background difference area occupied by the moving individual. An apparatus for detecting an individual in a photographed image. In the detection apparatus of the individual | organism | solid in the picked-up image of Claim 11,
The detected individual display unit creates a stationary individual index indicating a stationary individual based on the background difference area occupied by the stationary individual, and an operating individual index indicating the moving individual is calculated with respect to the inter-frame difference area occupied by the moving individual. An apparatus for detecting an individual in a captured image, which is created by performing correction based on a background difference area occupied by the moving individual. In the detection apparatus of the individual | organism | solid in the picked-up image of Claim 13,
The detected individual display unit performs correction by performing a logical OR operation on a figure constituting the region for the inter-frame difference region occupied by the moving individual and the background difference region occupied by the moving individual. A device for detecting an individual in an image. In the detection apparatus of the individual | organism | solid in the picked-up image in any one of Claims 3, 4, 6-9,
A detected individual display unit creates an individual detected image by superimposing a stationary individual index or a moving individual index on an original image, and displays the individual detected image on a display screen. Detection device. In the detection device of the individual in the picked-up image according to claim 15,
The detected individual display unit is arranged in the vicinity of the area occupied by the target individual with the stationary individual index or the moving individual index, and the inside is painted with a predetermined color or the closed area index in which the predetermined pattern is formed An apparatus for detecting an individual in a captured image, comprising: creating an individual detection image by superimposing the closed region index on an original image. In the device for detecting an individual in a captured image according to claim 16,
The closed region index is configured by a closed region that is painted in a semi-transparent color or has a partially spaced pattern, and when the closed region index is superimposed on the original image, the original image is A device for detecting an individual in a captured image, wherein the device is displayed through a region index. In the detection device of the individual in the picked-up image according to claim 15,
The detected individual display unit comprises a stationary individual index or a moving individual index by a frame-shaped index surrounding a target individual, and creates an individual detection image by superimposing the frame-shaped index on an original image. A device for detecting an individual in a captured image. In the detection device of the individual in the picked-up image according to claim 15,
The detected individual display unit includes a stationary individual index or a moving individual index by a contour index indicating a contour line of a target individual, and creates an individual detection image by superimposing the contour index on an original image. A device for detecting an individual in a photographed image. In the detection device of the individual in the picked-up image according to claim 15,
The detected individual display unit includes a stationary individual index or a moving individual index, a mask image in which an aperture window is arranged in a target individual portion and a portion other than the aperture window is covered, and the mask image is included in an original image. An apparatus for detecting an individual in a captured image, wherein an individual detection image is created by superimposing the individual detection images. In the detection apparatus of the individual | organism | solid in the picked-up image in any one of Claims 11-14,
The detected individual display unit creates an individual detected image by superimposing a stationary individual index and a moving individual index on the original image, and displays the individual detected image on a display screen. Detection device. In the detection device of the individual in the picked-up image according to claim 21,
The detected individual display unit is configured by a first closed region index in which a stationary individual index is arranged in the vicinity of a region occupied by the target individual and the inside is painted with a predetermined color or a predetermined pattern is formed inside The moving individual indicator is configured by a second closed region indicator which is arranged in the vicinity of the region occupied by the target individual and whose interior is painted with a predetermined color or has a predetermined pattern formed therein, The closed region index of 1 and the second closed region index are closed regions in which different colors are applied or different patterns are formed, and the first closed region index and the An apparatus for detecting an individual in a captured image, wherein an individual detection image is created by superimposing a second closed region index. In the detection apparatus of the individual | organism | solid in the picked-up image of Claim 22,
The first closed region index and the second closed region index are constituted by closed regions that are painted in a semi-transparent color or partially formed with a gap pattern, and each closed region index is added to the original image. An apparatus for detecting an individual in a photographed image, wherein when superimposed, an original image is displayed through each closed region index. In the detection device of the individual in the picked-up image according to claim 21,
The detected individual display unit configures the stationary individual index by a first frame-shaped index surrounding the target stationary individual, and configures the moving individual index by the second frame-shaped index surrounding the target moving individual, The first frame-shaped index and the second frame-shaped index are configured by frame lines having at least one of line type, color, shape, and thickness different from each other, and the first frame-shaped index is included in the original image. A device for detecting an individual in a photographed image, wherein an individual detection image is created by superimposing the frame-shaped index and the second frame-shaped index. In the detection device of the individual in the picked-up image according to claim 21,
The detected individual display unit configures the stationary individual index by the first contour index indicating the contour line of the target stationary individual, and the moving individual index indicates the second contour index indicating the contour line of the target moving individual. The first contour index and the second contour index are configured by contour lines that are different from each other in at least one of line type, color, and thickness, and the first image is included in the original image. A device for detecting an individual in a photographed image, wherein an individual detection image is created by superimposing the second contour index and the second contour index. In the detection apparatus of the individual | organism | solid in the picked-up image in any one of Claims 21-25,
The detected individual display section
An original image display mode for displaying the original image on the display screen;
An all-individual display mode for displaying an individual detection image in which a stationary individual index, a motion individual index, and an original image are superimposed on a display screen;
A stationary individual display mode for displaying an individual detection image in which a stationary individual index and an original image are superimposed on a display screen;
An action individual display mode for displaying an individual detection image in which an action individual indicator and an original image are superimposed on a display screen;
A display device for detecting an individual in a photographed image, which has a display function based on the following four display modes and performs display by switching to an arbitrary display mode. In the detection apparatus of the individual | organism | solid in the picked-up image in any one of Claims 1-26,
The background image storage unit has a function of creating and storing a background image based on a plurality of frame unit images input in the past by the original image input unit,
An apparatus for detecting an individual in a captured image, wherein a background difference image creation unit creates a background difference image using the latest background image stored in the background image storage unit. In the detection apparatus of the individual | organism | solid in the picked-up image in any one of Claims 1-27,
The original image input unit samples and inputs images in frame units arranged in time series given from the outside at a predetermined sampling period,
An inter-frame difference image creating unit creates an inter-frame difference image by performing a difference operation on the sampled image of each frame unit with the immediately preceding image in time series. Individual detection device. In the device for detecting an individual in a photographed image according to any one of claims 1 to 28,
The original image input unit inputs a color image composed of a set of pixels having pixel values of the three primary colors RGB as an original image, and converts the color image into a luminance image composed of a set of pixels having a pixel value indicating luminance Y. Process to convert,
The background image storage unit stores a background image composed of a collection of pixels having pixel values indicating luminance Y,
A background difference image creation unit binarizes a difference image composed of a collection of pixels having pixel values indicating an absolute value ΔY of a difference between luminance values of the luminance image and the background image using a predetermined threshold value. To create a background difference image,
The inter-frame difference image creation unit binarizes a difference image composed of a set of pixels having pixel values indicating an absolute value ΔY of a difference in luminance value between two frame-unit images using a predetermined threshold value. A device for detecting an individual in a photographed image, characterized in that an inter-frame difference image is created by converting into an image. The device for detecting an individual in a captured image according to claim 29,
The background difference image creation unit gives a pixel value “1” for a pixel whose absolute value ΔY of the luminance value difference is equal to or greater than the threshold value, and a pixel value “1” for a pixel whose absolute value ΔY of the luminance value difference is less than the threshold value. By giving “0”, a background difference image including a continuous area of pixels having a pixel value “1” as a background difference area is created,
The inter-frame difference image creation unit gives a pixel value “1” for a pixel for which the absolute value ΔY of the luminance value difference is greater than or equal to the threshold value, and a pixel value for the pixel for which the absolute value ΔY of the luminance value difference is less than the threshold value An apparatus for detecting an individual in a photographed image, characterized in that, by giving “0”, an inter-frame difference image including a continuous area of pixels having a pixel value “1” as an inter-frame difference area is created. In the detection apparatus of the individual | organism | solid in the picked-up image in any one of Claims 1-30,
Based on the background difference image and the inter-frame difference image, the duplication presence / absence determining unit creates an overlap confirmation image including an overlapping partial region indicating an overlapping portion of the background difference region and the inter-frame difference region,
The individual recognition unit refers to a corresponding area on the overlap confirmation image for a specific background difference area included in the background difference image, and the overlapping partial area is not substantially included in the corresponding area. In this case, an individual occupying the specific background difference area is recognized as a stationary individual, and if the overlapping partial area is substantially included in the corresponding area, the individual occupying the specific background difference area A device for detecting an individual in a captured image, characterized in that In the detection apparatus of the individual | organism | solid in the picked-up image of Claim 31,
When the area of the overlapping partial region included in the corresponding region does not satisfy a predetermined criterion, the individual recognition unit determines that the overlapping partial region is not included in the corresponding region. A device for detecting an individual in a captured image. The apparatus for detecting an individual in a captured image according to claim 32,
The background difference image creation unit is a binary image composed of a collection of pixels having a pixel value “1” or “0”, and represents the background difference region as a continuous region of pixels having the pixel value “1”. Created background difference image,
The inter-frame difference image creation unit is a binary image composed of a collection of pixels having a pixel value “1” or “0”, and the inter-frame difference area is a continuous area of pixels having the pixel value “1”. Create an inter-frame difference image that represents
The duplication presence / absence determination unit includes a set of pixels having a pixel value “1” or “0” by performing an AND operation on corresponding pixels of the background difference image and the inter-frame difference image. Create a duplicate confirmation image that is a binary image,
The individual recognition unit refers to a corresponding region on the overlap confirmation image for a specific background difference region included in the background difference image, and has a pixel value “1” among the pixels in the corresponding region. When the total number of pixels does not satisfy a predetermined criterion, the individual occupying the specific background difference area is recognized as a stationary individual, and the total number of pixels having the pixel value “1” among the pixels in the corresponding area When the value is equal to or more than a predetermined reference, an individual occupying the specific background difference area is recognized as an operating individual, and an individual detection apparatus in a captured image is characterized. In the detection apparatus of the individual | organism | solid in the picked-up image in any one of Claims 1-30,
A background difference area including a pixel difference information that the background difference image creating unit assigns a background difference area code to each background difference area and identifies pixels included in the background difference area corresponding to each background difference area code Process to create labeling information for
The inter-frame difference image creating unit assigns an inter-frame difference area code for each inter-frame difference area, and specifies pixel specifying information for specifying a pixel included in the inter-frame difference area corresponding to each inter-frame difference area code. Including the process of creating the labeling information of the inter-frame difference area,
When the overlapping presence / absence determining unit determines the presence / absence of an overlapping portion for a predetermined target background difference region, the inter-frame difference is determined for each target pixel specified by the pixel specifying information in the labeling information of the target background difference region. A device for detecting an individual in a photographed image, which checks whether or not there is a pixel at the same position in the pixel specifying information in the labeling information of the region, and determines that there is an overlapping portion if it exists. In the detection apparatus of the individual | organism | solid in the picked-up image in any one of Claims 1-30,
When the overlap presence / absence determination unit determines the presence / absence of an overlapping portion with respect to the inter-frame difference area in the inter-frame difference image for the background difference area in the background difference image, the circumscribed rectangle of the background difference area and the inter-frame difference area A device for detecting an individual in a photographed image, wherein a simple determination method for determining the presence or absence of an overlapping portion with respect to a circumscribed rectangle is performed. In the detection apparatus of the individual | organism | solid in the picked-up image of Claim 35,
The individual recognition unit recognizes the individual occupying the background difference area from which the determination result is obtained that there is no overlapping part about the circumscribed rectangle by the simple determination method performed by the duplication presence / absence determination unit as a stationary individual,
In a captured image, further comprising a detected individual display unit that creates a stationary individual index indicating an individual recognized as a stationary individual based on a circumscribed rectangle of a background difference area occupied by the stationary individual and displays the rectangle Individual detection device. In the detection apparatus of the individual | organism | solid in the picked-up image of Claim 35,
The individual recognition unit recognizes the individual occupying the background difference area from which the determination result is obtained that there is an overlapping part about the circumscribed rectangle by the simple determination method performed by the duplication presence / absence determination unit, as an operation individual,
In a captured image, further comprising: a detected individual display unit that creates an operating individual index indicating an individual recognized as an operating individual based on a circumscribed rectangle of a background difference area occupied by the operating individual, and displays the rectangle Individual detection device. In the detection apparatus of the individual | organism | solid in the picked-up image of Claim 35,
The individual recognition unit recognizes an individual that occupies the inter-frame difference area obtained as a result of determination that there is an overlapping part for the circumscribed rectangle by the simple determination method performed by the duplication presence / absence determination unit, as an operation individual,
The motion individual index indicating the individual recognized as the motion individual is created by performing correction based on the circumscribing rectangle of the background difference region occupied by the motion individual on the circumscribing rectangle of the inter-frame difference region occupied by the motion individual. An apparatus for detecting an individual in a photographed image, further comprising a detected individual display unit for displaying the detected individual. In the detection device of the individual in the picked-up image according to claim 38,
The detected individual display unit obtains an inclusion rectangle that includes both circumscribed rectangles for the circumscribed rectangle of the inter-frame difference area occupied by the moving individual and the circumscribed rectangle of the background difference area occupied by the moving individual, and based on the obtained inclusion rectangle An apparatus for detecting an individual in a photographed image, characterized by creating an action individual index.   A program that causes a computer to function as the individual detection apparatus according to any one of claims 1 to 39. A monitoring system in which the individual detection device according to any one of claims 1 to 39 is incorporated,
A fixed-point camera that acquires images in frame units arranged in time series by performing moving image shooting on a predetermined shooting target area;
A computer for inputting a frame unit image acquired by the fixed point camera as an original image and executing a predetermined process;
A display device for displaying an image based on an instruction from the computer;
With
The monitoring system, wherein the computer functions as the individual detection device by an operation based on an installed program. A monitoring system in which the individual detection device according to any one of claims 1 to 39 is incorporated,
A client system and a server system connected to each other via a network;
The client system is
A fixed-point camera that acquires images in frame units arranged in time series by performing moving image shooting on a predetermined shooting target area;
A client computer that inputs a frame-unit image acquired by the fixed-point camera as an original image and executes a predetermined process;
A display device that displays an image based on an instruction of the client computer,
The server system is
A server computer that executes predetermined processing while performing communication with the client computer via the network,
The client computer performs a part of processing necessary as the individual detection device by an operation based on the installed client computer program,
The monitoring system, wherein the server computer executes another part of processing necessary as the individual detection device by an operation based on an installed server computer program.   43. A client computer program installed on a client computer or a server computer program installed on a server computer in the monitoring system according to claim 42. The individual monitoring system according to claim 41 or 42,
A monitoring system in which a computer or a client computer issues an alarm to the outside based on a recognition result of a stationary object or a moving object. The individual monitoring system according to claim 41,
An image storage device for input that stores images in frames arranged in time series obtained by performing moving image shooting on a predetermined shooting target area instead of or in addition to the fixed point camera. With
A monitoring system, wherein a computer inputs an image of a frame unit stored in the input image storage device as an original image and executes predetermined processing. The individual monitoring system according to claim 41 or 45,
In addition to or in addition to the display device, an image storage device for output for storing an image given from a computer is provided.
A monitoring system in which a computer outputs an image to be output to a display device to the output image storage device to be stored. Among individuals existing within a predetermined imaging target area, an individual detection device that detects a stationary individual that is stationary,
An original image input unit for inputting an original image obtained as an image in frame units arranged in time series by photographing with respect to a predetermined photographing target region;
An individual recognition unit for recognizing whether each individual appearing on the original image is a stationary individual or a moving individual;
Based on the recognition result by the individual recognition unit, a detected individual display unit that superimposes and displays a stationary individual index indicating an individual recognized as a stationary individual on the original image;
A device for detecting an individual in a photographed image. An individual detection device for detecting an operating individual among individuals existing in a predetermined imaging target region,
An original image input unit for inputting an original image obtained as an image in frame units arranged in time series by photographing with respect to a predetermined photographing target region;
An individual recognition unit for recognizing whether each individual appearing on the original image is a stationary individual or a moving individual;
Based on the recognition result by the individual recognition unit, a detected individual display unit that superimposes and displays an operation individual index indicating an individual recognized as an operation individual on the original image;
A device for detecting an individual in a photographed image. In the detection apparatus of the individual in the picked-up image of Claim 47 or 48,
The detected individual display unit is arranged in the vicinity of the area occupied by the target individual with the stationary individual index or the moving individual index, and the inside is painted with a predetermined color or the closed area index in which the predetermined pattern is formed An apparatus for detecting an individual in a captured image, wherein the device is configured to display the closed region index superimposed on an original image. The apparatus for detecting an individual in a captured image according to claim 49,
The closed region index is configured by a closed region that is painted in a semi-transparent color or has a partially spaced pattern, and when the closed region index is superimposed on the original image, the original image is A device for detecting an individual in a captured image, wherein the device is displayed through a region index. In the detection apparatus of the individual in the picked-up image of Claim 47 or 48,
The detected individual display unit configures a stationary individual index or a moving individual index with a frame-shaped index surrounding a target individual and having a predetermined line type, color, shape, and thickness, and the frame-shaped index is displayed on an original image. An apparatus for detecting an individual in a captured image, wherein the individual image is displayed in a superimposed manner. In the detection apparatus of the individual in the picked-up image of Claim 47 or 48,
The detected individual display unit displays the stationary individual index or the moving individual index by indicating a contour line of the target individual, and is configured by a contour index having a predetermined line type, color, and thickness, and the contour index is superimposed on the original image A device for detecting an individual in a photographed image, characterized by being displayed. An individual detection device for detecting an individual existing in a predetermined imaging target region as a stationary individual at rest or an operating individual at motion,
An original image input unit for inputting an original image obtained as an image in frame units arranged in time series by photographing with respect to a predetermined photographing target region;
An individual recognition unit that recognizes each individual appearing on the original image as a stationary individual or a moving individual,
Based on the recognition result by the individual recognition unit, on the original image, a stationary individual index indicating an individual recognized as a stationary individual and a moving individual index indicating an individual recognized as a moving individual are superimposed, And the detected individual display unit that displays the stationary individual index and the moving individual index in such a manner as to be visually distinguishable,
A device for detecting an individual in a photographed image. The apparatus for detecting an individual in a captured image according to claim 53,
The detected individual display unit is configured by a first closed region index in which a stationary individual index is arranged in the vicinity of a region occupied by the target individual and the inside is painted with a predetermined color or a predetermined pattern is formed inside The moving individual indicator is configured by a second closed region indicator which is arranged in the vicinity of the region occupied by the target individual and whose interior is painted with a predetermined color or has a predetermined pattern formed therein, The closed region index of 1 and the second closed region index are closed regions in which different colors are applied or different patterns are formed, and the first closed region index and the A device for detecting an individual in a photographed image, wherein the second closed region index is superimposed and displayed. In the detection device of the individual in a photography picture according to claim 54,
The first closed region index and the second closed region index are constituted by closed regions that are painted in a semi-transparent color or partially formed with a gap pattern, and each closed region index is added to the original image. An apparatus for detecting an individual in a photographed image, wherein when superimposed, an original image is displayed through each closed region index. The apparatus for detecting an individual in a captured image according to claim 53,
The detected individual display unit configures the stationary individual index by a first frame-shaped index surrounding the target stationary individual, and configures the moving individual index by the second frame-shaped index surrounding the target moving individual, The first frame-shaped index and the second frame-shaped index are configured by frame lines having at least one of line type, color, shape, and thickness different from each other, and the first frame-shaped index is included in the original image. An apparatus for detecting an individual in a photographed image, wherein the frame-shaped index and the second frame-shaped index are superimposed and displayed. The apparatus for detecting an individual in a captured image according to claim 53,
The detected individual display unit configures the stationary individual index by the first contour index indicating the contour line of the target stationary individual, and the moving individual index indicates the second contour index indicating the contour line of the target moving individual. The first contour index and the second contour index are configured by contour lines that are different from each other in at least one of line type, color, and thickness, and the first image is included in the original image. A device for detecting an individual in a photographed image, wherein the contour index and the second contour index are superimposed and displayed.   A program that causes a computer to function as the individual detection apparatus according to any one of claims 47 to 57.

Images