JP6851246B2 - Object detector - Google Patents

Description

The present invention relates to an object detection device that detects an individual object from a photographed image in which a space in which a predetermined object such as a person can exist is photographed. In particular, the present invention relates to an individual object from a photographed image in which a space where congestion may occur is photographed. The present invention relates to an object detection device for detecting.

In spaces where congestion may occur, such as event venues, it is necessary to take measures such as allocating a large number of security guards in areas where congestion is occurring in order to prevent accidents. Therefore, by arranging surveillance cameras at various places in the venue, estimating the distribution of people from the captured image, and displaying the estimated distribution, it is possible to facilitate the grasp of the congestion situation by the observer.

At that time, the position of each person is detected, a model imitating the shape of the person is displayed at each detected position, or / and the positional relationship of the person (for example, forming a matrix or surrounding) is displayed. Further improvement in monitoring efficiency can be expected by analyzing and notifying the analysis result.

A method of detecting the position of an individual person from a photographed image taken by multiple people, a method of applying a combination of multiple models imitating a person to the photographed image, and a method of preliminarily determining the feature amount of an image taken by a single person. There is a method of detecting the position where the image feature of a single person appears from the photographed image by using the image feature of a single person prepared in advance, such as a method of scanning the photographed image using the discriminator learned in 1.

For example, in the moving object tracking device described in Patent Document 1, a moving object model that imitates the shape of the moving object being tracked is being tracked at a position where change pixels are extracted by comparing a surveillance image and a background image. The position of each moving object is detected by combining and applying as many as the number of moving objects. In this moving object tracking device, it is exemplified to use a moving object model that approximates the shape of the whole body of a person.

Further, for example, the object detection device described in Patent Document 2 detects a person from an input image using a classifier trained in advance using a large number of "human" image data and "non-human" image data. .. It is suggested that the classifier used by this object detection device was learned using image data of the whole body of a person.

JP 2012-159858 JP 2011-186633

However, in a photographed image in which a space where congestion can occur is taken, the concealed state of a person changes according to the congestion state. Therefore, as an image feature of a single person, if the image feature of each part of the object is always evaluated uniformly regardless of the congestion state, there is a problem that it becomes difficult to continuously detect each person with high accuracy. ..

That is, for captured images in which many people are photographed with no congestion and the whole body is photographed, both the method using a model imitating a person and the method using a discriminator that has learned the image of a person are used. The person can be detected with higher accuracy by using the image feature than by using the image feature of a part (for example, only in the vicinity of the head).

On the other hand, for captured images that are congested and frequently concealed, the image features of the whole body are used in both the method using a model imitating a person and the method using a classifier that has learned the image of a person. However, it is possible to detect the person with higher accuracy by using the image feature of only the part where the possibility of concealment is low.

Therefore, the person photographed on the left side of the crowded area in the captured image is photographed on the left side, the person photographed on the right side is photographed on the right side, and the person photographed above is photographed above and below. It is possible to detect with higher accuracy by evaluating the lower part with more emphasis than other parts.

Further, for example, if the image feature only in the vicinity of the head is always used in order to improve the detection accuracy at the time of congestion, the detection accuracy at the time of no congestion is lowered and the detection accuracy at the time of no congestion is improved. Therefore, if the image features of the whole body are always used, the detection accuracy at the time of congestion decreases.

In this way, in the captured image in which a space where congestion can occur is taken, the individual concealment state of the object to be detected changes according to the congestion state, so that each part of the object is always evaluated uniformly. It was difficult to accurately detect individual objects from the captured image.

The present invention has been made in view of the above problems, and provides an object detection device capable of accurately detecting individual objects in a photographed image even in a photographed image in a space where congestion may occur. The purpose is to do.

In order to achieve such an object, the present invention is an object detection device that detects an individual object from a captured image in which a space where congestion due to a predetermined object may occur is captured, and the object at the density at each predetermined density. A density estimation means that estimates the distribution of the density of objects captured in the captured image using a density estimator that learns the image characteristics of each density image captured in the space in which the image exists, and individual objects in the captured image. A candidate position that may exist is set, and a partial area corresponding to each of a plurality of parts constituting the object is set in the captured image based on the candidate position, and an image of the part corresponding to the partial area is set for each partial area. A predetermined integrated evaluation value is obtained by calculating a partial evaluation value indicating the degree of appearance of a feature, and emphasizing the partial evaluation value of a plurality of partial regions set based on the candidate position as the density of the partial region becomes lower. Provided is an object detection device provided with an object position determination means for determining a candidate position satisfying a determination criterion as an object position.

Further, the object position determination means sets a weighting coefficient for each partial region, which is higher as the density in the partial region is lower and lower as the density in the partial region is higher, and partial evaluation of a plurality of partial regions set based on the candidate position. It is preferable to calculate the integrated evaluation value by weighting the values with the weighting coefficient of the partial region and summing them up.

Further, it is preferable that the object position determining means sets a partial region corresponding to a small part of the parts constituting the object as the density at the candidate position increases.

Further, the object position determination means includes an arrangement generation means that generates a plurality of different arrangements each including one or more candidate positions, and a plurality of parts based on each candidate position for each of the plurality of arrangements. A model image generation means that draws a partial model that imitates the relevant part in the corresponding partial area to generate a model image, and similarities to the captured image of the partial model for each partial area for each of the model images in a plurality of arrangements. The evaluation value calculation means that calculates the partial evaluation value indicating the degree of the above and integrates the partial evaluation values of multiple partial areas to calculate the integrated evaluation value, and the candidate position in the arrangement where the integrated evaluation value is the maximum is the position of the object. It is preferable to include an optimum arrangement determining means for determining.

Further, the object position determining means sets the candidate position setting means for setting a plurality of candidate positions in the captured image, and the image feature of the partial region corresponding to each of the plurality of parts set based on each candidate position. The image features are input to the learned classifier to calculate the partial evaluation value of the relevant subregion, and the partial evaluation values of multiple subregions set based on the candidate position are integrated for each candidate position to obtain an integrated evaluation value. It is preferable to include a means for calculating the evaluation value to be calculated and a means for determining the position where the candidate position for which the integrated evaluation value satisfying the determination criteria is calculated is determined as the position of the object.

According to the present invention, individual objects can be accurately detected from a captured image in which a space where congestion can occur is captured.

It is a block diagram which shows the schematic structure of the image monitoring apparatus which concerns on 1st and 2nd Embodiment. It is a functional block diagram which shows the function of the image monitoring apparatus which concerns on 1st and 2nd Embodiment. It is a functional block diagram which shows the function of the image monitoring apparatus which concerns on 1st Embodiment. It is a figure which represented typically an example of the information of the object model stored in the object model storage means which concerns on 1st Embodiment. It is a figure which represented typically an example of the information of the object model stored in the object model storage means which concerns on 1st Embodiment. It is a figure which represented typically an example of the weight (the ratio of the weighting coefficient) stored in the weight storing means which concerns on 1st Embodiment. It is a figure which shows typically the processing example by the density estimation means, arrangement generation means, and model image generation means which concerns on 1st Embodiment. It is an image which schematically shows the weighting coefficient calculated by the model image generation means corresponding to each partial region of the model image 743 which concerns on 1st Embodiment. It is a flowchart which showed the operation of the image monitoring apparatus which concerns on 1st and 2nd Embodiment. It is a flowchart of the object position determination process by the image monitoring apparatus which concerns on 1st Embodiment. It is a flowchart of the object position determination process by the image monitoring apparatus which concerns on 1st Embodiment. It is a functional block diagram which shows the function of the image monitoring apparatus which concerns on 2nd Embodiment. It is a figure which represented typically the information of the partial classifier which the partial classifier storage means which concerns on 2nd Embodiment, and the weight information which a weight storage means stores. It is a figure which shows typically how the evaluation value calculation means which concerns on 2nd Embodiment calculates an integrated score about a candidate position. It is a flowchart of the object position determination process by the image monitoring apparatus which concerns on 2nd Embodiment.

[First Embodiment]
Hereinafter, as an embodiment of the present invention, an example of an image monitoring device 1 that includes an example of an object detection device that detects an individual person from a photographed image taken at an event venue and displays the detection result to an observer will be described. To do. In the image monitoring device 1 according to this embodiment, in particular, the object detection device detects an individual person by using an object model imitating a person, and at that time, the object detection device forms a plurality of partial models of the object model. Includes an example of using to detect an individual person.

<Configuration of image monitoring device 1 according to the first embodiment>
FIG. 1 is a block diagram showing a schematic configuration of the image monitoring device 1. The image monitoring device 1 includes a photographing unit 2, a communication unit 3, a storage unit 4, an image processing unit 5, and a display unit 6.

The photographing unit 2 is a surveillance camera, is connected to the image processing unit 5 via the communication unit 3, photographs the monitoring space at predetermined time intervals to generate a photographed image, and sequentially transfers the photographed images to the image processing unit 5. It is a shooting means to input. For example, the photographing unit 2 is installed on a pole installed at the event venue with a field of view overlooking the monitoring space. The field of view may be fixed, or may be changed according to a schedule in advance or an instruction from the outside via the communication unit 3. Further, for example, the photographing unit 2 photographs the monitoring space with a frame period of 1 second to generate a color image. A monochrome image may be generated instead of the color image.

The communication unit 3 is a communication circuit, one end of which is connected to the image processing unit 5, and the other end of the communication unit 3 is connected to the photographing unit 2 and the display unit 6 via a communication network such as a coaxial cable, LAN (Local Area Network), or the Internet. Be connected. The communication unit 3 acquires a captured image from the photographing unit 2 and inputs it to the image processing unit 5, and outputs the detection result input from the image processing unit 5 to the display unit 6.

The storage unit 4 is a memory device such as a ROM (Read Only Memory) and a RAM (Random Access Memory), and stores various programs and various data. The storage unit 4 is connected to the image processing unit 5 and inputs / outputs these information to and from the image processing unit 5.

The image processing unit 5 is composed of arithmetic units such as a CPU (Central Processing Unit), a DSP (Digital Signal Processor), and an MCU (Micro Control Unit). The image processing unit 5 is connected to the storage unit 4 and operates as various processing means / control means by reading and executing a program from the storage unit 4, and stores and reads various data in the storage unit 4. Further, the image processing unit 5 is also connected to the photographing unit 2 and the display unit 6 via the communication unit 3, and detects an individual person by analyzing the captured image acquired from the photographing unit 2 via the communication unit 3. The detection result is displayed on the display unit 6 via the communication unit 3.

The display unit 6 is a display device such as a liquid crystal display or a CRT (Cathode Ray Tube) display, and is a display means connected to the image processing unit 5 via the communication unit 3 and displaying the detection result by the image processing unit 5. .. The observer visually recognizes the displayed detection result, judges the occurrence of congestion, etc., and takes measures such as changing the staffing as necessary.

In this embodiment, the image monitoring device 1 in which the number of the photographing unit 2 and the image processing unit 5 is 1: 1 is illustrated, but in another embodiment, the number of the photographing unit 2 and the image processing unit 5 is illustrated. Can be many-to-one or many-to-many.

<Function of image monitoring device 1 according to the first embodiment>
2 and 3 are functional block diagrams showing the functions of the image monitoring device 1. The communication unit 3 functions as an image acquisition means 30, an object position output means 31, and the like, and a storage unit 4 functions as a density estimator storage means 40, a single feature storage means 41, and the like. The image processing unit 5 functions as a density estimation means 50, an object position determination means 51, and the like. Further, the single feature storage means 41 includes functions as an object model storage means 410a and a weight storage means 412a, and the object position determination means 51 includes an arrangement generation means 510a, a model image generation means 512a, an evaluation value calculation means 514a, and an optimum arrangement determination. It includes a function as means 516a.

The image acquisition means 30 sequentially acquires captured images from the photographing unit 2 which is a photographing means, and sequentially outputs the acquired captured images to the density estimation means 50 and the object position determination means 51.

The density estimator storage means 40 is an estimated density calculation function that learns the image features of each density image obtained by photographing a space in which an object (person) exists at the density at a predetermined density, and obtains the feature amount of the image. When input, the estimated value (estimated density) of the density of the object captured in the image is calculated, and the information of the estimator (density estimator) that outputs the calculated estimated density is stored in advance. That is, parameters such as the coefficient of the estimated density calculation function are stored in advance as information of the density estimator.

The density estimation means 50 extracts the feature amount for density estimation (feature amount for estimation) from various parts of the captured image input from the image acquisition means 30, and reads out the density estimator from the density estimator storage means 40 to extract the feature amount. The distribution of the estimated density (density distribution) is estimated by inputting each of the estimated feature quantities to the density estimator, and the estimated density distribution is output to the object position determination means 51.

The processing of density estimation and the density estimator will be specifically described.

The density estimation means 50 sets a window (estimation extraction window) at the position of each pixel of the captured image, and extracts the estimation feature amount from the captured image in each estimation extraction window. The estimation feature quantity is a GLCM (Gray Level Co-occurrence Matrix) feature.

It is desirable that the areas in the monitoring space photographed by each estimation window are the same size. That is, preferably, the density estimation means 50 reads out the camera parameters of the photographing unit 2 stored in advance from the camera parameter storage means (not shown), and is photographed on an arbitrary pixel of the captured image by homography conversion using the camera parameters. The captured image is deformed so that the areas in the monitoring space are the same size, and then the estimation feature amount is extracted.

The density estimator can be realized by a discriminator that discriminates a multi-class image, and can be a discriminant function learned by a multi-class SVM (Support Vector Machine) method.

The density is, for example, a "background" class with no people, a "low density" class ^{higher than 0 people / m 2} and 2 people / m ^{2 or less, higher than 2 people / m 2} and 4 people / m ² or less. It can be defined as 4 classes of "medium density" class and ^{"high density" class higher than 4 people / m 2.}

The estimated density is a value given in advance to each class and is a value output as a result of distribution estimation. In this embodiment, the values corresponding to each class are described as "background", "low density", "medium density", and "high density".

That is, the density estimator applies the multi-class SVM method to the features of a large number of images (density images) belonging to each of the "background" class, "low density" class, "medium density" class, and "high density" class. It is an identification function for distinguishing the image of each class from other classes. The parameters of the discriminant function derived by this learning are stored as a density estimator. The feature amount of the density image is the same as the feature amount for estimation, and is a GLCM feature.

The density estimation means 50 acquires the estimated density, which is the output value, by inputting each of the estimation feature quantities extracted corresponding to each pixel into the density estimator. When the captured image is deformed to extract the feature amount for estimation, the density estimation means 50 transforms the density distribution into the shape of the original captured image by homography transformation using camera parameters.

The density distribution is a collection of estimated densities for each pixel of the captured image obtained in this way.

From the density distribution output by the density estimation means 50, the density of people in various parts of the photographed image can be known, but from the density distribution, the position of each person cannot be known.
On the other hand, the object position determination means 51 at the subsequent stage of the density estimation means 50 is a means for determining the position of an individual person appearing in the captured image.

The object position determining means 51 detects an individual object by searching on a captured image for a place where an image feature as a single person (object) appears, and determines the position of the individual object. That is, the object position determining means 51 sets a candidate position where each object can exist in the captured image, and represents the degree to which the image feature (single feature) of a single object appears in the captured image at the candidate position. The evaluation value (integrated evaluation value) is calculated, and the candidate position where the integrated evaluation value is equal to or higher than a predetermined value is determined as the position of the object. For example, the single feature is a human shape, and the single feature storage means 41 stores the single feature in advance. Further, for example, the integrated evaluation value is the degree of similarity between the edge of the photographed image and the model representing the shape of a person. The integrated evaluation value is an evaluation value that integrates evaluation values (partial evaluation values) for each of a plurality of parts constituting the object.

Here, in a captured image in which a space where congestion can occur is captured, it is considered that the reliability of the evaluation value is low because concealment is more likely to occur in a region having a higher density. On the contrary, it is considered that the lower the density, the less concealment occurs and the higher the reliability of the evaluation value. Further, in a photographed image in which a space where congestion may occur is taken, many objects such as an object existing at a boundary of density in which a highly reliable portion and a low reliability portion coexist may be included.

Therefore, the object position determination means 51 does not uniformly evaluate the entire object, but improves the detection accuracy of the object by weighting each part of the object according to the density with reference to the density distribution.
That is, the object position determining means 51 sets a candidate position in which an individual object may exist in the captured image, and sets a partial region corresponding to each of a plurality of portions constituting the object in the captured image with reference to the candidate position. Set, calculate the partial evaluation value indicating the degree of appearance of the image feature of the part corresponding to the partial area for each partial area, and set the partial evaluation value of each partial area based on the candidate position as the relevant partial area. The lower the density of the object, the more important the integrated evaluation value is, and the candidate position where the integrated evaluation value satisfies a predetermined criterion is determined as the position of the object. More specifically, the object position determination means 51 sets a weighting coefficient for each of the partial regions, which is higher as the density in the partial region is lower and lower as the density in the partial region is higher, and is set based on the candidate position. The partial evaluation value of the subregion is weighted by the weighting coefficient of the subregion and summed up to calculate the integrated evaluation value.

In addition, the higher the density of the region, the higher the possibility that severe concealment will occur, and if the concealed portion is also evaluated, the chance of erroneous evaluation will increase, leading to a decrease in the detection accuracy of the object. For example, when the photographing unit 2 is installed from a bird's-eye view, concealment is more likely to occur as it is closer to the feet, and concealment is less likely to occur as it is closer to the head. In consideration of this, if the single feature is limited to the human head in order to adapt to congestion, the failure to detect during congestion is reduced. However, since a relatively high integrated evaluation value is calculated for the shoulder and the like as a single feature of only the head, erroneous detection when it is not crowded increases.

By referring to the density distribution, the object position determining means 51 eliminates the trade-off existing between the amount of the evaluated portion and the detection accuracy of each object. That is, the object position determination means 51 sets a partial region corresponding to a smaller portion of the portions constituting the object as the density at the candidate position increases. That is, the object position determination means 51 evaluates the image feature of a small portion of the portions constituting the object and calculates the integrated evaluation value as the density at the candidate position increases. For example, the object position determining means 51 evaluates the image features of the whole body to calculate the integrated evaluation value if the estimated density of the candidate positions is low, and evaluates the image features of the upper body to evaluate the integrated evaluation if the density is medium. The value is calculated, and if the density is high, the image feature near the head is evaluated and the integrated evaluation value is calculated.

Hereinafter, the detection of individual objects and the characteristics of individual objects will be described.

The single feature storage means 41 is an object model storage means 410a that stores information of an object model that imitates the shape of a single person (object) in advance, and a weight storage means 412a that stores weight information used in calculating an evaluation value in advance. It functions as, and stores the information of the object model and the information of the weight as a single feature.

4 to 6 are diagrams schematically showing the single features stored in the single feature storage means 41. Of these, FIGS. 4 and 5 are examples of object model information stored by the object model storage means 410a, and FIG. 6 is an example of weights (weight coefficient ratio) stored by the weight storage means 412a. is there.

The object model stored in the object model storage means 410a is specifically a three-dimensional model 700 composed of three spheroids corresponding to the head, body, and legs of a standing person. The center of gravity of the head is the representative position of the person. Further, the object model storage means 410a stores the evaluation range 702 for each density together with the stereoscopic model 700, and stores the camera parameter 701 of the photographing unit 2 in order to project the stereoscopic model 700 onto the coordinate system of the captured image. ing. The camera parameter 701 includes external parameters such as the installation position and imaging direction of the photographing unit 2 in the actual monitoring space, internal parameters such as the focal length, angle of view, lens distortion and other lens characteristics of the photographing unit 2 and the number of pixels of the imaging element. Information to include.

The evaluation range 702 is divided into a plurality of parts and set for each density, and the higher the density, the smaller the part that constitutes a single object.
Specifically, the object model storage means 410a has six parts of a person "upper 1/3, left 1/2", "upper 1/3, right 1/2", in association with a value representing a low density class. Evaluation representing "middle 1/3, left 1/2", "middle 1/3, right 1/2", "bottom 1/3, left 1/2" and "bottom 1/3, right 1/2" I remember the range. When these six parts are combined, it becomes the "whole" of a person.
Further, the object model storage means 410a associates the four parts of the person with the values representing the medium density class, "upper 1/3, left 1/2", "upper 1/3, right 1/2", and "middle 1". The evaluation range representing "/ 3, left 1/2" and "middle 1/3, right 1/2" is stored. When these four parts are combined, it becomes the "upper two-thirds" of a person.
Further, the object model storage means 410a has an evaluation range representing four parts of a person "upper 1/3, left 1/2" and "upper 1/3, right 1/2" in association with a value representing a high-density class. I remember. When these two parts are combined, it becomes the "upper 1/3" of a person.

Hereinafter, the object model 710 represented by the combination of the evaluation range "upper 1/3, left 1/2", the three-dimensional model 700, and the camera parameter 701 is the upper left model, and the evaluation range "upper 1/3, right 1/2". The partial model 711 represented by the combination of the three-dimensional model 700 and the camera parameter 701 is shown by the upper right model, the evaluation range "middle 1/3, left 1/2" and the combination of the three-dimensional model 700 and the camera parameter 701. The partial model 712 is the left middle model, and the partial model 713 represented by the combination of the evaluation range "middle 1/3, right 1/2", the three-dimensional model 700, and the camera parameter 701 is the right middle model, and the evaluation range "bottom". The partial model 714 represented by the combination of "1/3, left 1/2", the three-dimensional model 700, and the camera parameter 701 is the lower left model, the evaluation range "lower one-third, right 1/2", and the three-dimensional model 700. The partial model 715 represented by the combination with the camera parameter 701 is referred to as a lower right model. Further, the object model 720 represented by the combination of the evaluation range "whole", the three-dimensional model 700 and the camera parameter 701 is represented by the whole body model, and the evaluation range "upper two-thirds" is represented by the combination of the three-dimensional model 700 and the camera parameter 701. The object model 721 is referred to as an upper body model, and the object model 722 represented by a combination of the evaluation range “upper 1/3”, the three-dimensional model 700, and the camera parameter 701 is referred to as a head proximity model.

As described above, the object model storage means 410a is a whole body including the upper left model 710, the upper right model 711, the left middle model 712, the right middle model 713, the lower left model 714, and the lower right model 715 in association with the low density class. The upper left model 721 consisting of the upper left model 710, the upper right model 711, the left middle model 712 and the right middle model 713 is associated with the high density class, and the upper left model 710 and the upper right are associated with the model 720. The head vicinity model 722 composed of the part model 711 is stored as the information of the object model.
What is drawn as a partial model is a contour line (solid line portion in FIG. 5) representing the shape of the object.

The weight is the degree to which the portion of the object is emphasized according to the density of the corresponding partial region, and is expressed as a relative ratio between the densities. The lower the density, the more important the part, and the higher the density, the lighter the weight. Therefore, the lower the density, the higher the weight, and the higher the density, the lower the weight. For example, low density, medium density and high density weights can have a ratio of 10: 7: 5. The background class can be regarded as a low density class when determining the density of a partial region.

As described above, the weight storage means 412a stores the weight for each density, which is higher as the density is lower and lower as the density is higher.

The arrangement generation means 510a generates a plurality of different arrangements each including one or more candidate positions, and outputs each generated arrangement to the model image generation means 512a.

Therefore, the arrangement generation means 510a selects one or more pixels (the number of arrangements) from among the pixels of the captured image having an estimated density of low density, medium density, or high density based on random numbers. Arrangement is generated by randomly determining and using the determined position of each pixel as a candidate position. The arrangement generation means 510a repeats this generation for each number of arrangements by a predetermined number of times while sequentially increasing the number of arrangements, thereby generating a plurality of different arrangements. The upper limit of the number of arrangements can be the upper limit of the number of objects that can exist in the monitoring space. For example, a three-dimensional model of a standing person can be arranged in a virtual space imitating the monitoring space without overlapping. It can be calculated as a number.

The model image generation means 512a draws a partial model imitating the portion in the partial region corresponding to each of the plurality of portions based on each candidate position for each of the plurality of arrangements input from the arrangement generation means 510a. To generate a model image. At that time, the model image generation means 512a draws an object model at each candidate position to imitate a small part of the parts constituting a single object as the density at the candidate position increases, and draws a model image. It is generated, and each generated model image is output to the evaluation value calculation means 514a.

Therefore, the model image generation means 512a reads out the camera parameters from the object model storage means 410a, and uses the camera parameters for each arrangement to set each candidate position at the height of the center of gravity of the head of the three-dimensional model (for example, 1.5 m). By back-projecting onto the horizontal plane of, the representative position of the three-dimensional model projected at the candidate position in the virtual space that imitates the monitoring space is calculated.

Further, the model image generation means 512a reads out the head neighborhood model from the object model storage means 410a, arranges the head neighborhood model at a representative position in the virtual space corresponding to each candidate position, and uses the camera parameters to arrange the head neighborhood model. The neighborhood model is projected onto the coordinate system of the captured image. Then, the model image generation means 512a aggregates the estimated densities in the projection region of the model near the head corresponding to each candidate position with reference to the density distribution input from the density estimation means 50, and the largest number in each candidate position. The estimated density (excluding the background class) is determined as the density of the candidate position.

Further, the model image generation means 512a reads out an object model corresponding to the density of the candidate position for each candidate position from the object model storage means 410a. Specifically, the model image generation means 512a reads out a whole body model consisting of six partial models if the density of the candidate positions is low, and reads out an upper body model consisting of four partial models if the density is medium, and is high. If the density is high, a head-near model consisting of two partial models is read out. Then, the model image generation means 512a arranges the object model read out corresponding to each candidate position at a representative position in the virtual space corresponding to the candidate position for each arrangement, and uses the camera parameters to set each object model. A model image for each arrangement is generated by projecting each of the partial models to be performed onto the coordinate system of the captured image and drawing the shape (contour line) of the object.
The model image generation means 512a sequentially projects from the object model arranged at the representative position far from the photographing unit 2 and overwrites the projection area to generate a model image expressing the concealment between the object models. ..

Further, the model image generation means 512a calculates the degree of concealment representing the degree of overlap between the object models in the model image for each arrangement according to the following equation.
Concealment = Area of overlapping area between object models / Area of sum area of projection area of object model (1)

Further, the model image generation means 512a calculates a weighting coefficient corresponding to the estimated density of the partial region for each partial region in the model image in correspondence with each model image.

Therefore, the model image generation means 512a aggregates the estimated densities in each sub-region with reference to the density distribution, and determines the highest estimated density in each sub-region as the density of the sub-region. However, the background class is regarded as a low density class and aggregated.
Next, the model image generation means 512a reads out the weights (ratio of weighting coefficients) corresponding to the densities of each partial region from the weight storage means 412a, and obtains the sum of the weights of all the partial regions for each arrangement.
Subsequently, the model image generation means 512a divides the weight of each partial region by the sum of the weights for each arrangement to calculate the weight coefficient of the partial region. That is, the sum of the weighting coefficients in each arrangement is normalized to be 1.

The model image generation means 512a associates the arrangement obtained in this way with the model image, the degree of concealment, and the weighting coefficient, and outputs the result to the evaluation value calculation means 514a.

FIG. 7 is a diagram schematically showing a processing example by the density estimation means 50, the arrangement generation means 510a, and the model image generation means 512a according to the first embodiment.
The image 740 is an image of the density distribution estimated by the density estimation means 50. In the density distribution, the white area is the area where the estimated density is the background, the horizontal line area is the area where the estimated density is low, the shaded area is the area where the estimated density is medium density, and the lattice area is the area where the estimated density is high. Each area is shown.
The image 741 is a plot of eight candidate positions included in the arrangement generated by the arrangement generation means 510a on the coordinate system of the captured image with a cross.
The three-dimensional model 742 illustrates how the model image generation means 512a arranges the three-dimensional model at the representative positions in the virtual space corresponding to the eight candidate positions shown in the image 741.
The image 743 was created by the model image generation means 512a specifying the density of each candidate position based on the density distribution shown in the image 740 and projecting a three-dimensional model in the evaluation range corresponding to the density onto each candidate position. The model image is shown.

In the arrangement represented by the model image 743, one partial region (the middle left middle part of the middle right person in the group) is completely concealed, and 33 partial regions are drawn. The breakdown of the 33 subregions is 23 with low density or background, 6 with medium density, and 4 with high density. The sum of the weighting coefficient ratios for all the partial regions of the arrangement represented by the model image 743 is 10 × 23 + 7 × 6 + 5 × 4 = 292. Of the 33 partial regions, the partial region 744 has a low density, the partial region 745 has a medium density, and the partial region 746 has a high density. The weighting factor of the partial region 744 is 10/292, the weighting coefficient of the partial region 745 is 7/292, and the weighting coefficient of the partial region 746 is 5/292. The weighting coefficient is calculated in the same manner for the other subregions.

The image 750 of FIG. 8 is an image schematically showing the weighting coefficient calculated by the model image generation means 512a corresponding to each partial region of the model image 743.

The evaluation value calculation means 514a calculates an integrated evaluation value indicating the degree of similarity of the model image input from the model image generation means 512a to the captured image for each of the plurality of arrangements, and optimally arranges the integrated evaluation value for each arrangement. Output to the determination means 516a.

Specifically, the evaluation value calculation means 514a calculates the weighting similarity between each model image and the captured image according to the following equation.
Weighted similarity = Weighted shape goodness of fit − W _Ha × Concealment degree (2)
However, W _Ha is a weighting coefficient larger than 0 and is preset based on prior experiments. The concealment degree deducted from the weighted shape fit is a penalty value for suppressing excessive overlap of object models. By determining the optimum arrangement based on the degree of similarity including the degree of concealment in this way, it is possible to prevent erroneous detection of the object position due to the application of an object model larger than the original number of objects.

The evaluation value calculation means 514a calculates the weighted shape suitability by weighting the shape suitability of each partial region of the model image and the captured image with the weighting coefficient of the partial region and summing them up. The shape conformity for each partial area is the partial evaluation value. The goodness of fit can be the similarity of edges. The evaluation value calculation means 514a extracts an edge from each of the model image and the captured image, and for each model image, the pixel obtained by extracting a valid edge from the model image and the edge of the pixel of the captured image corresponding to the pixel are used. The absolute value of the difference was calculated and summed, and the sum was divided by the sum of the number of pixels where valid edges were extracted from the captured image and the number of pixels where valid edges were extracted from the model image, and the sign was inverted. The value is calculated as the weighted shape suitability.

Alternatively, the evaluation value calculation means 514a generates an edge image from each of the model image and the captured image, and for each model image, chamfer matching between the edge image generated from the captured image and the edge image generated from the model image. The value obtained by performing (Chamfer Matching) with the sign of the distance inverted is divided by the sum of the number of pixels in which valid edges are extracted from the captured image and the number of pixels in which valid edges are extracted from the model image. It can also be the weighted shape conformity of the model image.

The optimum arrangement determining means 516a refers to the integrated evaluation value for each arrangement input from the evaluation value calculating means 514a, determines the candidate position in the arrangement with the maximum integrated evaluation value as the position of the object, and determines the information on the determined object position. Is output to the object position output means 31. That is, the optimum arrangement determining means 516a determines each candidate position included in the arrangement in which the maximum similarity is calculated as the position of each person photographed in the captured image.
For example, the optimum arrangement determining means 516a draws an object model at each object position in different colors according to the density of the object position so that the observer can easily see it, and generates and outputs information on the object position. Alternatively, the object position information can be the coordinate value of the object position itself, and the object position information can be a region of each drawn object model that does not overlap with other object models. Alternatively, the object position information may be data including two or more of the above-mentioned data.

The object position output means 31 sequentially outputs the object position information input from the object position determination means 51 to the display unit 6, and the display unit 6 displays the object position information input from the object position output means 31. For example, the information on the position of the object is transmitted and received via the Internet and displayed on the display unit 6. By visually recognizing the displayed information, the observer grasps the point where congestion is occurring in the monitoring space, and takes measures such as dispatching or increasing the number of guards to the point.

<Operation of the image monitoring device 1 according to the first embodiment>
The operation of the image monitoring device 1 will be described with reference to the flowcharts of FIGS. 9 to 11.

When the image monitoring device 1 starts operation, the photographing unit 2 installed at the event venue photographs the monitoring space at predetermined time intervals and sequentially transmits the photographed images to the image analysis center where the image processing unit 5 is installed. To do. Then, each time the image processing unit 5 receives the captured image, the image processing unit 5 repeats the operation according to the flowchart of FIG.

First, the communication unit 3 operates as the image acquisition means 30, and is in a state of waiting for reception of the captured image from the photographing unit 2. The image acquisition means 30 that has acquired the captured image outputs the captured image to the image processing unit 5 (step S1).

The image processing unit 5 to which the captured image is input operates as the density estimation means 50, and estimates the density distribution from the captured image (step S2). The density estimation means 50 extracts the estimation feature amount at the position of each pixel of the captured image, reads the density estimator from the density estimator storage means 40 of the storage unit 4, and uses each estimation feature amount as the density estimator. The density distribution is estimated by inputting and acquiring the estimated density in each pixel of the captured image.

The image processing unit 5 that estimates the density distribution also operates as the object position determination means 51, and the captured image is input from the image acquisition means 30 and the density distribution is input from the density estimation means 50 to the object position determination means 51. .. The object position determination means 51 in which these are input confirms whether or not the density distribution includes an estimated density other than the background class (step S3).

When an estimated density other than the background class is included (YES in step S3), the object position determination means 51 determines the position of each object from the captured image, assuming that at least one person has been photographed. Perform the determination process (step S4). On the other hand, in the case of only the background class (NO in step S3), it is assumed that no person has been photographed, and the processes of steps S4 and S5 are omitted.

The object position determination process in step S4 will be described with reference to the flowcharts of FIGS. 10 and 11. The single feature storage means 41 operates as the object model storage means 410a and the weight storage means 412a, and the object position determination means 51 operates as the placement generation means 510a, the model image generation means 512a, the evaluation value calculation means 514a, and the optimum placement determination means 516a. Then, the object position determination process is executed.

The arrangement generation means 510a sequentially sets the number of arrangements in the range from 1 to the upper limit number or less (step S100), and controls the loop processing in steps S100 to S117.

Further, the arrangement generation means 510a prepares a variable T for counting the number of iterations, initializes T to 0 (step S101), and starts the iteration process of steps S102 to S116.

Next, the arrangement generation means 510a places the same number of candidate positions as the number of arrangements set in step S100 in the region where the estimated density is low density, medium density, or high density in the density distribution input from the density estimation means 50. By setting at random, the T-th arrangement in the number of arrangements is generated and output to the model image generation means 512a (step S102).

The model image generation means 512a reads the camera parameters from the object model storage means 410a, and uses the camera parameters to convert each candidate position included in the arrangement generated in step S102 into three-dimensional coordinates in the virtual space (step S103). ..

Next, the model image generation means 512a prepares and initializes a model image having the same size as the captured image, calculates the distance to the photographing unit 2 in the three-dimensional coordinates of each candidate position, and determines the distance to the distant candidate position. The processing targets are set in order from (step S104), and the loop processing of steps S104 to S110 is executed.

Subsequently, the model image generation means 512a specifies the density of the candidate position to be processed with reference to the density distribution (step S105). The model image generation means 512a reads out the head vicinity model from the object model storage means 410a, arranges it at the three-dimensional coordinates of the candidate position, and projects the head vicinity model onto the coordinate system of the captured image using the camera parameters. , The highest estimated density in the projection area (but other than the background class) is specified as the density of the candidate position.

Subsequently, the model image generation means 512a reads out the object model corresponding to the density specified in step S105 from the object model storage means 410a (step S106), arranges the object model at the three-dimensional coordinates of the candidate position to be processed, and sets the camera parameter. Is used to overwrite and project the arranged object model on the model image (step S107). At this time, the model image generation means 512a records the projected area of the object model.
The object model is composed of a plurality of partial models, and the contours of the objects in the plurality of partial models are overwritten and projected on each position based on the candidate position in the model image. The projection area on which the partial model of each part is projected is the partial area of the part. The model image generation means 512a assigns an identification number to the combination of the candidate position and the portion, and sets the identification number of the projected portion together with the intensity value of the outline of the object in the partial model in the pixel value of the partial region corresponding to the portion. To do.

Subsequently, the model image generation means 512a specifies the density of each partial region with reference to the density distribution (step S108). The model image generation means 512a specifies the highest estimated density in each subregion (however, the background class is regarded as a low density class) as the density of the subregion.

Subsequently, the model image generation means 512a sets the weight (ratio of weighting factors) corresponding to the density of each partial region with reference to the weight storage means 412a (step S109). That is, the model image generation means 512a records the weight of each partial region together with the identification number of the portion corresponding to the partial region. At this time, the weight of the partial region completely hidden on the model image by the overwrite projection in step S107 is deleted from the recording.

Then, the model image generation means 512a confirms whether or not all the candidate positions included in the T-th arrangement in the current arrangement number have been processed (step S110), and if there is an unprocessed candidate position (step S110). NO in step S110), the process is returned to step S104, and the next candidate position is processed.

On the other hand, when all the candidate positions have been processed (YES in step S110), the model image for the arrangement of the Tth street in the current arrangement number is completed. The model image generation means 512a that has completed the model image calculates the degree of concealment of the object model in the model image (step S111). That is, the model image generation means 512a obtains the area of the projected area on the model image, which is the “area of the sum area of the projected areas of the model”, and sums the projected areas of each object model recorded in step S107. , The area of the sum area of the projected area of the model is subtracted from the total value to obtain the "area of the overlapping area between the models", and these are substituted into the equation (1) to calculate the degree of concealment.

Further, the model image generation means 512a totals the weights of the partial regions recorded in step S109, and divides the weights of each partial region by the total value to calculate the weight coefficient of each partial region (step S112).

The model image generation means 512a for which the weighting coefficient is calculated outputs the model image, the degree of concealment, and the weighting coefficient for each partial region to the evaluation value calculating means 514a.

The evaluation value calculation means 514a in which the model image, the concealment degree, and the weighting coefficient are input calculates the shape conformity of each partial region of the model image and the captured image as a partial evaluation value of the partial region (step S113), and further. From the weighted shape suitability and concealment degree, which is the sum of the product of the shape fit degree for each partial region and the weighting coefficient of the partial region, the weighted similarity between the model image and the captured image is determined for the T-th arrangement in the current number of arrangements. Calculated as an integrated evaluation value (step S114). That is, the evaluation value calculation means 514a generates an edge image from each of the model image and the captured image input from the model image generation means 512a, and the similarity of each partial region of these edge images and the weighting coefficient for each partial region. Is added to calculate the weighted shape conformity. Then, the weighted shape goodness of fit and the concealment degree are substituted into the equation (2) to calculate the weighted similarity.

When the weighted similarity for the T-th arrangement in the current number of arrangements is calculated, the evaluation value calculating means 514a records the arrangement and the weighted similarity in association with each other, and the arrangement generating means 510a sets the number of repetitions T to 1 only. is increased (step S115), and compared with the specified number of times _{T MAX} (step S116), if T is less than _{T MAX} iteration in the current arrangement number back (NO at step S116), the processing to step S102 Let it continue.

If the number of iterations T has reached a predetermined number T _MAX (YES in step S116), disposed generation unit 510a ends the iteration in the current arrangement number, checks whether finished setting all numbers arranged (Step S117). If there is an unset number of arrangements (NO in step S117), the process is returned to step S100 and the process for the next number of arrangements is performed.

On the other hand, when all the arrangement numbers have been set (YES in step S117), the evaluation value calculation means 514a inputs the arrangement and the weighting similarity recorded in step S115 into the optimum arrangement determination means 516a, and determines the optimum arrangement. The means 516a identifies the arrangement having the maximum weighting similarity among them (step S118), and determines that the arrangement is information representing the position of an individual person photographed in the captured image.

The description will be continued with reference to FIG. 9 again. The object position determination means 51 outputs the information of the position (object position) of each person determined in step S4 to the communication unit 3 (step S5). The communication unit 3 to which the object position information is input operates as the object position output means 31, and transmits the object position information to the display unit 6.

When the above processing is completed, the processing is returned to step S1, and the processing for the next captured image is performed.

[Second Embodiment]
Hereinafter, as a preferred embodiment of the present invention different from the first embodiment, an individual person is detected by using a discriminator that has learned the image features of a single person, and in particular, a plurality of persons constituting the single person. An example of the image monitoring device 1 including an example of an object detection device that detects an individual person by using a partial classifier that has learned the image features of each part will be described.

The image monitoring device according to the second embodiment is different from the image monitoring device according to the first embodiment in the details of the single feature stored in the single feature storage means 41 and the details of the processing performed by the object position determination means 51. , Schematic configuration, schematic functions and some of the operations are common. Therefore, a schematic configuration, a schematic function, and a part of the operation will be described again with reference to the block diagram of FIG. 1, the functional block diagram of FIG. 2, and the flowchart of FIG. 9, respectively, which are referred to in the first embodiment. ..

<Configuration of image monitoring device 1 according to the second embodiment>
A schematic configuration of the image monitoring device 1 according to the second embodiment will be described with reference to the block diagram of FIG.
Similar to the first embodiment, the image monitoring device 1 has a photographing unit 2 that photographs the monitoring space at predetermined time intervals and outputs the captured image, and a display unit 6 that inputs the object position information and displays the information. And the image processing unit 5 that acquires a captured image, detects an individual person (object) from the captured image, generates information on the position (object position) of the detected object, and outputs the captured image and the object. It is connected to a communication unit 3 that mediates input / output of position information and the like, and a storage unit 4 that stores programs and various data and inputs / outputs them is connected to an image processing unit 5.

<Function of image monitoring device 1 according to the second embodiment>
The function of the image monitoring device 1 according to the second embodiment will be described with reference to the functional block diagrams of FIGS. 2 and 12.

Similar to the first embodiment, the communication unit 3 is input from the image acquisition means 30 and the object position determination means 51 that acquire the captured image from the imaging unit 2 and output it to the density estimation means 50 and the object position determination means 51. It includes a function as an object position output means 31 or the like that outputs information on the position of the object to the display unit 6.

Further, as in the first embodiment, the storage unit 4 stores a density estimator that has learned the image features of each density image obtained by photographing a space in which an object exists at a predetermined density at a predetermined density. The estimator storage means 40 includes functions as a single feature storage means 41 that stores image features (single features) of a single object generated by prior learning, and is stored by the single feature storage means 41. The single feature is the information of the image feature for each part constituting the object and the information of the weight with respect to the density.

Further, the image processing unit 5 estimates the density distribution of the object captured in the captured image by scanning the captured image with the density estimator as in the first embodiment, and determines the estimated density distribution for the object position. An evaluation value indicating the degree to which the image feature of a single object appears in the captured image of the candidate position by setting the density estimation means 50 output to the means 51 and the candidate position where each object can exist in the captured image. The function as an object position determination means 51 or the like that calculates (integrated evaluation value), determines a candidate position whose integrated evaluation value satisfies the determination criteria as an object position, and outputs the object position information to the object position output means 31. Including, the object position determination means 51 sets a partial area corresponding to each of a plurality of parts constituting the object in the captured image with reference to the candidate position, and an image of the part corresponding to the partial area for each partial area. The integrated evaluation value is calculated by calculating the partial evaluation value indicating the degree of appearance of the feature, and integrating the partial evaluation value of each partial area set based on the candidate position with emphasis as the density of the partial area decreases. To do.

However, as described above, the details of the processing performed by the object position determination means 51 according to the second embodiment and the details of the single feature stored by the single feature storage means 41 are the image monitoring device according to the first embodiment. Different from 1. These points will be described with reference to the functional block diagram of FIG.

The single feature storage means 41 according to the second embodiment is a partial identification in which a classifier (partial classifier) for each part that has learned the image features of each of a plurality of parts constituting a single person (object) is stored in advance. It functions as the instrument storage means 411b and the weight storage means 412b in which the weight information used in the calculation of the evaluation value is stored in advance, and the information of the partial classifier and the weight information are stored as a single feature.

FIG. 13 shows the single feature stored in the single feature storage means 41 according to the second embodiment, that is, the partial classifier information and the weight storage means 412b stored in the partial classifier storage means 411b. It is a figure which represented the information of the weight which is present.

Each of the partial classifiers for each part of the object calculates a partial score (partial evaluation value) indicating the plausibility that the image is an image (partial image) of the part when the feature amount of the image is input. It is represented by the coefficient of the score calculation function that is output. The partial classifier storage means 411b stores parameters such as a coefficient representing each partial classifier and the correspondence between the relative position and density of each part from the reference position and the partial classifier used. The reference position is, for example, the center of gravity of the head.

The partial classifier can be, for example, a classifier learned by applying the linear SVM method to the features of a learning image composed of a partial image of a large number of people and a large number of unmanned images. When linear SVM is used as the learning algorithm, the coefficient of the score calculation function is a weight vector. This weight vector is a weight for each element of the feature amount, and the value of the inner product of the feature amount and the weight vector of the input image represents the partial score. In learning, if the inner product of the weight vector and the feature amount is 0 or more, it is identified as a human partial image, and if it is less than 0, it is not identified as a human partial image. Therefore, the threshold value for discriminating whether or not the input image is an image of a single person when applied to the integrated score is 0 in principle, and the threshold value can usually be set to 0. However, the threshold value may be set to a value smaller than 0 in order to reduce the error of identifying the image of a single person as not the image of a single person.
The feature amount of the learning image can be, for example, a HOG (Histograms of Oriented Gradients) feature amount.

Specifically, the partial classifier storage means 411b is associated with a value representing a low density class and has six parts of a person "upper 1/3, left 1/2", "upper 1/3, right 1/2". , "Middle 1/3, Left 1/2", "Middle 1/3, Right 1/2", "Bottom 1/3, Left 1/2" and "Bottom 1/3, Right 1/2" The partial classifiers 800 to 805 that have learned the image features are stored. When these six parts are combined, it becomes the "whole" of a person.
Below, "upper 1/3, left 1/2" is the upper left part, "upper 1/3, right 1/2" is the upper right part, "middle 1/3, left 1/2" is the left middle part, and "middle 1". "3/3, right 1/2" is referred to as the middle right, "lower 1/3, left 1/2" is referred to as the lower left, and "lower 1/3, right 1/2" is referred to as the lower right. Further, the partial discriminator 800 for the upper left part is the upper left part discriminator, the partial discriminator 801 for the upper right part is the upper right part discriminator, the partial discriminator 802 for the left middle part is the left middle part discriminator, and the partial discriminator for the right middle part. The device 803 is referred to as a right middle discriminator, the partial discriminator 804 for the lower left is referred to as a lower left discriminator, and the partial discriminator 805 for the lower right is referred to as a lower right discriminator. Further, a set 810 of six partial classifiers 800 to 805 whose evaluation range is "whole" when combined is referred to as a whole classifier.
Further, the partial classifier storage means 411b stores the upper left part classifier 800, the upper right part classifier 801 and the left middle part classifier 802 and the right middle part classifier 803 in association with the value representing the medium density class. When the evaluation ranges of these four partial classifiers are combined, it becomes the "upper two-thirds" of a person. A set 811 of four partial classifiers 800 to 803 having an evaluation range of "upper 2/3" when combined is referred to as an upper body classifier.
Further, the partial classifier storage means 411b stores the upper left class classifier 800 and the upper right class classifier 801 in association with the values representing the high-density class. The combined evaluation range of these two partial classifiers is the "upper 1/3" of a person. A set 812 of two partial classifiers 800 and 801 having an evaluation range of "upper 1/3" when combined is called a head proximity classifier.

In this way, the partial classifier storage means 411b is associated with the low density class, the upper left class classifier 800, the upper right class classifier 801 and the left middle class classifier 802, the right middle class classifier 803, the lower left lower class classifier 804, and the right side. The overall classifier 810 consisting of the lower class classifier 805 is associated with the middle density class, and the upper body classifier 800, the upper right board classifier 801 and the upper body classifier 811 composed of the left middle class classifier 802 and the right middle class classifier 803 are designated. The head neighborhood classifier 812 including the upper left discriminator 800 and the upper right discriminator 801 is stored together with the threshold value applied to the integrated evaluation value in association with the high-density class.
Although the example in which the partial classifier of the same portion is shared between the densities is shown here, the division of the portion may be different for each density.

The weight 820 is a degree of emphasizing the portion according to the density of each portion of the object and the corresponding partial region, and is represented by a relative ratio between the densities. The lower the density, the more important the portion, and the higher the density, the lighter the weight. Therefore, the lower the density, the higher the density, and the lower the weight 820 is set. For example, low density, medium density and high density weights 820 can have a ratio of 10: 7: 5. The background class can be regarded as a low density class when determining the density of a partial region.

As described above, the weight storage means 412b stores the weight for each density, which is higher as the density is lower and lower as the density is higher.

The candidate position setting means 511b sets a plurality of candidate positions in the captured image at predetermined intervals, and outputs the set candidate positions to the evaluation value calculating means 514b. Specifically, the predetermined interval is one pixel, and the candidate position setting means 511b sequentially sets the position of each pixel of the captured image to the candidate position. The candidate position represents the center of gravity of the human head.

The evaluation value calculating means 514b sets a partial area corresponding to each of a plurality of parts constituting a single object in the captured image with reference to each candidate position input from the candidate position setting means 511b, and sets a plurality of parts. The image feature of the partial area corresponding to each of the above is input to the partial classifier that learned the image feature of the relevant part, the partial evaluation value of the partial area is calculated, and the part set for each candidate position based on the candidate position. The integrated evaluation value is calculated by emphasizing the partial evaluation value of the region as the density of the partial region decreases, and the calculated integrated evaluation value and the information associated therewith are output to the position-fixing means 517b.
At that time, the evaluation value calculating means 514b sets a partial region corresponding to a small part of the parts constituting a single object as the density at the candidate position increases at each candidate position.

Therefore, the evaluation value calculation means 514b sets an upper 1/3 window at each candidate position, and totals the estimated density in the window with reference to the density distribution input from the density estimation means 50. However, the background class is excluded and aggregated. Then, the evaluation value calculation means 514b determines the highest estimated density at each candidate position as the density of the candidate position.

Further, the evaluation value calculating means 514b reads out the information of the partial classifier corresponding to the density of each candidate position from the partial classifier storage means 411b, and sets each candidate position as a partial classifier associated with the density of the candidate position. The corresponding window (partial area) is set, and the feature amount for identification (feature amount for identification) is extracted from the captured image in each partial area. These partial regions have the shape of the partial image (solid rectangle shown in FIG. 13) used for learning the partial classifier of each part, and the size of the window is enlarged or reduced by a plurality of predetermined magnifications. Is. Six sub-regions that make up the "whole" of a person for low-density candidate positions, and four sub-regions that make up the "upper two-thirds" of a person when combined for medium-density candidate positions, high Two subregions, which together form the "upper 1/3" of a person, are set for the density candidate positions. The identification feature amount is the same type as the feature amount of the learning image, and is a HOG feature amount.

Further, the evaluation value calculation means 514b inputs the identification feature amount of each part into the partial classifier of the part and acquires the partial score which is the output value as the partial evaluation value.

In addition, the evaluation value calculation means 514b calculates a weighting coefficient for each subregion according to the estimated density of the subregion.

Therefore, the evaluation value calculation means 514b aggregates the estimated densities in each sub-region with reference to the density distribution, and determines the highest estimated density in each sub-region as the density of the sub-region. However, the background class is regarded as a low density class and aggregated.
Next, the evaluation value calculation means 514b reads out the weights (ratio of weighting coefficients) corresponding to the density of each partial region from the weight storage means 412b, and sums the weights of the partial regions set for each candidate position based on the candidate position. Ask for.
Subsequently, the evaluation value calculating means 514b calculates the weighting coefficient of the subregion by dividing the weight of each subregion by the sum of the weights of all the subregions for each candidate position. That is, the sum of the weighting coefficients at each candidate position is normalized to be 1.
In this way, the evaluation value calculating means 514b sets a weighting coefficient for each subregion, which is higher as the density in the subregion is lower and lower as the density in the subregion is higher.

Further, the evaluation value calculation means 514b calculates the integrated evaluation value by weighting the partial evaluation values of each sub-region set based on the candidate position with the weighting coefficient of the sub-region for each candidate position and summing them up.

That is, the partial score by the upper left discriminator is _SUL , the partial score by the upper right discriminator is _SUR , the partial score by the left middle discriminator is _SML , the partial score by the right middle discriminator is _SMR , and the lower left discriminator. The partial score by is S _LL , the partial score by the lower right classifier is S _LR , the weight coefficient in the upper left is W _UL , the weight coefficient in the upper right is W _UR , the weight coefficient in the middle left is W _ML , and the weight in the middle right is W ML. Assuming that the coefficient is W _MR , the lower left weight coefficient is W _LL , and the lower right weight coefficient is W _LR , the evaluation value calculation means 514b calculates the integrated score as follows.

If the density of the candidate position of interest is low, the evaluation value calculating means 514b calculates the integrated score of the candidate position by the following equation.
Integrated score = W _UL S _UL + W _UR S _UR + W _ML S _ML + W _MR S _MR
+ W _LL S _LL + W _LR S _LR (3)
Further, if the density of the candidate position of interest is medium density, the evaluation value calculating means 514b calculates the integrated score of the candidate position by the following equation.
Integrated score = W _UL S _UL + W _UR S _UR + W _ML S _ML + W _MR S _MR (4)
Further, if the density of the candidate positions of interest is high, the evaluation value calculating means 514b calculates the integrated score of the candidate positions by the following equation.
Integrated score = W _UL S _UL + W _UR S _UR (5)

FIG. 14 is a diagram schematically showing how the evaluation value calculation means 514b calculates the integrated score for each candidate position illustrated in FIG. 7 when the density distribution illustrated in FIG. 7 is obtained. Image 830 shows the relationship between each part and the weighting coefficient for three candidate positions having a low density among these candidate positions. Image 831 shows the relationship between each part and the weighting factor for the three candidate positions having medium densities. Image 832 shows the relationship between each part and the weighting factor for the two candidate positions with high densities.

For example, since the density of the candidate position 840 is low, six subregions are set based on the candidate position 840, and since the densities of the six subregions are all low, the weight coefficient ratio of each subregion is low. Is 10, and the sum is 60. Therefore, the weighting coefficient is W _UL = W _UR = W _ML = W _MR = W _LL = W _LR = 10/60, and these weighting factors and the partial score of each part are substituted into the equation (3) to obtain the integrated score. Calculated.
Further, for example, since the density of the candidate position 841 is medium density, four subregions are set based on the candidate position 841, and the upper two of the four subregions have a medium density, so that the weighting coefficient The ratio is 7, and the lower two have low densities, so the weighting factor ratio is 10, and the sum is 34. Therefore, the weighting factors are W _UL = W _{UR = 7/34} and W ML = W _MR = 10/34, and the integrated score is calculated by substituting these weighting factors and the partial scores of each part into the equation (4). To.
Further, for example, since the density of the candidate position 842 is high, two subregions are set based on the candidate position 842, and since the densities of the two subregions are both high, the weighting coefficient of each subregion is high. The ratio is 5, and the sum is 10. Therefore, the weighting coefficient becomes W _UL = W _UR = 5/10, and the integrated score is calculated by substituting these weighting factors and the partial scores of each part into the equation (5).
The integrated score is calculated in the same manner for the other candidate positions.

Then, the evaluation value calculation means 514b outputs information in which the candidate position, the density of the candidate positions, the integrated score, and the sum region (integrated window) of the used partial regions are associated with each candidate position to the position determining means 517b.

The position determining means 517b refers to the information input from the evaluation value calculating means 514b, and determines the candidate position where the integrated evaluation value satisfying the predetermined determination criteria is calculated as the position of the object.

Specifically, the position-determining means 517b extracts candidate positions whose integrated score is equal to or higher than a predetermined threshold value (for example, 0), and among the extracted candidate positions, a plurality of extracted candidate positions having the same density and close to each other. The candidate positions (candidate positions where the overlap between the integrated windows is larger than the predetermined ratio) are combined into one, and the combined candidate positions are determined as the positions where the person is photographed.

This process of summarizing the candidate positions is performed in order to deal with the fact that a high integrated score is calculated for the same person not only in the position where the person is actually photographed but also in the vicinity thereof. Specifically, for example, the position-determining means 517b sets the candidate positions for which the integrated score equal to or higher than the threshold value is calculated to the attention positions in descending order of the integrated score for each density of the candidate positions, and the integrated score is higher than the attention position. Set the lower candidate position as the comparison position. Then, the position-determining means 517b deletes information on the comparison position in which the overlap between the integrated window at the comparison position and the integrated window at the attention position is larger than a predetermined ratio among the comparison positions, thereby selecting a plurality of candidate positions. Summarize in.

Then, the position-determining means 517b outputs the position where the person is photographed and the candidate position determined to be the position where the person is photographed to the object position output means 31 as the object position information.

<Operation of the image monitoring device 1 according to the second embodiment>
Hereinafter, the operation of the image monitoring device 1 according to the second embodiment will be described with reference to FIGS. 9 and 15.

When the image monitoring device 1 starts operation, the photographing unit 2 sequentially transmits the captured images, and the image processing unit 5 operates according to the flowchart of FIG. 9 each time the captured image is received, as in the first embodiment. repeat.

The communication unit 3 operates as the image acquisition means 30, receives the captured image, and outputs the captured image to the image processing unit 5 (step S1). The image processing unit 5 to which the captured image is input operates as the density estimation means 50, reads the density estimator from the density estimator storage means 40 of the storage unit 4, and scans the captured image with the density estimator to distribute the density. Is estimated (step S2).

Next, the image processing unit 5 operates as the object position determination means 51, and the object position determination means 51 receives the captured image from the image acquisition means 30 and the density distribution from the density estimation means 50, and the density distribution is other than the background class. It is confirmed whether or not the estimated density of is included (step S3).

When the object position determination means 51 includes an estimated density other than the background class (YES in step S3), the object position determination means 51 performs a process of determining the position of each object from the captured image (step S4), and only the background class. In the case of (NO in step S3), the processing of steps S4 and S5 is omitted.

The object position determination process in step S4 will be described with reference to the flowchart of FIG. The single feature storage means 41 operates as the partial classifier storage means 411b and the weight storage means 412b, and the object position determination means 51 operates as the candidate position setting means 511b, the evaluation value calculation means 514b, and the position determination means 517b. Judgment processing is executed.

The candidate position setting means 511b sequentially sets the position of each pixel in the captured image to the candidate position and inputs it to the evaluation value calculating means 514b (step S200), and controls the loop processing of steps S200 to S206.

The evaluation value calculation means 514b in which the candidate position is input specifies the density of the candidate position with reference to the density distribution (step S201). That is, the evaluation value calculation means 514b sets a window defined in the shape of the upper 1/3 of a single person at the candidate position, and sets the highest estimated density (excluding the background class) in the window as the density of the candidate position. Identify.

The evaluation value calculation means 514b that specifies the density of the candidate position reads out a plurality of partial classifiers according to the density from the partial classifier storage means 411b, sets a partial area corresponding to each partial classifier, and within the partial area. Each of the identification features is extracted from the captured image (step S202), the extracted identification features are input to the corresponding partial classifier, and each partial score (partial evaluation value) is calculated (step S203). ..

The evaluation value calculation means 514b that calculated the partial evaluation value specifies the density of each partial region with reference to the density distribution (step 204). That is, the evaluation value calculation means 514b specifies the highest estimated density in each subregion (however, the background class is regarded as the low density class) as the density of the subregion.

The evaluation value calculation means 514b that specifies the density of each partial region reads out the ratio of the weighting coefficient according to the density of each partial region from the weight storage means 412b, and sets the ratio of the weighting coefficient of each partial region as the weighting coefficient of all the partial regions. The weighting coefficient of the relevant partial region is calculated by dividing by the sum of the ratios of, and the integrated evaluation value of the candidate position is calculated by summing the calculated weighting coefficient and the partial evaluation value calculated in step S203 (step S205). .. When the density of the candidate region is low, the product is summed according to the formula (3), when the density is medium, the product is summed according to the formula (4), and when the density is high, the product is summed according to the formula (5).

Then, the evaluation value calculation means 514b records the candidate position, the window (integrated window) of the sum region of the partial region, the density of the candidate position, and the integrated evaluation value in association with each other, and all the pixels of the captured image. It is confirmed whether or not the position of is set to the candidate position (step S206), and if there is an unset pixel (NO in step S206), the process is returned to step S200 and the position of the next pixel is processed. To do.

On the other hand, when the positions of all the pixels have been set to the candidate positions (YES in step S206), the position determining means 517b has the candidate position, the integrated window, the density of the candidate positions, and the integrated evaluation value recorded in step S206. The pair whose integrated evaluation value is less than the threshold value is deleted from the pairs (step S207), and for the remaining pairs that are not deleted, the ratio of the integrated windows of each other is higher than the predetermined ratio for each density of candidate positions. The large overlapping groups are grouped into one group as the same person (step S208). Then, the position-determining means 517b determines that the candidate positions of each set after being collected are the positions (object positions) of the individual persons photographed in the captured image.

The description will be continued with reference to FIG. 9 again. The object position determination means 51 outputs the object position information determined in step S4 to the communication unit 3 (step S5), and the communication unit 3 operates as the object position output means 31 to display the object position information on the display unit 6. Send.

When the above processing is completed, the processing is returned to step S1, and the processing for the next captured image is performed.

<Modification example>
(1) In each of the above embodiments and modifications thereof, an example in which the object to be detected is a human is shown, but the object to be detected is not limited to this, and the object to be detected is an animal such as a vehicle, a cow or a sheep. You can also.

(2) In each of the above-described embodiments and modifications thereof, an example is shown in which an object is divided into three parts in the height direction and a partial area is set in units of two parts in the width direction, but the division method is limited to this. Absent. Depending on the difference in the detection target, the characteristics of the monitoring space to be photographed, the feature amount to be adopted, the type of evaluation value, etc., the partial area can be divided by a different ratio suitable for each. Further, the partial areas may be set by overlapping.

(3) The weights shown in each of the above embodiments and their modifications are examples, and are suitable for each of them according to the difference in the detection target, the characteristics of the monitoring space to be photographed, the feature amount to be adopted, the type of evaluation value, and the like. It can be another value.

(4) In each of the above embodiments and each modification thereof, the density estimator learned by the multi-class SVM method is illustrated, but instead of the multi-class SVM method, a decision tree type random forest method and a multi-class method are used. Various density estimators such as density estimators learned by the AdaBoost method or the multiclass logistic regression method can be used.
Alternatively, it can be a density estimator using an identification type CNN (Convolutional Neural Network).

(5) In each of the above-described embodiments and modifications thereof, the density classes other than the background estimated by the density estimator are set to 3 classes, but the classes may be further divided.
In that case, instead of the three-step weight, a finer-step weight corresponding to the classification can be used, and the class and the weight can be associated and stored in the single feature storage means 41. Alternatively, the class and the weights of the three stages can be associated with each other in a many-to-one manner and stored in the single feature storage means 41.

(6) In each of the above embodiments and each modification thereof, a density estimator classified into multiple classes is illustrated, but instead of this, a regression type density estimation that returns a density value (estimated density) from a feature amount is used. It can also be used as a vessel. That is, with a density estimator that has learned the parameters of the regression function for obtaining the estimated density from the features by the ridge regression method, the support vector regression method, the random forest method of the regression tree type, or the Gaussian process regression. can do.
Alternatively, it can be a density estimator using a regression type CNN.
In these cases, the range of the estimated density output as a continuous value instead of the value of the density class is stored in the single feature storage means 41 in association with the partial model and the weight, or the partial classifier and the weight. In these cases, the density of the partial region can be the average value or the median value of the estimated densities in the partial region, in addition to the highest estimated density in the partial region.

(7) In each of the above-described embodiments and modifications thereof, the GLCM features are exemplified as the feature amount learned by the density estimator and the feature amount for estimation, but these are local binary patterns (Local) instead of the GLCM features. Binary Pattern: LBP) features, Haar-like features, HOG features, luminance patterns, and other features, or GLCM features and a combination of multiple of these. You can also do it.

(8) In each of the above-described embodiments and modifications thereof, an example is shown in which the density estimation means 50 and the object position determination means 51 scan at one pixel interval to perform processing, but these scans are performed by two or more pixels. It is also possible to do it at intervals.

(9) In each of the above-described embodiments and modifications thereof, an example in which the candidate position is selected and set from the region where the estimated density is low density, medium density, or high density is shown, but the arrangement generation means 510a and the candidate position are shown. Each of the setting means 511b can also set the candidate position only within the change area. In that case, the storage unit 4 includes a background image storage means (not shown) for storing the background image of the monitoring space, and the image processing unit 5 performs difference processing between the captured image and the background image so that the difference value is a predetermined difference. A change area extraction means for extracting a group of pixels having a threshold value or more as a change area, or performing a correlation process between a captured image and a background image to extract a group of pixels having a correlation value equal to or less than a predetermined correlation threshold value as a change area. (Not shown), each of the arrangement generation means 510a and the candidate position setting means 511b sets a candidate position with reference to the change area extracted by the change area extraction means.
When limiting the area for setting the candidate position, the arrangement generation means 510a can change the upper limit number of arrangements according to the size of the limited area.
By limiting the area for setting the candidate position in this way, it is possible to prevent accidental similarity between the captured image and the model image or accidental calculation of a high discrimination score for the background, and it is possible to reduce erroneous detection of the object position.

(10) In the first embodiment and each modification thereof, an example in which the arrangement generation means 510a randomly generates an arrangement each time the repetition is performed is shown, but from the candidate position one time before the second and subsequent repetitions. Arrangements may be generated by updating to slightly shifted candidate positions, or stochastically by the MCMC (Markov chain Monte Carlo) method with reference to the similarity to the previous arrangement after the second iteration. Arrangements may be generated by sequentially improving the candidate positions by a method of searching for a candidate position or a mountain climbing method.

(11) In each of the above embodiments and each modification thereof, a projection area of the model defined in the shape of the upper 1/3 of the person or a window defined in the shape is set at the candidate position of interest. An example of determining the estimated density at the candidate position by aggregating the estimated densities in the area is shown, but in order to reduce the processing amount, the pixels at the candidate position and the area near 8 of the candidate position are replaced with the area. Alternatively, it may be a small region such as a region near 16. Alternatively, in order to increase the accuracy, the projection area of the model defined in the shape of the upper two-thirds of a single person whose representative position is the candidate position, the window defined in the shape, or the candidate position is used instead of the region. It can also be a projection area of the model defined in the shape of the whole body of a single person as a representative position or a large area such as a window defined in the shape.

(12) In the second embodiment and its modifications, the partial classifier learned by the linear SVM method is illustrated, but various conventionally known learning methods such as the AdaBoost method are used instead of the linear SVM method. It can also be a partial classifier learned using the method. A pattern matching device can also be used instead of the classifier, and the partial score in that case is the inner product of the average pattern of the features extracted from the human learning image and the features of the input image, and the evaluation value is calculated. The function can be a function that uses the score as an output value and the feature amount of the captured image as an input value. Further, an identification type CNN may be used as the partial classifier.

(13) In the second embodiment and each modification thereof, HOG features are exemplified as features used by the partial classifier for learning and discrimination, but these are local binary pattern features instead of HOG features. It can be a variety of features such as a quantity, a haul-like feature, and a luminance pattern, or it can be a combination of a HOG feature and a plurality of these features.

(14) In the second embodiment and each modification thereof, an example in which the evaluation value calculation means 514b normalizes so that the sum of the weights is 1 for each candidate position is shown, but further in the horizontal direction (horizontal direction). ) May be normalized so that the sum of the weights is 1.

(15) In the second embodiment and each modification thereof, an example in which the degree of emphasis on the part of the object is switched by the weight is shown, but the threshold value (partial threshold value) for the partial evaluation value can also be switched. In that case, for example, the single feature storage means 41 stores a predetermined partial threshold value corresponding to each density, which is lower as the density is lower and higher as the density is higher, and the evaluation value calculation means 514b has a partial evaluation value equal to or higher than the partial threshold value. The ratio of the number of subregions is calculated as the integrated evaluation value. Then, the position determining means 517b determines the candidate position whose integrated evaluation value is equal to or higher than a predetermined threshold value (for example, 50%) as the object position.

According to each of the above embodiments and variations thereof, the object detection device switches the characteristics and the degree of emphasis of the portion suitable for the concealed state that can occur in the object depending on the candidate position and the density of each partial region. Since the position of each object is determined, it is possible to detect the object with high accuracy in accordance with the change in the concealed state of the object due to the change in the congestion state.

Further, the object detection device according to the first embodiment and its modification has a density of a partial model representing the image features of each part of the object and a degree of importance when evaluating the partial evaluation value of the captured image of the partial model. By switching according to the above, it is possible to detect an object with high accuracy adapted to the change in the concealed state of the object due to the change in the congestion state.

Further, the object detection device according to the second embodiment and the modified example thereof attaches importance to the partial classifier that has learned the image features of each part of the object and the partial evaluation value for each part by the partial classifier. By switching the degree according to the density, it is possible to detect an object with high accuracy adapted to the change in the concealment state of the object due to the change in the congestion state.

1 ... Image monitoring device 2 ... Imaging unit 3 ... Communication unit 30 ... Image acquisition means 31 ... Object position output means 4 ... Storage unit 40 ... Density estimator storage means 41 ... Single feature storage means 410a ... Object model storage means 411b ... Partial classifier storage means 412a, 412b ... Weight storage means 5 ... Image processing unit 50 ... Density estimation means 51 ... -Object position determination means 510a ... Arrangement generation means 511b ... Candidate position setting means 512a ... Model image generation means 514a, 514b ... Evaluation value calculation means 516a ... Optimal arrangement determination means 517b ... Position determination means 6 ... Display unit

Claims (5)

An object detection device that detects individual objects from captured images in a space where congestion due to a predetermined object can occur.
Density image of the space where the object exists is estimated for each predetermined density Using a density estimator that learns the image features of each image, the distribution of the density of the object captured in the photographed image is estimated. Density estimation means and
A candidate position where each object may exist in the captured image is set, and a partial region corresponding to each of a plurality of portions constituting the object is set in the captured image with reference to the candidate position. For each of the partial regions, a partial evaluation value indicating the degree to which the image feature of the portion corresponding to the partial region appears is calculated, and the partial evaluation value of the plurality of the partial regions set based on the candidate position is used as the partial evaluation value. An object position determination means for determining a candidate position in which the integrated evaluation value integrated with emphasis as the density of the region is lower and satisfying a predetermined determination criterion is the position of the object.
An object detection device characterized by being equipped with. The object position determining means sets a weight coefficient for each of the partial regions, which is higher as the density in the partial region is lower and lower as the density is higher in the partial region, and a plurality of said portions set based on the candidate position. The object detection device according to claim 1, wherein the partial evaluation value of the region is weighted by the weighting coefficient of the partial region and summed to calculate the integrated evaluation value. The object detection device according to claim 1 or 2, wherein the object position determination means sets the partial region corresponding to a smaller portion of the portions constituting the object as the density at the candidate position increases. The object position determination means is
Arrangement generation means, each of which generates a plurality of different arrangements including one or more of the candidate positions, and
A model image generation means for generating a model image by drawing a partial model imitating the portion in the partial region corresponding to each of the plurality of portions based on each of the candidate positions for each of the plurality of arrangements. When,
For each of the model images in the plurality of arrangements, the partial evaluation value indicating the degree of similarity of the partial model to the captured image is calculated for each of the partial regions, and the partial evaluation values of the plurality of the partial regions are integrated. And the evaluation value calculation means for calculating the integrated evaluation value,
Optimal placement determining means for determining the candidate position in the placement with the maximum integrated evaluation value as the position of the object, and
The object detection device according to any one of claims 1 to 3. The object position determination means is
Candidate position setting means for setting a plurality of the candidate positions in the captured image, and
The image feature of the partial region corresponding to each of the plurality of portions set based on each candidate position is input to the classifier that has learned the image feature of the portion, and the partial evaluation value of the partial region is calculated. An evaluation value calculation means for calculating the integrated evaluation value by integrating the partial evaluation values of a plurality of the partial regions set based on the candidate position for each candidate position.
A position-determining means for determining the candidate position from which the integrated evaluation value satisfying the determination criteria is calculated as the position of the object, and
The object detection device according to any one of claims 1 to 3.

Images