JP2021149687A - Device, method and program for object recognition - Google Patents

Abstract

To provide an object recognizing technique capable of preventing efficiently, reduction of accuracy of object recognition due to congestion.SOLUTION: An object recognizing device 1 recognizes an object, on the basis of photographed images photographed by plural imaging means 10a, 10b, 10c having a common visual field. Congestion degree estimating means 130 estimates a congestion degree of an object photographed in the photographed image, for every imaging means 10a, 10b, 10c. Individual recognizing means 131 recognizes a whole part or a part of the object in the photographed image by analyzing photographed images for every imaging means 10a, 10b, 10c, for generating an individual recognizing result. Weighting determining means 132 determines weighting of each imaging means 10a, 10b, 10c, according to the congestion degree of the position where the individual recognizing means 131 recognizes the object in the photographed images photographed by the imaging means 10a, 10b, 10c. Integration recognizing means 133 integrates the individual recognizing results of the imaging means 10a, 10b, 10c on the basis of the weighting.SELECTED DRAWING: Figure 1

Description

The present invention relates to a technique for recognizing an object based on an image, and more particularly to a technique for recognizing an object based on an image taken by a plurality of photographing means having a common field of view.

For the purpose of security and the like, an object such as a person is detected and tracked from an image taken by a camera, or an object is recognized by recognizing a posture. At that time, the recognition accuracy can be improved by giving a common field of view to the plurality of cameras and shooting from a plurality of directions.

For example, Patent Document 1 is a moving object tracking device that tracks moving objects such as people with a plurality of cameras having a common field of view, and template matching is performed on captured images of each camera to obtain each camera. A moving object tracking device that weights the positions of moving objects by their likelihood and integrates them in a common coordinate system is described. By doing so, moving objects overlap each other on the image of some cameras, and even if the likelihood of that camera decreases, it can be supplemented with the information of other cameras, so that highly accurate tracking can be continued.

Japanese Unexamined Patent Publication No. 2010-049296

However, in the prior art, there is a problem that a decrease in recognition accuracy caused by congestion around an object of interest cannot be effectively prevented. That is, for example, in the moving object tracking device described in Patent Document 1, as long as objects of the same type overlap, a high likelihood may be accidentally generated due to erroneous matching, and which camera is based on the likelihood obtained after the fact. It was difficult to exclude the position obtained by the erroneous matching from the integration because it was not possible to distinguish whether or not the erroneous matching was performed in the captured image of. Then, the higher the degree of congestion, the more likely it is that erroneous matching will occur.

The present invention has been made in view of the above problems, and an object of the present invention is to provide an object recognition device, an object recognition method, and an object recognition program capable of effectively preventing a decrease in the accuracy of object recognition caused by congestion. do.

(1) The object recognition device according to the present invention is an object recognition device that recognizes an object based on captured images taken by a plurality of photographing means having a common field of view, and the photographed image is photographed for each of the photographing means. The congestion degree estimating means for estimating the degree of congestion of the object and the photographed image for each photographing means are analyzed to recognize all or a part of the object on the photographed image and generate an individual recognition result. Individual recognition means, weighting determination means for determining the weighting of each imaging means according to the degree of congestion at the position where the individual recognition means recognizes the object on the captured image captured by each imaging means. An integrated recognition means that integrates the individual recognition results for each of the photographing means based on the weighting is provided.

(2) In the object recognition device according to the present invention described in (1) above, when the captured image is input, the congestion degree estimating means outputs the congestion degree at an arbitrary position in the captured image. The captured image is input to a pre-learned estimator to estimate the degree of congestion at an arbitrary position in the captured image, and the weighting determining means is used for each region of the captured image according to the degree of congestion. The weighting of the photographing means is determined.

(3) In the object recognition device according to the present invention according to the above (1) or (2), the individual recognition means analyzes the photographed image for each image pickup means and displays the photographed image on the photographed image at the current time. The position information of the object is obtained, and the integrated recognition means integrates the position information for each imaging means based on the weighting to determine the position of the object at the current time.

According to the present invention, it is possible to provide an object recognition device, an object recognition method, and an object recognition program that can effectively prevent a decrease in the accuracy of object recognition caused by congestion.

It is a block diagram which shows the schematic structure of the 3D position estimation apparatus. It is a figure which shows the relationship between the person and the crowd, and the photographed image of each photographing means. It is an enlarged view of the person 200 of FIG. It is a schematic flow chart which shows the whole processing of the 3D position estimation apparatus in Embodiment 1. FIG. It is a subflow diagram which shows the 3D position estimation process. It is a block diagram which shows the schematic structure of the 3D tracking device. It is a figure which shows the relationship between the tracking person and the crowd, and the photographed image of each photographing means. It is explanatory drawing of the hypothesis of the tracking person, the likelihood and the weighting. It is a flow chart which shows the overall processing of a 3D tracking apparatus. It is explanatory drawing explaining another example of the object recognition apparatus.

[Embodiment 1]
Hereinafter, a three-dimensional position estimation device, which is an example of the object recognition device according to the embodiment of the present invention (hereinafter referred to as the first embodiment), will be described. The three-dimensional position estimation device estimates the three-dimensional position of a person in a common field of view based on captured images taken by a plurality of photographing means having a common field of view.

FIG. 1 is a block diagram showing a schematic configuration of the three-dimensional position estimation device 1. The three-dimensional position estimation device 1 includes photographing means 10a, 10b, 10c, a communication unit 11, a storage unit 12, an image processing unit 13, and a display unit 14.

The photographing means 10a, 10b, and 10c are cameras that acquire an image that is a collection of target data, and are surveillance cameras in the present embodiment. The photographing means 10a, 10b, and 10c have a common field of view and are synchronized. The photographing means 10a, 10b, and 10c are connected to the image processing unit 13 via the communication unit 11, captures the monitoring space at predetermined time intervals to generate an image, and sequentially inputs the generated images to the image processing unit 13. do. For example, the photographing means 10a, 10b, and 10c are installed on an indoor wall which is a monitoring space with a predetermined fixed field of view overlooking the monitoring space, and the monitoring space is photographed at a time interval of 1/5 second in color. Generate an image or a monochrome image. Although the first embodiment shows an example of three photographing means, at least two photographing means may be sufficient. In order to increase the possibility that an image with a low degree of congestion is taken, the larger the number of shooting means, the better, and the larger the directional difference from the center of gravity of the common field of view to the installation position of each shooting means.

The photographing means 10a, 10b, and 10c are calibrated in advance, and a common three-dimensional coordinate system (so-called world coordinate system) is defined. Hereinafter, this coordinate system will be referred to as an XYZ coordinate system. Further, a two-dimensional coordinate system (so-called camera coordinate system) peculiar to each photographed image of the photographing means 10a, 10b, and 10c is referred to as an xy coordinate system.

The communication unit 11 is a communication circuit, one end of which is connected to the image processing unit 13, and the other end of which is connected to the photographing means 10a, 10b, 10c and the display unit 14. The communication unit 11 acquires an image from the photographing means 10a to 10c and inputs it to the image processing unit 13. Further, the communication unit 11 outputs the recognition result of the object from the image processing unit 13 to the display unit 14.

The photographing means 10a to 10c, the communication unit 11, the storage unit 12, the image processing unit 13, and the display unit 14 are appropriately connected in a form according to the installation location of each unit. For example, when the photographing means 10a to 10c and the communication unit 11 and the image processing unit 13 are remotely installed, the photographing means 10a to 10c and the communication unit 11 can be connected by an internet line. Further, the communication unit 11 and the image processing unit 13 may be connected by a bus. In addition, a LAN (Local Area Network), various cables, or the like can be used as the connection means.

The storage unit 12 is a memory device such as a ROM (Read Only Memory) and a RAM (Random Access Memory), and stores various programs and various data. For example, the storage unit 12 stores learning data and information of an estimator that is a learned model, and inputs and outputs such information to and from the image processing unit 13. That is, information used for learning the estimator, information generated in the process of the processing, and the like are input / output between the storage unit 12 and the image processing unit 13.

The image processing unit 13 is composed of arithmetic units such as a CPU (Central Processing Unit), a DSP (Digital Signal Processor), an MCU (Micro Control Unit), and a GPU (Graphics Processing Unit). The image processing unit 13 operates as various processing means / control means by reading a program from the storage unit 12 and executing the program, reads various data from the storage unit 12 as necessary, and stores the generated data in the storage unit 12. Remember. For example, the image processing unit 13 learns and generates an estimator, and stores the generated estimator in the storage unit 12 via the communication unit 11.

The display unit 14 is a liquid crystal display, an organic EL (Electro-Luminescence) display, or the like, and displays a recognition result of a moving object input from the image processing unit 13 via the communication unit 11.

The image processing unit 13 functions as a congestion degree estimation means 130, a two-dimensional position estimation means (individual recognition means) 131, a weighting determination means 132, a three-dimensional position estimation means (integrated recognition means) 133, and an estimation result output means 134. ..

The congestion degree estimating means 130 estimates the congestion degree of the object photographed in the photographed image for each of the photographing means 10a, 10b, and 10c. In the present embodiment, the congestion degree estimation means 130 inputs the photographed image into the estimator learned in advance so as to output the congestion degree at an arbitrary position in the photographed image when the photographed image is input, and within the photographed image. Estimate the degree of congestion at any position in. Specifically, the congestion degree estimation means 130 inputs a captured image into an estimator trained in advance to output a congestion degree map that estimates the congestion degree of each pixel when an image is input, and the captured image is of the captured image. The congestion degree map is output, and the obtained congestion degree map is stored in the storage unit 12.

The estimator can be specifically realized by using deep learning technology. That is, the estimator can be modeled by a CNN (convolutional neural network) that outputs a congestion map of the image when the image is input. For learning, for example, a large number of learning images taken by a crowd and a probability density function having a variance according to the size of the head, with the position of the center of gravity of each person's head in each of the learning images as the average value. Is set and the value of the function for each head is added for each pixel to prepare a congestion degree map. Then, learning is performed in advance to bring the output when each of the training images is input to the model closer to the congestion degree map corresponding to the image. The trained model thus obtained is stored in the storage unit 12 as an estimator forming a part of the program of the congestion degree estimation means 130. For example, MCNN (multi-column convolutional neural network) described in “Single image crowd counting via multi-column convolutional neural network”, Zhang, Y., Zhou et al., CVPR 2016 is an example of an estimator. The crowd density map described in is an example of a congestion map. In the present embodiment, the congestion degree estimation means 130 sets an upper limit value T0 of the congestion degree that can tolerate a decrease in recognition accuracy in advance, divides the congestion degree output from the estimator by the upper limit value T0, and then divides the result. When is 1.0 or more, standardization to 1.0 shall be performed. That is, in the present embodiment, the range of the degree of congestion is [0,1].

The congestion degree estimation means 130 extracts a region in which the congestion degree is a predetermined threshold value T1 or more in each congestion degree map as a high congestion degree region. The congestion degree estimating means 130 sends the congestion degree information in which the photographing means ID for identifying each of the photographing means 10a to 10c and the high congestion degree region in the photographed image of the photographing means 10a to 10c are associated with each other in the weighting determination means 132. Output.

The two-dimensional position estimation means 131, which is an individual recognition means, analyzes the captured image of each photographing means, recognizes all or a part of the object on the captured image, and generates an individual recognition result. Specifically, the captured images taken by each of the photographing means 10a to 10c are input to the detector that has learned the detection of the human image region (personal region) from the image in advance, and the captured images are input to the detector. The individual recognition result is generated by outputting (detecting) the person area in the above image, generating an individual recognition result in which the photographing means ID of the photographing means 10a to 10c is associated with the detected person area and the position of the center of gravity of the person area, and the generated individual recognition result. Is output to the weighting determination means 132 and the three-dimensional position estimation means 133.

The detector causes, for example, deep learning of a CNN by using learning data consisting of a large amount of learning images and correct answer data indicating a person area surrounding a person's image in the learning image. It is a trained model. An example of such a CNN is described in “Faster R-CNN: Towards real-time object detection with region proposal networks”, Shaoqing Ren et al., NIPS, 2015.

The weighting determining means 132 determines the weighting of each photographing means according to the degree of congestion at the position where the individual recognition means recognizes the object on the photographed image photographed by the photographing means 10a, 10b, 10c.

Specifically, the weighting determination means 132 refers to the individual recognition result input from the two-dimensional position estimation means 131, and is the upper 1/3 of each person area included in the individual recognition result for each photographing means (hereinafter, hereafter. (Also referred to as the head region) is set as "the position where the individual recognition means recognizes the object". Then, the weighting determining means 132 refers to the congestion degree information input from the congestion degree estimating means 130, and calculates as a weight the ratio of the head region not occupied by the high congestion degree region of the photographing means for each photographing means. Then, the individual recognition result including the weight is output to the three-dimensional position estimation means 133.

For example, the weight is determined by the following equation based on the non-overlapping rate between the head region and the high congestion region.
Weight = 1.0-Area of overlap between head region and high congestion area / area of head region The weight may be determined by the following formula according to the degree of quietness in the head region. In this case, the congestion degree estimation means 130 outputs the congestion degree information in which the photographing means ID and the congestion degree map are associated with each other.
-Weight = 1.0-Sum of congestion in the head region / Area of the head region In other words, the higher the degree of congestion in the head region, the smaller the weight. This means that the reliability of the individual recognition result is low due to the influence of the crowd behind. On the other hand, the lower the degree of congestion in the head region, the higher the weight. This means that the influence of the crowd is small and the reliability of the individual recognition result is high. Such a difference in weight occurs because the positional relationship between the object to be recognized and the crowd behind it on the captured image differs depending on the positional relationship with the photographing means. Therefore, by determining the weighting of each photographing means according to the degree of congestion at the position where the individual recognition means recognizes the object, it is possible to evaluate the reliability of the individual recognition result for the position which changes due to the influence of the crowd.

The storage unit 12 stores the camera parameters 120 of the photographing means 10a to 10c in order to back-project the position of the center of gravity of the person area obtained on the photographed image of the xy coordinate system onto the XYZ coordinate system. The camera parameter 120 includes external parameters such as the installation position and imaging direction of the photographing means 10a to 10c in the actual monitoring space, the focal length, the angle of view, the lens distortion and other lens characteristics of the photographing means 10a to 10c, and the number of pixels of the imaging element. It is information including internal parameters such as.

The three-dimensional position estimation means 133, which is an integrated recognition means, integrates the individual recognition results for each photographing means based on the weighting. In the first embodiment, the position information of each photographing means is integrated based on the weighting to determine the position of the object, and the determined position is output to the estimation result output means 134. The position information for each photographing means is the position of the center of gravity included in the individual recognition results for the photographing means 10a, 10b, 10c, and the position of the determined object is the three-dimensional position of the object.

Specifically, the three-dimensional position estimating means 133 first receives individual recognition results of the photographing means 10a, 10b, 10c input from the two-dimensional position estimating means 131, and the photographing means 10a stored in the storage unit 12. With reference to the camera parameters 120 of 10b and 10c, for each photographing means, the position of the center of gravity of each object included in the individual recognition result of the photographing means is back-projected to the XYZ coordinate system using the camera parameter 120 of the photographing means. Then, a line-of-sight vector passing through the position of the center of gravity of each object is derived.

Next, with reference to the weights of the photographing means 10a, 10b, and 10c input from the weighting determining means 132, the three-dimensional position where the weighted sum of the distances from the line-of-sight vector from each photographing means is minimized is determined for each object. Calculated as the three-dimensional position of the object.

The three-dimensional position P of each object is from the photographing means C passing through the position of the center of gravity of the object, where the weight of the photographing means C (the photographing means whose imaging means ID is C is referred to as the photographing means C) _{with respect to the object is W C.} When the line of sight distance between the vector V _C and the three-dimensional position P and _{D (V C, P),} ΣW C × D (V C, P) the three-dimensional position P which is the minimum by solving the minimum square method I want it. However, Σ is the sum of C.

It is difficult to specify in advance the combination of the line-of-sight vector from the photographing means 10a, the line-of-sight vector from the photographing means 10b, and the line-of-sight vector from the photographing means 10c by the same object. Therefore, for example, the three-dimensional position estimation means 133 attempts to calculate the three-dimensional position for the round-robin combination, deletes the combination in which the weighted sum of the minimized distances is equal to or greater than the predetermined threshold value TD. Only combinations in which the weighted sum of the minimized distances is less than the threshold value TD are considered to be the same object.

That is, the position of the center of gravity from the photographing means having a large weight is emphasized, and the position of the center of gravity from the photographing means having a small weight is disregarded and integrated to determine the three-dimensional position. By doing so, it is possible to perform highly accurate integration by reducing the influence of errors that occur in the individual recognition results for each photographing means due to the presence of the crowd. Therefore, the object can be recognized with high accuracy.

The estimation result output means 134 generates an estimation result and outputs it to the outside of the image processing unit 5. An image obtained by synthesizing the captured image and the projection drawing obtained by drawing a x mark representing the three-dimensional position of the person in the virtual space of the XYZ coordinate system and projecting the two dimensions is generated and output to the communication unit 11. It is transmitted by the communication unit 11 and displayed on the display unit 14.

Next, a processing example of the three-dimensional position estimation device 1 in the first embodiment will be described. As shown in FIG. 2, in each of the photographing means 10a, 10b, and 10c, the person 200 and the crowd 210 existing in the common field of view are photographed as photographed images 221, 222, 223.

The two-dimensional position estimation means 131 generates an individual recognition result for at least the person 200. That is, for the photographing means 10a, the person area 231 surrounding the person 200 and the position of the center of gravity thereof 241 are generated on the photographed image 221. The photographing means 10b generates a person area 232 surrounding the person 200 and a position of the center of gravity 242 thereof on the photographed image 222. The person area 232 is detected to be larger than the original person area due to the influence of the image of the crowd, and the center of gravity position 242 is also deviated from the original center of gravity position. For the photographing means 10c, the person area 233 surrounding the person 200 and the position of the center of gravity 243 thereof are generated on the photographed image 223. The congestion degree estimation means 130 extracts high congestion degree regions 251,252, 253 from the captured images 221, 222, 223.

The weighting determining means 132 calculates the weight according to the non-overlapping rate between the upper 1/3 (head area) of the person area and the high congestion area. For the photographing means 10a and 10c, there is no overlap between the upper 1/3 of the person areas 231,233 and the high congestion areas 251,253, and the weight is 1.0. Regarding the photographing means 10b, the upper 1/3 of the person area 232 and the high congestion area 252 overlap, and the weight is 0.2.

The three-dimensional position estimation means 133 uses the camera parameters 120 of the photographing means 10a, 10b, and 10c to derive the line-of-sight vectors V1, V2, and V3 passing through the center-of-gravity positions 241, 242, and 243, respectively. As for the photographing means 10b, since the person area 232 and the center of gravity position 242 are deviated from the original ones, the line-of-sight vector V2 is deviated from the line-of-sight vectors V1 and V3.

FIG. 3 is an enlarged view of the area around the person 200 in FIG. The three-dimensional position 360 is the three-dimensional position of the person 200 when the distance between the line-of-sight vectors V1, V2, and V3 is determined to be the minimum without weighting. The three-dimensional position 360 is a position deviated from the position of the center of gravity of the actual person 200.

The three-dimensional position 361 is a position determined so that the weighted sum of the distances from the line-of-sight vectors V1, V2, and V3 is minimized. The three-dimensional position 361 indicates the position of the center of gravity of the actual person 200. The distance D1 from the line-of-sight vector V1 to the three-dimensional position 361 and the distance D3 from the line-of-sight vector V3 to the three-dimensional position 361 are shorter than the distance D2 from the line-of-sight vector V2 to the three-dimensional position 361. This indicates that the distances D1 and D3 were evaluated with a large weight, and the distance D2 was evaluated with a small weight. In this way, by weighting the photographing means 10a, 10b, and 10c, the contribution of the line-of-sight vector V2 to the three-dimensional position 361 is reduced, and the contribution of the line-of-sight vectors V1 and V3 is increased, so that the calculation of the three-dimensional position 361 is highly accurate. Be made.

[Operation of 3D position estimation device 1]
FIG. 4 is a flowchart showing the overall processing of the three-dimensional position estimation device 1 in the first embodiment. Steps S100 to S150 of FIG. 4 are repeated every time a photographed image is input from the photographing means 10a, 10b, 10c.

The captured images from the photographing means 10a, 10b, and 10c are input to the image processing unit 13 (S100). The image processing unit 13 operates as the congestion degree estimation means 130, inputs each of the captured images from the photographing means 10a, 10b, and 10c into the estimator, generates a congestion degree map for each photographing means, and sets a threshold value from each congestion degree map. A highly congested region of T1 or higher is extracted (S110).

The image processing unit 13 operates as the two-dimensional position estimation means 131, inputs the captured images from the photographing means 10a, 10b, and 10c into the detector to detect the person area, and determines the photographing means ID, the person area, and the person area. An individual recognition result associated with the position of the center of gravity is generated (S120). The image processing unit 13 operates as the weighting determination means 132, inputs the high-congestion degree region and the individual recognition result, and weights according to the non-overlapping ratio of the head region and the high-congestion degree region of the upper 1/3 of the person region. Is determined (S130).

The image processing unit 13 operates as the three-dimensional position estimation means 133, inputs the individual recognition result and the weight, and estimates the three-dimensional position (S140). FIG. 5 is a sub-flow chart showing the processing of the three-dimensional position estimation means 133.

The three-dimensional position estimation means 133 reads out the camera parameter 120 from the storage unit 12, back-projects the position of the center of gravity of each person for each shooting means included in the individual recognition result, and passes through the center of gravity position from the shooting means. The line-of-sight vector of is calculated (S141). The three-dimensional position estimation means 133 generates a combination of line-of-sight vectors by brute force under the condition that one line-of-sight vector is selected for each of the photographing means 10a, 10b, and 10c, and the generated combinations are sequentially processed. Is set to (S142).

The three-dimensional position estimation means 133 derives the three-dimensional position at which the weighted sum of the distances from each line-of-sight vector constituting the combination is minimized for the combination to be processed (S143). The three-dimensional position estimation means 133 determines whether or not the weighted sum of the distances when the distance is minimized is less than the predetermined threshold value TD (S144). If the weighted sum of the distances is less than the threshold value TD, the process proceeds to S145, and if the weighted sum of the distances is greater than or equal to the threshold value TD, S145 is skipped and the process proceeds to S146. If the weighted sum of the distances is less than the threshold value TD, the storage unit 12 temporarily stores the three-dimensional position as a combination of line-of-sight vectors for the same object (S145).

The three-dimensional position estimation means 133 confirms whether or not all the combinations generated in step S142 have been processed (S146). When all the combinations have been processed, the process proceeds to S147, and if there are unprocessed combinations, the process returns to S142 to perform processing for the next combination.

Regarding the three-dimensional positions temporarily stored in step S145, the three-dimensional positions that are close to each other are grouped together as being related to the same person (S147). That is, since a plurality of three-dimensional positions may be calculated for one person, these duplications are eliminated. This prevents erroneous detection caused by the detection of a plurality of person areas for one person in the processing of the two-dimensional position estimation means 131. Further, it prevents erroneous detection caused by the weighted sum of the distances for the combination of the line-of-sight vectors of different objects in the combination generated by the three-dimensional position estimation means 133 in step S142 accidentally remaining below the threshold value TD. For example, the three-dimensional position estimation means 133 clusters the three-dimensional positions by using a method such as the group averaging method or the Ward method, and sets the representative value of each cluster as the three-dimensional position of one person. The three-dimensional position estimation means 133 erases the temporarily stored three-dimensional position and proceeds to step S150 of FIG.

The image processing unit 13 operates as the estimation result output means 134, inputs the three-dimensional position after the integration in step S147 to generate a display image indicating the position, and displays the display image via the communication unit 11 in the display unit 14. Is displayed (S150).

[Modification of Embodiment 1]
(1-1) In the first embodiment, the two-dimensional position estimation means 131, which is an individual recognition means, generates an individual recognition result by using the person area output by the detector as it is, but the person areas having a high degree of overlap are generated. The individual recognition result may be generated by calculating the position of the center of gravity after performing the process of combining them into one. In that case, there are methods such as selecting the person area having the highest likelihood at the time of detection, weighting with the likelihood at the time of detection, and averaging.

(1-2) In the first embodiment, an example of shooting with three shooting means 10a, 10b, 10c has been described, but the number of shooting means may be four or more. When the number of photographing means is four or more, the combination of the line-of-sight vectors generated by the three-dimensional position estimating means 133, which is the integrated recognition means, can be a combination of a number of line-of-sight vectors smaller than the number of the number of photographing means. For example, a combination of selecting the line-of-sight vectors of the three shooting means from the line-of-sight vectors of each of the four shooting means is generated by brute force.

(1-3) In the first embodiment, an example is shown in which the two-dimensional position estimation means 131, which is an individual recognition means, detects a person area from a captured image (so to speak, a still image) at each time. A person area may be detected by performing tracking processing of each person using (so to speak, a moving image). In that case, a person whose combination of line-of-sight vectors of the same object is once identified can omit the trial of the brute force combination thereafter.

[Embodiment 2]
In the second embodiment, a three-dimensional tracking device, which is an example of the object recognition device, will be described. The three-dimensional tracking device according to the second embodiment tracks a person in a common field of view based on captured images taken by a plurality of photographing means having a common field of view.

Further, in the second embodiment, tracking is performed by a method similar to the particle filter. At each time, for each object being tracked, a plurality of candidates for the position of the object are set, hypotheses corresponding to each candidate are set, and the position of the object is determined by integrating the hypotheses. In the present specification, a position determined once for each object being tracked at each time is referred to as an object position, and a plurality of candidates set for each of the objects being tracked at each time are referred to as a candidate position. That is, the candidate of the object position becomes the candidate position.

In the first embodiment, the upper part 1 of the person region in which the two-dimensional position estimation means 131, which is the individual recognition means, detects the "position where the individual recognition means recognizes the object" referred by the weighting determination means 132 when determining the weighting. It was set to 3/3, and the weighting target was the position of the center of gravity. In the second embodiment, the position where the candidate position setting / evaluation means 531 which is the individual recognition means calculates the likelihood of the object by referring to the "position where the individual recognition means recognizes the object" which the weighting determination means 532 refers to when determining the weighting. That is, the head projection area determined by the candidate position is used, and the weighting target is the likelihood. Hereinafter, the likelihood calculated by the candidate position setting / evaluation means 531 is referred to as an individual likelihood, and the likelihood obtained by integrating the individual likelihoods is referred to as an integrated likelihood.

FIG. 6 is a block diagram showing the configuration of the three-dimensional tracking device 5 according to the second embodiment. The photographing means 50a, 50b, 50c, the communication unit 51, and the display unit 54 are the same as the photographing means 10a, 10b, 10c, the communication unit 11, and the display unit 14 of the first embodiment. The image processing unit 53 functions as congestion degree estimation means 530, candidate position setting / evaluation means (individual recognition means) 531, weighting determination means 532, object position determination means (integrated recognition means) 533, and tracking result output means 534. .. In addition to the camera parameter 520, the storage unit 52 stores object information 521.

The congestion degree estimation means 530 of the second embodiment is the same as the congestion degree estimation means 130 of the first embodiment, but the output destinations are the weighting determination means 532 and the tracking result output means 534. The camera parameter 520 is the same as the camera parameter 120 of the first embodiment, but the second embodiment is used to project a candidate position or the like of the XYZ coordinate system onto the xy coordinate system.

The object information 521 stores information on a three-dimensional shape model of a moving object and the moving object being tracked. Specifically, the three-dimensional shape model of a moving object is a model formed by connecting three spheroids that imitate the three-dimensional shapes of the head, body, and legs of a standing person. Alternatively, a model that imitates the three-dimensional shape of the whole body of a standing person with one spheroid may be used.

The information of the moving object being tracked is associated with the object ID that identifies each person being tracked, the template of the person associated with the shooting means ID of each shooting means, and the XYZ coordinate system of the person. The position of the object and the hypothesis of the person are stored. For each hypothesis, the hypothesis ID and the candidate position in the XYZ coordinate system are stored. In addition, each hypothesis corresponds to the imaging means ID of each imaging means, and the whole body projection area and the head projection area of the three-dimensional shape model arranged at the candidate positions on the xy coordinate system of the imaging means, and the imaging means. The individual likelihood of the candidate position calculated using the captured image of the above and the weight of the candidate position of the photographing means are stored.

The candidate position setting / evaluation means 531 which is an individual recognition means analyzes the photographed image of each photographing means, recognizes all or a part of the object on the photographed image, and generates an individual recognition result. In the second embodiment, for each of the objects being tracked, the candidate position at the current time is predicted from the past position information (object position or candidate position), and each candidate position and object shape are predicted on the captured image captured by each imaging means. The regions (whole body projection region and head projection region) determined by It is generated as a recognition result and stored in the object information 521 of the storage unit 52.

Specifically, the candidate position setting / evaluation means 531 first refers to the object information 521 stored in the storage unit 52, and for each person being tracked, the current object position (object position) is set to the past object position for each person being tracked. Estimated value) is extrapolated, and multiple candidate positions are randomly set near the current object position. Further, the current candidate position may be extrapolated to the past candidate position. For a person whose past object position or past candidate position is less than 2 hours, the candidate position is set in the vicinity of the object position 1 hour ago. The object position and the candidate position at this stage are the coordinate values of the XYZ coordinate system.

Next, the candidate position setting / evaluation means 531 refers to the three-dimensional shape model of the object information 521 stored in the storage unit 52 and the camera parameter 520, and for each of the candidate positions, the three-dimensional shape arranged at the candidate position. The model is projected onto the xy coordinate system of the photographing means 10a, 10b, 10c. Further, for each of the candidate positions, the three-dimensional shape model of the head arranged at the candidate position is projected onto the xy coordinate system of the photographing means 10a, 10b, 10c. Subsequently, the candidate position setting / evaluation means 531 generates a hypothesis including the candidate position, the whole body projection area and the head projection area to each imaging means for each candidate position of each person being tracked, and the object information 521. Add to. Then, the candidate position setting / evaluation means 531 extracts the image feature of the whole body projection region in the captured image of the photographing means 10a, 10b, 10c for each candidate position of each person being tracked, and the image feature of the template of the person. The individual likelihoods La, Lb, and Lc are calculated based on the similarity with the above, and the calculated individual likelihoods La, Lb, and Lc are added to the corresponding hypothesis to update the object information 521. The upper 1/3 of the whole body projection area may be approximately used as the head projection area. Further, when the three-dimensional shape of the whole body is one spheroid, the upper third of the whole body projection area may be the head projection area.

The weighting determining means 532 determines the weight W of each photographing means according to the degree of congestion at the position where the individual recognition means recognizes the object on the photographed image photographed by each photographing means. In the second embodiment, for each candidate position, every shooting means 10a, 10b, 10c according to the degree of congestion of the head projection area determined by the candidate position and the object shape on the shot image taken by each shooting means. The weights Wa, Wb, and Wc for the individual likelihoods La, Lb, and Lc of

Specifically, the object information 521 stored in the storage unit 52 and the congestion degree information input from the congestion degree estimation means 530 are referred to, and the heads of the photographing means 10a, 10b, and 10c are referred to for each candidate position. The non-overlapping degree of the high congestion degree region with respect to the projection region is calculated as the weights Wa, Wb, and Wc, and the calculated weight W is added to the corresponding hypothesis to update the object information 521. Here, the degree of quietness may be the weight W instead of the degree of non-overlap.

That is, the weight W becomes smaller as the individual recognition result has a higher degree of congestion in the head projection area. This means that the reliability of the individual recognition result is low due to the influence of the crowd behind. On the other hand, the weight W becomes higher as the individual recognition result has a lower degree of congestion in the head projection area. This means that the influence of the crowd is small and the reliability of the individual recognition result is high. Such a difference in weight W occurs because the positional relationship between the object to be recognized and the crowd behind it on the captured image differs depending on the positional relationship with the photographing means. Therefore, by determining the weight W of each photographing means according to the degree of congestion in the area of the object to be recognized, the reliability of the individual recognition result that changes depending on the positional relationship between the photographing means and the crowd can be evaluated.

The object position determining means 533, which is an integrated recognition means, integrates the individual recognition results for each photographing means based on the weighting. In other words, the object position determining means 533 obtains the object position of the moving object at the current time based on a plurality of candidate positions in each moving object.

In the present embodiment, the object position determining means 533 integrates the individual likelihood of each candidate position of the moving object for each photographing means based on the weight W in the XYZ coordinate system, and further integrates the integrated likelihood. Is used as the weight U, and the candidate positions are weighted and averaged to calculate the object position of the moving object. The calculated object position in the XYZ coordinate system is associated with the moving object and stored in the object information 521 of the storage unit 52.

The object position determining means 533 updates the object position, hypothesis and template of the object being tracked, determines the existence of the new object, registers the object information about the new object, and the disappeared object. Perform processing. Hereinafter, the processing for the object being tracked, the processing for the new object, and the processing for the disappearing object will be sequentially described.

[Moving object being tracked]
For an object whose object position has been determined by the object position determining means 533, the determined object position is additionally stored, and a shape model is placed at each object position at the current time and projected onto each captured image to produce a whole body projection area. The image features of the object are extracted, and the template for each photographing means of the object is updated with the image features at the current time. The update may replace the extracted image features with the stored image features, or the extracted image features and the stored image features may be weighted and averaged.

[New object]
The object position determining means 533 detects the background subtraction region by performing subtraction processing between the background image captured when the object (person) to be tracked does not exist in the monitoring space and each captured image, and also detects the object at the current time. A shape model is placed at each position and projected onto each captured image to extract a background subtraction region that does not overlap with any of the whole body projection regions. Then, the object position determining means 533 determines that a new object exists in the non-overlapping background subtraction region if the non-overlapping background subtraction region has an effective area TS as the object to be tracked. When it is determined that a new object exists, the three-dimensional position is estimated for the non-overlapping background subtraction region by the same method as in the first embodiment, and the object position in the XYZ coordinate system is derived. Further, the template of the object and the object position of the object are stored in the object information 521 of the storage unit 52 in association with the object ID. Further, the object position determining means 533 stores the captured image when the object to be tracked does not exist as a background image in the storage unit 4, and updates the background image with the captured image in the region where the background subtraction region is not detected.

[Disappearing object]
The object position determining means 533 determines that an object whose individual likelihood L is equal to or less than the threshold value TL, such as when the object is concealed by a shield or moved out of the captured image, is regarded as a disappearing object without an object position. , Delete the object information of the object.

The tracking result output means 534 generates, for example, a movement locus image in which the time-series object positions of each object being tracked are plotted in the XYZ coordinate system and projected onto the xy coordinate system of the photographing means 10a, 10b, 10c. Further, a color corresponding to the degree of congestion is determined in advance, and a congestion degree image is generated in which the pixel values of the colors corresponding to the degree of congestion of the pixels are set in the pixels corresponding to each pixel of the degree of congestion map. An image obtained by transparently synthesizing the moving locus image of each of the photographing means 10a, 10b, 10c and the congestion degree image of each of the photographing means 10a, 10b, 10c is output to the display unit 54. Further, the captured image at the current time may be superimposed.

Next, a processing example of the three-dimensional tracking device 5 in the second embodiment will be described with reference to FIGS. 7 and 8. FIG. 7 is a diagram showing the relationship between the tracking person and the crowd and the captured image of each photographing means. As shown in FIG. 7, each of the photographing means 10a, 10b, and 10c photographs the tracking person 600 and the crowd 610 existing in the common field of view as captured images 621, 622, 623.

In order to determine the position of the person 600 to be tracked in the three-dimensional space, a plurality of candidate positions 630 are set around the head of the person 600 in the three-dimensional space. The congestion degree estimation means 530 extracts high congestion degree regions 651,652,653 from the captured images 621, 622, 623. The tracked person 641, 643 does not overlap the high congestion area 651, 653 on the captured images 621 and 623 of the photographing means 10a and 10c, but the tracked person 642 on the captured image 622 of the photographing means 10b. Overlaps the high congestion region 652. Therefore, the weight W of the candidate positions with respect to the photographing means 10a and 10c becomes large, but the weight W of the candidate positions with respect to the photographing means 10b becomes small.

FIG. 8A is a diagram showing how the weight W of the photographing means 10b is determined for one of the candidate positions set for the person being tracked. As shown in FIG. 8A, the tracking person 600 and the crowd 610 in the three-dimensional space are photographed by the photographing means 10b. Assuming that the candidate position 700 is set for the person 600, a head projection region 720 centered on the head at the corresponding position 710 in the captured image 622 of the photographing means 10b can be obtained. Also, the position of the crowd 610 is set as the high congestion area 652. The weight W is determined according to the non-overlapping rate between the head projection region 720 and the high congestion region 652 for each imaging means and each hypothesis. Since the head projection region 720 for the candidate position 710 with respect to the photographing means 10b overlaps with the high congestion degree region 652 (the non-overlapping rate is low), the weight W becomes small.

FIG. 8B is a diagram showing individual likelihoods of the photographing means 10a, 10b, and 10c before weighting. A plurality of candidate positions are set for the person 600. All of these plurality of candidate positions are evaluated for likelihood for each of the captured images of the photographing means 10a, 10b, and 10c. The quadrangle 730, the triangle 731, and the pentagon 732 represent the same candidate positions. The position of the symbol indicates the candidate position. The size of the quadrangle 730 is the size of the individual likelihood obtained using the captured image of the photographing means 10a, and the size of the triangle 731 is the size of the individual likelihood obtained using the captured image of the photographing means 10b, the pentagon 732. Indicates the magnitude of the individual likelihood obtained by using the captured image of the photographing means 10c. Since the candidate positions 730 and 732 with respect to the photographing means 10a and 10c are not affected by the high congestion degree region 652, the likelihood evaluation can be performed correctly. The upper right side of the candidate position 731 with respect to the photographing means 10b is affected by the high-congestion degree region 652, and the likelihood cannot be evaluated correctly, and the individual likelihood is high.

FIG. 8 (c) shows the weighted individual likelihood obtained by multiplying the individual likelihood of FIG. 8 (b) by the weight W based on the degree of congestion. Since the candidate positions 740 and 742 with respect to the photographing means 10a and 10b have a low degree of congestion and a large weight W, the points of the candidate positions 740 and 742 are large. Since the candidate position 741 with respect to the photographing means 10b has a high degree of congestion and a small weight W, the point of the candidate position 741 is small. Therefore, the influence of the hypothesis regarding the photographing means 10b for which the individual likelihood could not be calculated correctly due to the crowd 610 (high congestion degree region 652) becomes small. Therefore, when the object position is obtained by the weighted average based on the candidate position, the weight W, and the individual likelihood, the influence of the hypothesis regarding the photographing means 10b can be reduced, and the object position can be set with high accuracy.

[Operation example of 3D tracking device 5]
The operation of the three-dimensional tracking device 5 will be described below. FIG. 9 is an overall flow diagram of the operation of the three-dimensional tracking device 5. When the operation of the three-dimensional tracking device 5 is started, the photographing means 10a, 10b, and 10c sequentially output the captured images to the image processing unit 53. The image processing unit 53 repeats a series of processes of steps S501 to S510 each time a captured image is input (step S500).

The image processing unit 53 outputs a congestion degree map to the photographed images acquired by the photographing means 10a, 10b, and 10c by the congestion degree estimating means 530. Further, a region having a congestion degree equal to or higher than a predetermined threshold value T1 is extracted as a high congestion degree region (step S501).

The image processing unit 53 performs tracking processing on the input captured image for each person recorded in the object information 521 of the storage unit 52 to estimate the current object position (steps S502 to S508). The image processing unit 53 sequentially selects the person to be tracked recorded in the object information 521 of the storage unit 52 as the target of the tracking process, and when the tracking process is completed for all the persons to be tracked, the image processing unit 53 Proceeds the process to step S509, while continuing the tracking process if there is an unprocessed person to be tracked (step S508).

Hereinafter, the tracking process of steps S502 to S508 will be described in more detail. The image processing unit 53 functions as a candidate position setting / evaluation means 531, sets a hypothesis for each tracking person in the XYZ coordinate system, and sets the three-dimensional shape model arranged at the candidate position indicated by each hypothesis in the photographing means 10a, 10b, Projection onto the xy coordinate system of 10c (step S502). That is, the candidate position setting / evaluation means 531 predicts the current candidate position from the past tracking information and sets the candidate position in the hypothesis.

The image processing unit 53 functions as the multiplicity determination means 532, refers to the object information 521 stored in the storage unit 52 and the congestion degree information input from the congestion degree estimation means 530, and refers to the congestion degree information input from the congestion degree estimation means 530, and for each of the candidate positions, the photographing means 10a , 10b, 10c The non-overlapping degree of the high congestion area with respect to the head projection area is calculated as weights Wa, Wb, Wc, and the calculated weights Wa, Wb, Wc are added to the corresponding hypothesis to update the object information 521. (Step S503).

The image processing unit 53 functions as the candidate position setting / evaluation means 531, and for each hypothesis set in step S502, the image features of the whole body projection region in the captured images of the photographing means 10a, 10b, 10c and the template of the person concerned. The individual likelihoods La, Lb, and Lc are calculated based on the similarity of the image features (step S504). By the way, the template is also for each shooting means.

After that, the image processing unit 53 functions as the object position determination means 533, determines whether the tracking can be continued based on the individual likelihood of the hypothesis calculated in step S504 (step S505), and determines that the tracking is not possible. If so, the tracking end process is performed (step S506). As a result, the tracking of the person determined to be untraceable is completed, and the object position determining means 533 deletes the information about the person from the object information 521 of the storage unit 52. Here, it is determined that the tracking cannot be continued for the person whose individual likelihood is less than the threshold value TL. As a result, the information of the person who is no longer shown in the captured image is deleted.

When it was determined in step S505 that the tracking could be continued, the object position determining means 533 calculated the candidate position of the hypothesis group set in step S502, the weight W calculated in step S503, and the weight W calculated in step S504. The integrated likelihood is calculated based on the individual likelihood, and the object position of the tracking person is estimated based on the integrated likelihood and the candidate position (step S507).

When the above-mentioned tracking processes S502 to S507 are performed on all the persons registered in the object information 521 of the storage unit 25, the image processing unit 53 advances the process to step S509 as described above, and the object position determination means. According to 533, a person who has not yet been tracked is detected in the captured image, and if detected, the person is added as a new tracked person (step S509). When added as a new tracking person, the position of the object is obtained by the method of the first embodiment.

When the tracking of the person is completed by the above-mentioned processes S501 to S509 for the captured image input in step S500, the image processing unit 53 outputs the tracking result to the display unit 54 (step S510). For example, the image processing unit 53 causes the display device or the like of the display unit 54 to display the object positions of all persons as a tracking result.

[Modified Example of Embodiment 2]
(2-1) In the second embodiment, the weighting determining means 532 calculates the weight W using the three-dimensional shape model, but the weight W can also be calculated without using the three-dimensional shape model. For example, a relational expression for calculating a higher weight W as the degree of congestion is lower is determined in advance, the degree of congestion of the projection point on which the candidate position is projected is acquired from the degree of congestion map, and the above relational expression is applied to the acquired degree of congestion. And the weight W is calculated.

Alternatively, the degree of congestion in the vicinity region (for example, 5 × 5 pixels) centered on the projection point on which the candidate position is projected is acquired from the degree of congestion map, and the above relational expression is applied to the acquired representative value of the degree of congestion to weight the weight. Calculate W. The representative value is, for example, the maximum value, the average value, or the mode value. In this modification, the "position where the individual recognition means recognizes the object" is defined as the "projection point where the candidate position is projected" or the "neighborhood region centered on the projection point where the candidate position is projected".

(2-2) In the second embodiment, an example in which the weighting determining means 532 determines the weight W with respect to the combination of the photographing means 10a, 10b, 10c and the candidate position is shown, but the photographing means 10a is approximately used. , 10b, 10c and the combination of objects may determine the weight W. That is, the weight W is determined for a group of a plurality of candidate positions.

(2-2-1) For example, for each object, a three-dimensional shape model of the head is arranged at each of a plurality of candidate positions of the object in the XYZ coordinate system, and the arranged plurality of three-dimensional shape models are collectively photographed. Projection onto the xy coordinate system of means 10a, 10b, 10c. The projection area of the plurality of three-dimensional shape models is regarded as "the position where the individual recognition means recognizes the object". Then, the weight W for the combination of the photographing means 10a, 10b, 10c and the object is calculated based on the degree of congestion in the projection region for each object for the photographing means 10a, 10b, 10c.

(2-2-2) Further, for example, for each object, a sphere or ellipsoid as small as possible that includes a plurality of candidate positions of the object is derived in the XYZ coordinate system, and the derived sphere or ellipsoid is photographed by the photographing means 10a. , 10b, 10c xy coordinate system. Similar to the above example, the projection area for this small sphere or ellipsoid is regarded as the "position where the individual recognition means recognizes the object". Then, the weight W for the combination of the photographing means 10a, 10b, 10c and the object is calculated based on the degree of congestion in the projection region for each object for the photographing means 10a, 10b, 10c.

(2-2-3) For example, for each object, the current object position is predicted by extrapolating to the past object position of the object in the XYZ coordinate system, and the three-dimensional shape model of the head is at the predicted position. Is arranged and projected onto the xy coordinate system of the photographing means 10a, 10b, 10c. The projection area can be positioned as a region representing the projection areas of the above two examples, and the projection area for each of the photographing means is regarded as the “position where the individual recognition means recognizes the object”. Then, the weight W for the combination of the photographing means 10a, 10b, 10c and the object is calculated based on the degree of congestion in the projection region for each object for the photographing means 10a, 10b, 10c.

Similar to the modified example (2-1), in the modified examples (2-2-1) and (2-2-3), the projection point or the projection point on which the candidate position itself is projected instead of the projection area of the three-dimensional shape model or The weight W may be calculated based on the degree of congestion in the neighboring region. In these cases, the same weight W is set for the hypothesis of the same object.

(2-3) In the second embodiment and its modifications, the weighting determining means 532 determines the weight W using only the degree of congestion, but in addition to this, the distance from the photographing means to the tracking target, The weight W can be determined by determining whether the photographing means is suitable for tracking from various factors such as the degree of concealment by another person or an obstacle.

(2-4) In the second embodiment and each modification thereof, the candidate position setting / evaluation means 531 calculates the individual likelihood of one hypothesis (that is, individual recognition) for all the photographing means. , One imaging means may be defined for each hypothesis and the individual likelihood may be calculated. In this case, the likelihoods are not integrated, and the object position determining means 533 can be configured to weight and average the candidate positions by the product of the weight W and the individual likelihoods. That is, in that configuration, the target of weighting by the weight W is a candidate position. Alternatively, it can be configured to be weighted according to the number of hypotheses. For example, the candidate position setting / evaluation means 531 predicts the object position as in the modified example (2-2-3), calculates the weight W for the combination of the photographing means and the object at the predicted position, and uses the photographing means. The number of candidate positions for the combination of objects is set according to the weight W of the combination. Assuming that there are N candidate positions per object and the weight related to the photographing means C of the object of interest is W _C , the candidate positions related to the photographing means C of the object are N × W _C / ΣW _C. Even in that configuration, the target of weighting by the weight W is a candidate position.

(2-5) In the second embodiment and each modification thereof, an example in which the object position determining means 533 detects a new object based on background subtraction processing is shown, but instead, an image of the object to be tracked is shown. A new object may be detected using a trained model in which an unspecified number of people are machine-learned (for example, images of an unspecified number of people are deep-learned). In that case, the object position determining means 533 inputs the captured image into the trained model to detect the area of the object, and the area that does not overlap with any of the shape models is a new object in the area of the object having a size equal to or larger than the threshold TS. Is determined to exist.

[Modification example common to Embodiments 1 and 2]
(3-1) In the first and second embodiments and their respective modifications, the calculation of the weight W based on the degree of congestion by the weighting determining means is simply performed based on the degree of congestion at the position of the object. , The weight W may be calculated in consideration of the degree of congestion in the region along the line-of-sight direction to the object.

In the example shown in FIG. 10A, the degree of congestion in the area 831 on the photographed image 821 of the photographing means 10a and the degree of congestion in the area 832 on the photographed image 822 of the photographing means 10b are about the same for the person 800. .. However, when viewed from the photographing means 10a, the person 800 is in front of the crowd 810 and is not concealed, whereas when viewed from the photographing means 10b, the person 800 is behind the crowd 810 and part of it is concealed. Therefore, the individual recognition result for the photographing means 10a is more reliable than the individual recognition result for the photographing means 10b.

Therefore, the weighting determining means 532 of the second embodiment has a candidate position on a straight line connecting the candidate position and the position of the photographing means in addition to the head projection area 850 in which the three-dimensional shape model of the head is arranged at the candidate position. The head projection area 851 arranged at a position closer to the photographing means and the head projection area 852 arranged at a position farther from the photographing means than the candidate position on the same straight line are further calculated, and each head projection area is calculated. Consider the degree of congestion in. In the example shown in FIG. 10B, the indexes (non-overlapping degree, off-peak degree or congestion degree) in the head projection area 851 on the side closer to the photographing means 10a and the head projection area 852 on the far side are calculated.

In the case of the weighting determination means 132 of the first embodiment, this is performed approximately. For example, in the case of a wide-angle camera in which the shooting means is installed from a bird's-eye view, the person area is shifted to the bottom of the screen to be a person area closer to the shooting means than the candidate position, and the person area is shifted to the screen to be closer to the candidate position than the candidate position. It is a person area far from the shooting means. Further, for example, in the case of a fisheye camera in which the photographing means is installed from a bird's-eye view, the person area is shifted in the direction closer to the center on the radiation from the center of the screen to be a person area closer to the photographing means than the candidate position. The person area is shifted in the direction away from the center on the same radiation to obtain the person area at a position farther from the photographing means than the candidate position.

The amount of shift may be adjusted according to the mounting position and angle of the photographing means, and may be set to an amount that overlaps with, for example, about half of the original area. Then, the weighting determining means 132, 532 obtains the average value of the index at the candidate position, the index at the position close to the photographing means, and the index at the position far from the photographing means, and calculates the weight W according to the average value. decide. At this time, it is preferable that the index at a position close to the photographing means is a weighted average value that is weighted larger than the index at a position far from the photographing means.

(3-2) Although the example in which the congestion degree estimation means 130 and 530 use an estimator that outputs continuous values is shown, an estimator that outputs a discrete degree of congestion can also be used.

For example, the estimator is modeled with a multi-class SVM (Support Vector Machine), and 4 of "background (unmanned)", "low congestion", "medium congestion", and "high congestion" depending on the degree of congestion. The model is trained using the training images classified into classes and labeled. Then, the congestion degree estimation means 130 and 530 set a window centered on each pixel of the captured image, input the feature amount of the image in the window into the estimator, and identify the class of each pixel. When the above-mentioned non-overlapping degree is used, the congestion degree estimation means 130 and 530 use the group of pixels of the “high degree of congestion” label as the high degree of congestion area, and when the above-mentioned low degree of use is used, each label is set as the degree of congestion. The congestion degree map of discrete values is obtained by replacing with a predetermined numerical value accordingly.

In addition to the multi-class SVM, the estimator can also be realized by various multi-class discriminators learned by the decision tree type random forest method, the multi-class AdaBoost method, the multi-class logistic regression method, or the like. Alternatively, an estimator can be realized by an identification type CNN (in the case of CNN, window scanning is unnecessary). Further, even when a class-classified learning image is used, it is possible to realize an estimator that outputs a continuous value of the congestion degree by using a regression type model that returns the congestion degree from the feature amount. In that case, the parameters of the regression function for obtaining the degree of congestion from the features are learned by the ridge regression method, the support vector regression method, the random forest method of the regression tree type, or the Gaussian process regression. Alternatively, it can be an estimator using a regression type CNN (in the case of CNN, window scanning is unnecessary).

(3-3) The present invention can be applied to objects other than humans, such as vehicles and animals, which can be in a congested state.

1 ... Three-dimensional position estimation device (object recognition device), 10a, 10b, 10c, 50a, 50b, 50c ... Imaging means, 11,51 ... Communication unit, 12,52 ... Storage unit, 13,53 ... Image processing unit, 14,54 ... Display unit, 120,520 ... Camera parameters, 130, 530 ... Congestion degree estimation means, 131 ... Two-dimensional position estimation means (individual recognition means), 132, 532 ... Overload determination means, 133 ... Three-dimensional position Estimating means (integrated recognition means), 134 ... Estimating result output means, 5 ... 3D tracking device, 521 ... Object information, 331 ... Candidate position setting / evaluation means (individual recognition means), 533 ... Object position determination means (integrated recognition) Means) 534 ... Tracking result output means

Claims (5)

An object recognition device that recognizes an object based on captured images taken by a plurality of photographing means having a common field of view.
For each of the photographing means, a congestion degree estimating means for estimating the congestion degree of the object photographed in the photographed image, and
An individual recognition means that analyzes the photographed image for each photographing means and recognizes all or a part of the object on the photographed image to generate an individual recognition result.
A weighting determination means that determines the weighting of each imaging means according to the degree of congestion at a position where the individual recognition means recognizes the object on the captured image captured by the imaging means.
Based on the weighting, the integrated recognition means that integrates the individual recognition results for each of the photographing means and the integrated recognition means.
An object recognition device characterized by being equipped with. The congestion degree estimating means inputs the photographed image into an estimator learned in advance to output the congestion degree at an arbitrary position in the photographed image when the photographed image is input, and arbitrarily in the photographed image. Estimate the degree of congestion at the position of
The object recognition device according to claim 1, wherein the weighting determining means determines the weighting of the photographing means according to the degree of congestion for each region of the photographed image. The individual recognition means analyzes the photographed image for each of the photographing means and obtains the position information of the object on the photographed image at the current time.
The object recognition device according to claim 1 or 2, wherein the integrated recognition means integrates the position information for each imaging means based on the weighting to determine the position of the object at the current time. It is an object recognition method by an object recognition device that recognizes an object based on captured images taken by a plurality of photographing means having a common field of view.
The congestion degree estimating means estimates the congestion degree of the object photographed in the photographed image for each of the photographing means.
The individual recognition means analyzes the photographed image for each photographing means, recognizes all or a part of the object on the photographed image, and generates an individual recognition result.
The weighting determining means determines the weighting of the respective photographing means according to the degree of congestion at the position where the individual recognition means recognizes the object on the photographed image photographed by the respective photographing means.
An object recognition method, characterized in that the integrated recognition means integrates the individual recognition results for each of the photographing means based on the weighting. An object recognition program executed in an object recognition device that recognizes an object based on captured images taken by a plurality of photographing means having a common field of view.
The congestion degree estimating means estimates the congestion degree of the object photographed in the photographed image for each photographing means.
A process in which the individual recognition means analyzes the captured image for each imaging means, recognizes all or a part of the object on the captured image, and generates an individual recognition result.
The weighting determination means determines the weighting of each imaging means according to the degree of congestion at the position where the individual recognition means recognizes the object on the captured image captured by the imaging means.
A process in which the integrated recognition means integrates the individual recognition results for each of the photographing means based on the weighting.
An object recognition program characterized by executing.

Images