JP7022040B2 - Object identification device, method and program - Google Patents

Description

The present invention relates to an object identification device, a method, and a program that recognize an ID unique to each object on camera images taken by a plurality of cameras having different viewpoints and identify each object based on the result of the ID recognition.

Conventionally, a technique has been proposed in which some object represented by a person is extracted and identified based on an image taken by a camera. In order to realize this identification, for example, if the object is an athlete, the uniform number and face, if it is a car, the license plate number, etc. are accurately extracted from the video, and the athlete's uniform number, etc. are extracted from the extracted part. It is necessary to correctly recognize the information of and realize the identification.

For example, if the identification of each player can be accurately realized in a sports image, the movement of each player can be accurately captured only from the image, which can be useful for tactical analysis and the like.

As a means of object identification, an identification technique using deep learning has been attracting attention in recent years because it can realize highly accurate identification. Non-Patent Document 1 discloses a technique for identifying an athlete's number with high accuracy using deep learning. Non-Patent Document 1 shows that by recognizing a certain uniform number image by a trained convolutional neural network, the correct number could be recognized with an accuracy of about 83%.

On the other hand, in order to always identify in the scene, the unique identification part such as the face and the uniform number must be frequently reflected on the camera. Therefore, there is a limit to robust identification by using only one camera.

In order to solve such technical problems, proposals have been made for an approach for efficiently identifying objects using a plurality of cameras. Patent Document 1 uses a plurality of cameras to identify an individual using images of a person captured from a plurality of directions with respect to a specific person. In Patent Document 1, high-precision identification is realized by comparing a plurality of images with registered images based on the relationship of relative orientations between images.

Japanese Unexamined Patent Publication No. 2016-001447

Sebastian Gerke; Karsten Muller; Ralf Schafer, "Soccer Jersey Number Recognition Using Convolutional Neural Networks," The IEEE International Conference on Computer Vision (ICCV) Workshops, pp. 17-24, 2015. Laurentini, A. "The visual hull concept for silhouette based image understanding.", IEEE Transactions on Pattern Analysis and Machine Intelligence, 16, 150-162 (1994). J. Redmon and A. Farhadi, "YOLO9000: Better, Faster, Stronger," 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 6517-6525 (2017). Gandhi, T and Trivedi, M. "Image based estimation of pedestrian orientation for improving path prediction." In Proc. 2008 IEEE Intelligent Vehicles Symposium, 506-511 (2008). Z. Cao, T. Simon, S. Wei and Y. Sheikh, "Realtime Multi-person 2D Pose Optimization Using Part Affinity Fields," 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1302-1310 (2017) ). JF Henriques, R. Caseiro, P. Martins and J. Batista, "High-Speed Tracking with Kernelized Correlation Filters," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 37, no. 3, pp. 583-596 ( 2015).

Non-Patent Document 1 shows that high-precision uniform number recognition can be performed by using deep learning. However, it is rare that the part to be identified is always visible in the image. For example, in the case of a sports player's uniform number, it is difficult to always capture the uniform number in the camera due to problems such as the angle at which the athlete stands with respect to the camera and the overlap between athletes. Even with the license plate of a car, there is a problem that the angle at which the license plate can be seen is limited. However, Non-Patent Document 1 does not disclose a means for performing identification with high accuracy even in such a situation.

On the other hand, since Patent Document 1 uses a plurality of cameras, it is possible to solve the problem that the above-mentioned identification target is rarely seen. However, Patent Document 1 is a technique for recognizing a person's face (head) mainly, and it is necessary that the face is clearly reflected on the camera at a level that can be identified in the face-targeted identification. ..

However, when identifying players in the entire field with a relatively small number of cameras for a wide area such as a stadium, it is necessary to shoot at an angle of view that reflects the entire stadium. However, since it is difficult to clearly project a face in such a shooting environment with the resolution of a general camera, there is a problem that it is difficult to apply it to a wide area.

In addition, face recognition is difficult to apply to competitions such as American football where a protector may be attached to the entire face or head. Further, although it is possible to use not only the face (head) but also the uniform number region for identification in Patent Document 1, the algorithm of Patent Document 1 presupposes that the feature portion used for identification can be seen from a plurality of cameras. Therefore, there is also a problem that it is difficult to effectively apply it to an identification target that is likely to be visible only from a specific camera, such as a uniform number, because it realizes efficient identification. rice field.

In addition, when capturing an object from a plurality of cameras, if the object is not projected due to being shielded by another object, there is a concern that the accuracy will be greatly reduced. However, the solution to this problem is not clearly disclosed in Patent Document 1.

An object of the present invention is to solve the above-mentioned technical problems, obtain the degree of obstruction by another object of each object for each camera, and perform ID recognition for a camera image having a small degree of obstruction for each object to identify an object. It is to provide an object recognition device, a method and a program which improve the accuracy of the object.

In order to achieve the above object, the present invention is characterized in that the object identification device, method and program for identifying an object based on a camera image are provided with the following configurations.

(1) Means for acquiring camera images of objects taken from multiple different viewpoints, means for estimating the position of each object, and the degree of shielding between objects based on the viewpoint of each camera and the position of each object for each camera. It is provided with a means for calculating the above, a means for selecting a camera to be used for identifying each object based on the degree of shielding, and a means for identifying each object based on the camera image of the selected camera for each object.

(2) The object holds an ID that can be recognized from the camera image, and each camera is further equipped with a means for estimating the direction of each object based on the camera image, and the means for selecting the camera is the direction of each object and the means for selecting the camera. A camera that recognizes the ID of each object is selected based on the degree of obstruction.

(3) The means for selecting a camera are a means for calculating the ID direction direction for each object, a means for calculating a candidate vector for each ID direction direction of each object, and a predetermined angle difference in the direction direction for each object. A means of integrating two candidate vectors below the threshold to newly generate one candidate vector and repeating this, and the newly generated one candidate reflecting the reliability of the two integrated candidate vectors. It is equipped with a means for setting the reliability of the vector, and the camera is selected based on the candidate vector whose high reliability satisfies a predetermined condition.

(4) The means for identifying the object is further equipped with a means for extracting the identification area including the ID of the object from the camera image of the object, and the ID recognition is executed for the extracted identification area.

(5) The means for estimating the orientation of each object is at least a means for estimating the orientation of each object based on the object image acquired from the camera image and a means for estimating the orientation of each object based on the movement vector of each object. I tried to include one.

(6) The means for estimating the orientation of each object is further provided with means for acquiring the reliability of each orientation estimation result.

According to the present invention, the following effects are achieved.

(1) Obtain the degree of shielding between objects for each camera, select a camera for each object that is estimated to have a high probability of object identification based on the degree of shielding of each object, and identify each object. Since the target is the camera image of the same camera, it is possible to identify objects with high accuracy by eliminating the influence of erroneous recognition due to shielding between objects.

(2) When recognizing the ID attached to an object and identifying the object based on the recognition result, the direction of the ID is determined by estimating the direction of the object, and the camera is selected based on this direction of direction. Therefore, the accuracy of ID recognition is improved, and highly accurate object identification is possible by eliminating the influence of erroneous recognition due to shielding between objects.

(3) Since the orientation estimation of each object reflects not only the estimation result based on the camera image but also the estimation result based on the movement vector, highly accurate orientation estimation becomes possible.

(4) Since the reliability is acquired for each orientation estimation result and the final orientation is estimated based on the orientation estimation result and its reliability for each object, highly accurate orientation estimation is possible. become.

(5) Since the reliability of the orientation estimation result based on the movement vector is obtained based on the movement speed of the object, the reliability of the orientation estimation result based on the movement vector can be obtained easily and accurately.

(6) When selecting a camera to perform ID recognition for each object, among the candidate vectors representing the direction of orientation of each object obtained for each camera image, the candidate vectors with a small angle difference are integrated and integrated. Since the reliability of the candidate vector is set to the new candidate vector generated by the integration and the camera is finally selected based on the highly reliable candidate vector, the influence of the outlier candidate vector on the camera selection can be eliminated. It will be like.

(7) For each object, score the recommendation level for each camera based on the orientation of each candidate vector and the orientation of each camera, and repeat this for all candidate vectors based on the cumulative score obtained. Since the camera is selected, it becomes possible to select a camera with a high probability of ID recognition.

(8) Since the identification area including the ID is extracted from the object image and ID recognition is executed for the identification area, the range of ID recognition can be narrowed in advance, and high-speed and high-precision ID recognition and thus object identification can be realized. It will be like.

It is a figure which showed the structure of the main part of the object identification apparatus which concerns on one Embodiment of this invention together with the content of the signal / information exchanged between each structure. It is a figure which showed the 3D model construction method of the object by the visual volume crossing method. It is a figure which showed the position estimation method of an object. It is a figure which showed the example of the object image extracted from the camera image. It is a figure which showed the relationship between an object image and an orientation. It is a figure which showed the calculation method of the degree of occlusion between objects. It is a figure which showed the relationship between the direction of an object and the directivity of an ID. It is a figure which showed the selection method of the candidate vector to be integrated. It is a figure which showed the method of determining the direction of a candidate vector after integration based on the score of two integrated candidate vectors. It is a figure which showed the method of selecting a camera based on a plurality of candidate vectors for which integration was completed. It is a flowchart which showed the selection procedure of the candidate vector to be integrated. It is a flowchart which showed the procedure of integrating two candidate vectors. It is a figure which showed the method of selecting a camera based on the cumulative score. It is a flowchart which showed the procedure of selecting a camera based on the cumulative score. It is a figure which showed the method of extracting the identification area from the object image. It is a figure which showed the output example of the object identification result.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a diagram showing a configuration of a main part of an object identification device according to an embodiment of the present invention together with the contents of signals / information exchanged between the configurations.

The object identification device of the present invention can be configured by mounting an application (program) that realizes each function described later on a general-purpose computer. Alternatively, a part of the application can be configured as a dedicated machine or a single-purpose machine that is made into hardware or ROM.

In the present embodiment, a person is assumed as an object, and each person object is identified based on the identification information (ID). In the present embodiment, the uniform number is assumed as the ID, but the face may be identified as the ID, and if the object is a vehicle, the license plate or the bib may be identified as the ID. Further, although the object identification is continuously performed for each camera image in frame units, the description here is limited to the processing of one frame. Well-known tracking techniques can be applied to track discrimination results between frames.

The camera image acquisition unit 1 has multiple cameras (n in this embodiment) having a clear installation position and orientation and different viewpoints (standing points). Camera images Icam1 and Icam2 from camN. … Get the IcamN.

The object position estimation unit 2 estimates the position of each object extracted from each camera image Icam. For the position estimation, the visual volume crossing method shown in Non-Patent Document 2 can be used.

As shown in FIG. 2, the visual volume crossing method is a method of generating a 3D model of an object by obtaining a product set of pyramids formed by silhouettes of the objects extracted from a plurality of camera images Icam. It is possible to estimate the position of each object from the position where the 3D model exists. At this time, if the generated 3D model has a certain size or more, the position can be estimated assuming that the object exists at that position.

In addition to the visual volume crossing method, after identifying the position of each object in the image using a deep learning-based method such as Non-Patent Document 3, which can extract a person from the image, the position in the image is determined. It is also possible to adopt a method of specifying the position of each object by projecting it on the position on the field. Alternatively, the position information may be estimated by attaching a device such as a sensor that can estimate the position to each object.

The object position estimation unit 2 shall specify the positions of all objects in space, and the estimation result of this position may be specified two-dimensionally as shown in FIG. 3, or three-dimensional coordinates. The position may be indicated as.

The object orientation estimation unit 3 includes an object image acquisition unit 301, a classification unit 302, a movement vector calculation unit 303, and a reliability acquisition unit 304, and estimates the orientation of each object for each camera image. The estimation result of the orientation of the object is used in the camera selection unit 5 in the subsequent stage to select a camera having a high possibility that an ID (in this embodiment, a uniform number) unique to each object is reflected.

The present embodiment is characterized in that the result of orientation estimation of each object is calculated for each camera image having a different viewpoint, and the object image acquisition unit 301 obtains the position information of each object obtained by the object position estimation unit 2. Based on this, the images A1 to A4 of each object are acquired as shown in FIG.

As disclosed in Non-Patent Document 4, for example, the classification unit 302 prepares a training image for each orientation of the object, and performs orientation estimation based on the characteristics thereof. In the present embodiment, as shown in FIG. 5, a training image is prepared by limiting the direction of orientation estimation to eight directions in advance, and the feature amount extracted from the acquired image of the object and the training image of each direction are used. Each object image is classified in either direction by comparing it with the feature amount.

In this embodiment, it is assumed that deep learning such as a convolutional neural network is used for orientation estimation, but as another method, HOG (Histograms of Oriented Gradients) features and learning based on the features are trained. Training and identification may be performed using SVM (Support Vector Machine) or the like.

Alternatively, as disclosed in Non-Patent Document 5, based on the position of the joint obtained by the method of detecting the skeleton, whether or not a specific joint can be seen and the convolutional neural network using the position of the joint as a feature point. Or SVM may be trained to perform orientation estimation.

The movement vector calculation unit 303 further performs orientation estimation by another approach using the movement vector in order to improve the accuracy of the orientation estimation by the deep learning. In this embodiment, for example, as disclosed in Non-Patent Document 6, a movement vector is acquired by using an algorithm for tracking objects between frames.

Once the movement vector is obtained, the orientation of the object often coincides with its movement direction. It is not always accurate because there are cases where the object moves while retreating, but the accuracy of the orientation estimation of the object can be improved by adding the result of the orientation estimation based on the movement vector.

In this embodiment, since it is assumed that there are n cameras, n orientation estimation results obtained by performing deep learning on the image of the object obtained from each camera image and tracking are successful. If this is the case, a total of n + 1 orientation estimation results, including one orientation estimation result obtained by the movement vector, will be obtained for each object.

The reliability acquisition unit 304 acquires the reliability Ri of each direction estimation result. Here, i represents an index of the orientation estimation result, and in this embodiment, i takes a value from 1 to n + 1 for each object. The reliability Ri can be calculated based on the probability of being output from the function of the output layer, for example, in the case of the orientation estimation result by the neural network.

In addition, in the direction estimation by the movement vector, in general, the faster the movement speed, the less likely it is that the player is making an unexpected movement such as turning or retreating. Therefore, Ri is based on the movement speed of the player. May be asked. For example, the faster the moving speed, the higher the reliability, and here Ri is normalized to a value of 0 to 1.

In the object obstruction degree calculation unit 4, each object whose position is estimated by the object position estimation unit 2 is shielded by another object located in front of the object before the orientation estimation or ID recognition is executed. Whether or not it is determined is performed for each camera, and finally the shielding degree Oj is calculated for each object (j is a camera identifier).

The degree of shielding Oj is also normalized to a value of 0 to 1, and is defined as a frequency indicating that the closer the value is to 1, the larger the degree of shielding, and the closer the value is to 0, the smaller the degree of shielding. If the degree of occlusion Oj is 1 of the maximum value, it means that the object of interest is completely obscured by other objects.

In the present embodiment, as shown in FIG. 6, the degree of shielding Oj is calculated based on how many other objects exist in front of the object of interest for each camera cam. The degree of obstruction Oj can be calculated by using, for example, a back projection mask of the visual volume, but if the object position estimation unit 2 calculates the visual volume of the object and estimates the position based on the visual volume, each of them. The calculation result of the visual volume of the target object can be used.

If the calculation result of the visual volume is used, as shown by hatching in FIG. 6, the existing area of another object (shielding object) that overlaps with the object of interest is defined first. The existing area is defined by the user in advance. For example, the existing width of the length L is defined on the left and right of the object of interest in the field of view of the camera, and the width 2L is defined as the base and the area of the triangle with the camera as the apex. Determine if another object exists.

Next, the mask M1 back-projected onto the camera screen from the visual volume of each object determined to exist in this existing area, and the mask M2 back-projected onto the camera screen from the visual volume of the object of interest are calculated. Then, the total area (number of pixels) Pal of the mask M2 and the number of pixels Ps in which the mask M1 overlaps the mask M2 are obtained, and the calculation result of Ps / Pall is the shielding degree Oj.

In this embodiment, although it is expressed as "shielding" for convenience, another object exists behind the object of interest instead of in front of the camera, and the other object existing behind the object of interest is the object of interest. If it is likely to affect the recognition result, the existing area may be expanded to the rear of the target object and the calculation may be performed in the same manner.

The method of calculating the degree of obstruction Oj is not limited to the above method, and when the target object is extracted based on the feature amount of the image or deep learning, the bounding box of the object is calculated and the degree of obstruction is calculated. You may. At this time, by setting the area of the bounding box of the target object to Pall and the area of the part where the bounding box of another object overlaps the bounding box of the object of interest to Ps, the shielding degree Oj can be calculated by the same procedure as above. ..

The camera selection unit 5 selects a camera to be used for object identification for each object based on the orientation of each object estimated by the object orientation estimation unit 3 and the obstruction degree Oj calculated by the object obstruction degree calculation unit 4. In this embodiment, n reliability Ris obtained for each camera by the deep learning and one reliability Ri obtained based on the movement vector will be described as having been acquired for each object.

In the camera selection unit 5, the ID direction direction calculation unit 501 calculates the direction direction of the ID based on the n + 1 direction estimation results for each object. If the ID is a uniform number, the ID-oriented direction is the direction facing the uniform number, in other words, the direction extending vertically from the back of the object.

Generally, if the orientation estimation result is 0 degrees, it can be said that the camera faces the uniform number and projects the uniform number with high probability. On the other hand, if the orientation estimation result is, for example, 90 degrees, it is unlikely that the image obtained from the camera reflects the uniform number, but as shown in FIG. 7, the direction vector obtained by the orientation estimation It is highly possible that the camera in the direction of rotating 90 degrees faces the uniform number and clearly projects the uniform number. Therefore, the direction rotated by 90 degrees is regarded as the directivity direction.

From such a viewpoint, in the present embodiment, n ID directivity directions are calculated from the n orientation estimation results obtained for each camera image. Further, in this embodiment, direction estimation using the movement vector of each object is also performed, but for this estimation result, the direction opposite to the movement direction of the object (direction rotated by 180 degrees) is used as the uniform number. The ID-oriented direction is set to face each other.

In this embodiment, since the direction of the line of sight is different for each camera, the orientation of each object estimated by the object orientation estimation unit 3 cannot be handled in a common orientation. For example, the object A1 whose orientation is estimated to be 0 ° on the image of cam1 and the object A2 whose orientation is estimated to be 0 ° on the image of cam2 do not have the same orientation on the field, and each camera cam1 and cam2 An angle difference is generated according to the difference in the line-of-sight direction.

On the other hand, since the line-of-sight direction of each camera is known in the present embodiment, in the following description, the object orientation estimation unit 3 estimates by calibrating the orientation estimation result of each object based on the line-of-sight direction of each camera. The explanation is continued assuming that the orientation and the orientation on the field match.

The candidate vector calculation unit 502 calculates n + 1 candidate vectors representing each ID-oriented direction based on the n + 1 ID-oriented directions. The camera evaluation unit 503 evaluates each camera based on the n + 1 candidate vectors for each object.

In the present embodiment, as an approach for evaluating a camera by the camera evaluation unit 503, two methods described in detail below, "a method of integrating candidate vectors" and "a method of scoring each camera". Either can be adopted.

Method A. [How to integrate candidate vectors]
When finally determining one orientation based on the n + 1 orientation estimation results obtained for each object, simply calculating the average of the n + 1 orientations will result in an estimated value as shown in FIG. When a candidate vector having a large deviation (“candidate vector by camera 4” in FIG. 8) is included, the estimation result is strongly influenced by the deviation value, and the estimation accuracy is lowered.

If a small number of such outliers appear, it is highly likely that the orientation estimation result is incorrect, and in particular, it is likely that the estimation result is from a camera that is likely to be shielded. Therefore, in the present embodiment, in order to eliminate such outliers, as described in detail below, each candidate vector is integrated under predetermined conditions, and by repeating this, one candidate vector is finally obtained. I try to do it.

9 and 10 are diagrams showing a method of integrating candidate vectors, and FIGS. 11 and 12 are flowcharts showing the procedure.

In step S1, each directivity direction is vectorized for each object and a candidate vector is calculated. In step S2, a threshold value (integration threshold value) θth for integrating candidate vectors having close directivity directions is defined. In step S3, the angle ∠ between each candidate vector (∠A to ∠E in FIG. 8) is calculated.

In step S4, the smallest angle θmin is obtained, and this minimum angle θmin is compared with the integrated threshold value θth. If the minimum angle θmin is less than the integration threshold θth, the process proceeds to step S5, and the two candidate vectors forming the minimum angle θmin are integrated to generate a new candidate vector. In the example of FIG. 9, since ∠B is the minimum angle θmin and ∠B <minimum angle θmin, the process proceeds to step S5 to integrate the “candidate vector by the camera cam2” and the “candidate vector by the camera cam3”.

FIG. 12 is a flowchart showing the procedure for integrating the candidate vectors in step S5. In step S101, the score Si is calculated according to the following equation (1) for the two candidate vectors to be integrated. Here, i is the index of the candidate vector, and j is the index of the camera used to calculate the candidate vector of the index i.

Si = Ri × (1-Oj)… (1)

Ri is the reliability of the estimation result in each direction, and Oj is the shielding degree. However, Oj may be used as a constant value for estimation results that cannot take the degree of shielding into consideration, such as the directivity constant obtained from the movement vector.

In step S102, angle division for determining the directivity direction of one candidate vector newly generated by integration is performed based on the calculation result of the score S. In the present embodiment, as shown in FIG. 9, when the score of one candidate vector to be integrated is S1 and the score of the other candidate vector is S2, the angle ∠B between these two candidate vectors is one. The angle divided by the ratio of S2: S1 from the candidate vector side of is to the other candidate vector side is the directing direction of the new integrated vector.

In FIG. 9, the score S2 of one candidate vector to be integrated (candidate vector of camera 2) is 0.4, and the score S3 of the other candidate vector (candidate vector of camera 3) is 0.6, so ∠B. Is divided from one side to the other at a ratio of 0.6: 0.4.

In step S103, the divided angle becomes the directivity direction of the new candidate vector after integration, and a new index i (here, i = 6) is attached. In step S104, the score S6 of the new candidate vector after integration is calculated as the sum of the scores of the two integrated candidate vectors (= S2 + S3).

Returning to FIG. 11, when the integration of the two candidate vectors is completed, the process returns to step S3, and each of the above processes including the new candidate vector generated by the integration has an angle below the integration threshold θth in the step S4. Repeat until it runs out. As shown in FIG. 10, when there is no angle below the integration threshold value θth, the process proceeds to step S6.

In step S6, the directivity direction of the candidate vector having the largest score S at that time is determined as the final ID directivity direction. In step S7, a camera used for recognizing the ID is selected based on the determined ID directivity direction.

In the present embodiment, one camera having an angle closest to the determined ID directivity direction may be selected, or all cameras existing within an angle range of ± φ degrees from the ID directivity direction may be selected. .. When a plurality of cameras are selected, the recognition result acquired from one camera having a high recognition likelihood is regarded as one final identification result, as will be described in detail later.

Further, even if the ID directivity direction is determined, there is not always a camera facing the directivity direction. From this point of view, it is possible to calculate how far the angle facing the direction of the ID is from the actual camera angle, and incorporate it into the likelihood when calculating the object identification unit in the subsequent stage.

B. [How to score each camera]
In the above method A, each candidate vector is scored, but in the present method B, each camera is scored. In the method B, from the viewpoint that the camera facing the direction facing the candidate vector is the most suitable camera for recognition, the scoring that maximizes the score of the camera facing the candidate vector is sequentially performed for each candidate vector.

FIG. 13 is a diagram showing a scoring method for each camera by the present method B, and FIG. 14 is a flowchart showing the procedure.

In step S21, one candidate vector of interest is selected. In step S22, the camera to be scored is selected. In step S23, the evaluation value Pi of the camera is calculated according to the following equation (2). In this embodiment, the inner product is focused on as an index for evaluating whether or not the camera is facing the camera, and a function is adopted in which the smaller the inner product value, the higher the score.

Pi = Ri × ((1-Oj) × (-cos (Φi-C))… (2)

Here, Ri is the reliability of the estimation result for each direction, and Oj is the shielding degree. Φi is the directivity direction of the candidate vector of interest, and C is the direction in which the camera is facing. The calculation of the cos part means that the inner product value is calculated (here, each vector is calculated on the assumption that it is a unit vector), and from the viewpoint that the more facing the direction is, the more desirable it is. Since the case where the inner product is -1 is the most desirable, it is converted to a positive value by adding a minus to the head of cos.

In step S24, the score Pi is added to the total score ΣPi of the camera of interest, and the total score ΣPi is updated. In step S25, it is determined whether or not the scoring for the current candidate vector is completed for all the cameras. If it is not completed, the process returns to step S22, the camera for which the score is calculated is switched, and each of the above processes is repeated.

After that, when the scoring for all the cameras with respect to the candidate vector this time is completed, the process proceeds to step S26. In step S26, it is determined whether or not the scoring for each camera is completed for all the candidate vectors. If it is not completed, the process returns to step S21, and each of the above processes is repeated while switching the candidate vector of interest.

When the scoring for each camera is completed for all the candidate vectors, the process proceeds to step S27, and the recommended camera is selected based on the total score ΣP of each camera. As the recommended camera, only one camera having the largest total score ΣP may be selected, or all cameras exceeding a predetermined threshold value may be selected. Alternatively, the top N best camera may be selected.

In the above description, the score is calculated for all the cameras for each candidate vector of interest, but the present invention is not limited to this, and the score is calculated for each candidate vector in advance. Some cameras that are expected to be expensive may be selected in advance by the internal product calculation or the like.

In that case, the above scoring is performed only for the preselected cameras, and as shown in FIG. 13, among the scores obtained for each camera, the scores obtained for the same camera are added. Finally, the camera with the maximum total score may be selected.

The object identification unit 6 includes the identification area extraction unit 601 and executes ID recognition for the identification area extracted by the identification area extraction unit 601 and identifies each object based on the recognition result of the ID.

The image to be extracted from the identification area is an image of an object projected by the camera selected by the camera selection unit 5, and for an object in which the camera selection unit 5 selects a plurality of cameras, the identification area is obtained from each camera image. Each is extracted. The identification area is a uniform number portion if the uniform number is an ID, and a license plate portion if the license plate of a car is an ID.

FIG. 15 is a diagram showing a method of extracting the identification area when the ID is a uniform number, and the uniform number portion is extracted from the whole body image used in the orientation estimation of the object.

The identification area extraction method includes a method of extracting the identification area based on the skeleton information of the person, a method of extracting a predetermined area such as the upper half of the image of the target object, and a deep layer again to extract the identification area. There are a method of extracting by performing learning and the like, and a method of extracting based on the information of the center of gravity of the silhouette created when the visual volume of the created target object is back-projected on each camera image. Here, an example of extracting an identification area based on the skeleton information of a person will be described.

Patent Document 5 discloses a technique capable of calculating a person's bone (skeleton) only from an image, and by applying this technique to a target object, the position of each part can be generally known. If it is a uniform number, the uniform number portion can be extracted with high accuracy if the position of the waist is generally known.

If there is an angle difference between the ID direction direction finally calculated by the ID direction direction calculation unit 501 of the camera selection unit 5 and the direction in which the camera is facing, this angle is used as a parameter to extract the image. An image processing such as performing affine transformation may be performed on the image of the identification area portion to add a function for improving the ID recognition accuracy.

As a method for the object identification unit 6 to perform ID recognition for the extracted identification area, there is a method of recognizing a uniform number by using machine learning as described in Non-Patent Document 1. When machine learning is adopted for uniform number recognition, it is necessary to create a model that can acquire the predicted recognition result (estimation result of what number the uniform number is) by inputting an image showing the uniform number. First, a model for uniform number recognition is generated using a training image.

It is desirable to create this model in advance. For example, a large number of training images are prepared, and a model for number recognition is created using a convolutional neural network. For the generation of the learning image, you may prepare a large number of images showing the uniform number and manually attach the correct answer label, or you can superimpose the characters in the font with numbers on any background image and artificially. A learning image may be generated. The latter method is efficient because it is possible to automatically generate a learning image with a correct answer label, so that it is not necessary to manually assign a correct answer label.

Also, if you rotate the font, distort it, and adjust the size to generate various learning images from the beginning, the spine numbers of the extracted images will be slightly diagonal or will be cut out neatly. Even if it is not, highly accurate recognition is possible.

In addition, the model generation method is not limited to the method using a convolutional neural network, and if it is possible to recognize the number, an approach such as template matching or a learner that trains by combining image features and SVM. You may take a method such as performing identification using.

Since two or more cameras are selected by the camera selection unit, two or more identification areas are extracted, and ID recognition is executed for each of them. As a result, the same recognition result may be obtained. For example, the recognition result of one camera is "38", the recognition result of the other camera is "39", and so on, there is a possibility that the recognition results may be inconsistent.

In this case, as an approach for selecting a more correct recognition result, when recognizing the uniform number, for example, if the uniform number is recognized by a convolutional neural network, the output layer of the model for uniform number recognition is activated. By using the softmax function as the function, the probability of the recognition result can be calculated.

Similarly, even with template matching and SVM, it is possible to calculate the likelihood for each recognition result. Therefore, it may be provided with a function of finally determining one ID when the results are different in a plurality of cameras based on the obtained likelihood.

In addition, when two or more ID recognition results are obtained because a plurality of cameras are selected, the degree of obstruction Oj calculated by the object obstruction degree calculation unit 4 may be reflected in the likelihood calculation. For example, the result of ID recognition from a camera that is likely to be obstructed is likely to be incorrect, so it is possible to take measures to prevent it from being adopted by lowering its likelihood according to the degree of obstruction Oj. can.

For this process, the shielding degree Oj used in the likelihood calculation in the object orientation estimation unit 3 may be used as it is, or the shielding degree Ij for the identification area extracted by the identification area extraction unit 601 may be newly calculated. You may fix it.

For example, it was extracted by obtaining the degree of overlap when the object shielding degree calculation unit 4 back-projects the visual volume only for the image area portion estimated to be the uniform number portion extracted by the identification area extraction unit 601. It is possible to calculate the degree of shielding Ij, which is how much the uniform number area is shielded.

The result output unit 7 displays the result of ID recognition in association with the position coordinates on the frame image of each object estimated by the object position estimation unit 2 and the ID of the object identified by the object identification unit 6. ..

There are various methods for displaying the result, and it is sufficient to display the position coordinates and ID of each object as numerical values on the console. However, as shown in FIG. 15, the ID of the object is linked to the position of each object. It may be attached and displayed graphically as a plane map.

In FIG. 16, a round marker indicating the position coordinates of each object (player) is arranged on a background imitating one side of a soccer field, and an ID indicating a uniform number is superimposed and displayed on each marker. ..

It is also possible to display such a plane map by outputting it for each frame of the video and moving it like a moving image. Further, at the time of this display, for example, the team to which the player belongs may be determined by acquiring the uniform color information from the image, and the color of the marker may be changed and reflected on the plane map as a result. Furthermore, an object that is judged to be a referee based on color information can be easily identified as a referee by judging that it is not a player and excluding it from the display of the result, or by not attaching an ID. You may do so.

1 ... camera image acquisition unit, 2 ... object position estimation unit, 3 ... object orientation estimation unit, 4 ... object shielding degree calculation unit, 5 ... camera selection unit, 6 ... object identification unit, 7 ... result output unit, 301 ... object Image acquisition unit, 302 ... Classification unit, 303 ... Movement vector calculation unit, 304 ... Reliability acquisition unit, 501 ... ID direction direction calculation unit, 502 ... Candidate vector calculation unit, 503 ... Camera evaluation unit, 601 ... Identification area extraction unit

Claims (15)

In an object identification device that identifies objects based on camera images
A means of acquiring camera images of objects taken from multiple different perspectives,
A means of estimating the position of each object,
A means to calculate the degree of occlusion between objects for each camera based on the viewpoint of each camera and the position of each object,
A means for selecting a camera to be used for identifying each object based on the degree of shielding for each object, and
An object identification device comprising a means for identifying each object based on a camera image of the selected camera for each object. The object holds an ID that can be recognized from the camera image,
Further equipped with a means to estimate the orientation of each object based on the camera image,
The object identification device according to claim 1, wherein the means for selecting the camera is to select a camera that recognizes the ID of each object based on the orientation and the degree of shielding of each object. The means for estimating the orientation of each object is
The second aspect of claim 2 comprises at least one of a means for estimating the orientation of each object based on an object image acquired from a camera image and a means for estimating the orientation of each object based on a movement vector of each object. Object identification device. The object identification device according to claim 3, wherein the means for estimating the orientation of each object further includes means for acquiring the reliability of the orientation estimation result. The means for estimating the orientation of each object performs deep learning-based orientation estimation for the object image, and the means for acquiring the reliability of the orientation estimation result is the output of the output layer function in the deep learning-based orientation estimation. The object identification device according to claim 4, wherein the value is acquired as a reliability. The object identification device according to claim 4 or 5, wherein the means for acquiring the reliability of the orientation estimation result obtains higher reliability as the moving speed of the object is faster in the orientation estimation based on the movement vector. .. The means for selecting the camera is
A means to calculate the ID orientation direction for each object,
A means to calculate a candidate vector for each ID-oriented direction of each object,
A means for scoring each candidate vector based on the degree of obstruction and reliability, and
For each object, a means of integrating two candidate vectors whose directivity angle difference is less than a predetermined threshold value to generate a new candidate vector and repeating this process.
A means for scoring the newly generated candidate vector based on the scores of the two integrated candidate vectors is provided.
The object identification device according to any one of claims 4 to 6, wherein a camera is selected based on a candidate vector whose score satisfies a predetermined condition. The means for selecting the camera is
A means to calculate the ID orientation direction for each object,
A means to calculate a candidate vector for each ID-oriented direction of each object,
For each object, each camera is scored based on the orientation of the candidate vector and the orientation of each camera, and this is repeated for all the candidate vectors to obtain the cumulative score of the recommendation.
The object identification device according to any one of claims 4 to 6, wherein a camera is selected based on a candidate vector whose cumulative score satisfies a predetermined condition. The object identification according to claim 8, wherein the means for obtaining the cumulative score of the recommendation degree is to score the recommendation degree for each camera based on the inner product of the orientation of the candidate vector and the orientation of each camera. Device. The means for identifying the object further includes means for extracting an identification area including the ID of the object from the camera image of the object.
The object identification device according to any one of claims 2 to 9, wherein ID recognition is executed for the extracted identification area. The object identification device according to claim 10, wherein the means for extracting the identification area is to extract skeleton information from a camera image of an object and extract the identification area based on the skeleton information. The means for calculating the degree of shielding is any one of claims 1 to 11, wherein the means for calculating the degree of shielding calculates the degree of shielding based on the ratio of other objects existing within the range connecting the predetermined width including the object of interest and the camera. The object identification device described in. The means for calculating the degree of obstruction determines the degree of obstruction based on the amount of overlap between the mask generated when the visual volume of the object of interest is projected onto the camera and the mask generated when the visual volume of another object is projected onto the camera. The object identification device according to any one of claims 1 to 12, characterized in that calculation is performed. In an object identification method in which a computer identifies an object based on camera images.
The procedure for acquiring camera images of objects taken from multiple different viewpoints,
The procedure for estimating the position of each object and
The procedure for calculating the degree of occlusion between objects based on the viewpoint of each camera and the position of each object, and the procedure for each camera.
The procedure for selecting a camera to be used for identifying each object based on the degree of shielding for each object, and
An object identification method including a procedure for identifying each object based on a camera image of the selected camera for each object. In an object identification program that identifies objects based on camera images
The procedure for acquiring camera images of objects taken from multiple different viewpoints,
The procedure for estimating the position of each object and
The procedure for calculating the degree of occlusion between objects based on the viewpoint of each camera and the position of each object, and the procedure for each camera.
The procedure for selecting a camera to be used for identifying each object based on the degree of shielding for each object, and
An object identification program that causes a computer to perform a procedure for identifying each object based on the camera image of the selected camera for each object.

Images