JP2022112831A - Face tracking apparatus and program - Google Patents

Abstract

To provide a face tracking apparatus which improves accuracy by merging a tracking processing result with a face recognition processing result.SOLUTION: A video analysis processing unit outputs information on motion prediction result of a face region by executing processing of motion prediction of the face region included in a video. A face recognition processing unit extracts a feature quantity of the face region included in the video, and executes face recognition processing of the face region on the basis of the feature quantity, to output first position information which is position information of the face region and a first pair of person identification information, which is information for identifying a person as a result of face recognition processing, and a recognition score corresponding to the person identification information. A tracking processing unit determines a track (tracking information) in a time direction of the face region. The track has second position information which is position information of the face region, and a second pair of person identification information of the face region and recognition score.SELECTED DRAWING: Figure 1

Description

The present invention relates to a face tracking device and program.

Face recognition technology for identifying a person by recognizing a face in an image has become highly accurate due to recent advances in AI (artificial intelligence) technology. Face recognition technology has reached the stage of practical use in the security field.

Patent Literature 1 describes a face recognition technique. According to Patent Document 1, face recognition technology collects face images for learning, calculates facial feature amounts for recognizing differences between people based on the images, and based on the similarity of the face feature amounts The basic idea is to recognize faces.

Efforts to efficiently add metadata such as personal identification information to videos by applying face recognition technology for images as described above to videos are spreading, mainly among broadcasters and the like. When image processing is extended to temporally connected video processing, object region tracking processing is useful.

Patent Literature 2 describes a procedure for using the result of tracking processing for face recognition.

Japanese Patent Laid-Open No. 2002-200001 employs a tracking process that utilizes template matching with an image that has been prepared in advance.

Various proposals have been made so far for a general object tracking technique in a video that does not specify the target as a human face. Representative examples of general tracking processing are described in Patent Document 4 and Patent Document 5.

Patent Literature 4 describes processing for tracking an object by clarifying the correspondence between regions according to the similarity of feature amounts of object regions detected in different frames.

Patent Literature 5 describes a method of performing statistical processing called particle filtering. The method of tracking processing described in Patent Document 5 attempts to improve the reliability of tracking by weighting the correspondence between regions using a particle filter.

In recent years, even in the field of tracking processing technology, attempts have been made to introduce processing utilizing deep learning processing. Tracking accuracy is greatly improved by utilizing deep learning processing.

Non-Patent Document 1 describes many methods of tracking processing using deep learning processing. Non-Patent Document 2, which is also cited in Non-Patent Document 1, also describes tracking processing using deep learning processing.

JP 2017-033372 A JP 2014-191479 A JP 2014-021896 A International publication WO2018/163243 A1 JP 2019-153112 A

Gioele Ciaparrone, Francisco Luque Sanchez, Siham Tabik, Luigi Troiano, Roberto Tagliaferri, Francisco Herrera, Deep Learning in video multi-object tracking: A survey, ArXiv:1907.12740, 2019, January 2021 Daily Search], Internet <URL: https://arxiv.org/pdf/1907.12740.pdf> Nicolai Wojke, Alex Bewley, Dietrich Paulus, Simple Online and Realtime Tracking with a Deep Association Metric, arXiv:1703.07402, 2017, [Searched January 1, 2021], Internet <URL: https: //arxiv.org/pdf/1703.07402.pdf>

However, there are problems with each background art.

The tracking processing described in Patent Documents 2 and 3 is only ancillary processing of face recognition processing, and there is no semantic connection between recognition results and tracking. The techniques disclosed in Patent Documents 2 and 3 have a problem that it is not possible to easily identify a person in a video based on the output of face recognition processing.

The tracking techniques described in Non-Patent Document 1 and Non-Patent Document 2 only obtain track information of the same object area. Non-Patent Document 1 and Non-Patent Document 2 do not describe fusion processing of advanced semantic information between recognition results such as face recognition and tracking results.

In addition, the conventional technology also has a problem that it cannot cope with blurred areas in images and scene changes. The present invention aims to realize consistent face recognition in a section in which a person appears in an image.

The present invention has been made in consideration of the above circumstances, and by combining the result of tracking processing with the result of face recognition processing, it is possible to provide track information with highly semantic information such as a person's name. It is an object of the present invention to provide a face tracking device and program capable of

[1] In order to solve the above problems, a face tracking device according to one aspect of the present invention performs motion prediction processing for a face region included in an image, and outputs information on the result of motion prediction for the face region. A video analysis processing unit extracts a feature amount of the face area included in the video and performs face recognition processing of the face area based on the feature amount, thereby obtaining first position information of the face area. a face recognition processing unit that outputs position information and a first set that is a set of person identification information that is information that identifies a person as a result of face recognition processing and a recognition score that corresponds to the person identification information; a tracking processing unit that obtains a track in the time direction of the face region, the track being a set of second position information that is position information of the face region, person identification information of the face region, and a recognition score. and a second set, wherein the tracking processing unit obtains the second position information after the face region has moved based on the information of the motion prediction result of the face region passed from the video analysis processing unit. based on the first set passed from the face recognition processing unit and the second set held by the track, and also based on the relationship between the first positional information and the second positional information. Then, by updating the track, the new second position information and the new second set are given to the track.

[2] In one aspect of the present invention, in the face tracking device described above, the face tracking device performs the processing by the video analysis processing unit more frequently than the processing by the face recognition processing unit, The analysis processing unit accumulates the information of the motion prediction result which is the result of the motion prediction processing a plurality of times, and the tracking processing unit receives the information of the accumulated motion prediction result passed from the video analysis processing unit. It is said that processing is performed based on

[3] In one aspect of the present invention, in the face tracking device described above, the video analysis processing unit further detects a scene change in the video, and the tracking processing unit detects the scene change. If so, it resets the track to its initial state.

[4] In one aspect of the present invention, in the face tracking device described above, the video analysis processing unit further determines whether the face region is a blurred region, and the tracking processing unit Regarding the face area which is the blurred area, the value of the recognition score is set to the lowest value (worst score value) when the track of the face area is updated.

[5] In one aspect of the present invention, in the face tracking device described above, the face recognition processing unit associates the feature amount when the face area is a blurred area with the feature amount of the blurred area. The video analysis processing unit preliminarily enables access to information associated with the special person identification information obtained from the above, and the video analysis processing unit performs the face recognition processing on the face region by the face recognition processing unit. When the special person identification information associated with the feature quantity of the blurred area is identified, the face area is determined to be the blurred area.

[6] In one aspect of the present invention, in the face tracking device described above, the tracking processing unit updates the track based on the degree of overlap between the first position information and the second position information. That's what it means.

[7] Further, in one aspect of the present invention, in the face tracking device described above, data of the set of tracks is obtained in a form in which information of a set of the person identification information and the recognition score is added to the information of the tracks. An output shaping processing unit for shaping and outputting is further provided.

[8] Further, in one aspect of the present invention, in the face tracking device of [7] above, the person identification information, the person name notation character string associated with the person identification information, and the character string associated with the person identification information a specified thumbnail image, and obtaining a search key, and based on the obtained search key, searching the data of the set of tracks output by the output shaping processing unit, thereby identifying a specific person. A search process for identifying the track associated with the information.

[9] Further, one aspect of the present invention includes: a video analysis processing unit that performs motion prediction processing of a face region included in a video and outputs information on the motion prediction result of the face region; and extracting a feature amount of the face area from the face area, and performing face recognition processing on the face area based on the feature amount, thereby obtaining first position information, which is position information of the face area, and as a result of the face recognition process, a face recognition processing unit that outputs a first set of person identification information that identifies a person and a recognition score corresponding to the person identification information; and a track in the time direction of the face region. a tracking processing unit, wherein the track has second position information that is position information of the face region and a second set that is a set of person identification information and a recognition score of the face region; The tracking processing unit obtains the second position information after the movement of the face region based on the information of the motion prediction result of the face region passed from the video analysis processing unit, and passes the second position information from the face recognition processing unit. By updating the track based on the first set and the second set possessed by the track, and also based on the relationship between the first position information and the second position information, The program causes a computer to function as a face tracking device that provides the track with the new second position information and the new second set.

According to the present invention, in a new frame, based on both the information of the feature amount as the image of the face area and the information of the track (tracking of the face area across multiple frames) considering the position, movement, etc. , track identification and facial recognition. As a result, both improved tracking accuracy and improved face recognition accuracy can be expected.

1 is a block diagram showing a schematic functional configuration of a face tracking device according to one embodiment of the present invention; FIG. 4 is a schematic diagram showing the temporal relationship between video analysis processing and tracking processing in the face tracking device according to the same embodiment; FIG. FIG. 5 is a schematic diagram schematically showing a method of constructing a face database for blur determination processing integrated with face recognition processing in the same embodiment. It is a flowchart which shows the procedure of the tracking process by the tracking process part in the same embodiment. FIG. 10 is a schematic diagram showing an example of a time transition of a recognition score for a certain track (face area track) in the same embodiment; FIG. 4 is a schematic diagram showing an example of a data configuration obtained as a result of tracking processing by the face tracking device of the same embodiment; FIG. 10 is a schematic diagram showing an example of an information path for retrieving a video content scene in which a person appears, starting from a person's name or a thumbnail image according to the same embodiment; FIG. 8 is a schematic diagram showing an example of a GUI (graphical user interface) for searching for the video scene described above (FIG. 7) in the same embodiment; It is a block diagram showing an example of the internal configuration of the face tracking device according to the same embodiment.

An embodiment of the present invention will now be described with reference to the drawings. The face tracking device of this embodiment is capable of recognizing the face of a person that appears continuously over time, a robust face area tracking method that can handle occlusion and temporary disappearance of a person, scene changes in images, motion blur, etc. Realize technical elements such as a robust method that responds to changes in This makes it possible to give highly semantic information such as a person's name to the track information based on the result of face recognition. In addition, the face tracking device of the present embodiment outputs recognition processing result data for facilitating person search for video. This facilitates the development of application software that utilizes face recognition results. For this reason, the face tracking device of this embodiment outputs a processing result with a data structure that integrates tracking processing and face recognition processing.

In this embodiment, the face is tracked in consideration of temporal relationships based on temporally consecutive images (in other words, temporally consecutive frame images). Also, face tracking processing and face recognition processing are integrated. As a result, the face tracking device of the present embodiment uses information on face tracking (information on the position of the face region and information on motion prediction of the face region) to improve the accuracy of face recognition processing. In addition, the face tracking apparatus of the present embodiment uses information (similarity of feature amounts of face regions (distance between feature amounts)) regarding face recognition processing to improve the accuracy of face tracking.

FIG. 1 is a block diagram showing a schematic functional configuration of a face tracking device according to this embodiment. As illustrated, the face tracking device 1 includes an image acquisition unit 10, an image analysis processing unit 20, a face database 30, a face recognition processing unit 40, a tracking processing unit 50, and an output shaping processing unit 60. composed of Each of these functional units can be realized by, for example, a computer and a program. In addition, each functional unit has storage means as required. The storage means are, for example, program variables and memory allocated by program execution. Also, if necessary, non-volatile storage means such as a magnetic hard disk drive or a solid state drive (SSD) may be used. Also, at least part of the function of each functional unit may be realized as a dedicated electronic circuit instead of a program.

The face tracking device 1 receives an image, recognizes a face included in the image, and outputs data that integrates the face recognition result and image information. The data output by the face tracking device 1 has a predetermined data structure. The function of each unit that configures the face tracking device 1 is as described below.

The video acquisition unit 10 externally acquires a video to be processed (face tracking target). Here, "external" may be another device or the like, or may be a recording medium or the like for recording data. The video is a sequence of continuous frame images at a predetermined frame rate (eg, 30 frames per second, 60 frames per second, etc.).

The image analysis processing unit 20 analyzes the image acquired by the image acquisition unit 10 . Specifically, the video analysis processing unit 20 performs motion prediction processing (extraction of motion parameter values) of objects, people, etc. between frame images constituting video, scene change analysis processing, and frame image processing. For example, blur detection is performed for each area inside. The video analysis processing unit 20 supplies the result of video analysis processing to the tracking processing unit 50 .

In other words, the video analysis processing unit 20 outputs motion prediction result information (motion prediction model information) for the face region by performing motion prediction processing for the face region included in the video.

The face database 30 is a database that associates and stores a person ID, which is information for identifying a person, and a feature amount of a face image. The database itself is implemented using existing technology. The database can use, for example, a magnetic disk device, a semiconductor storage device, or the like as storage means. The feature amount registered in the face database 30 is information obtained by machine learning processing or the like based on a large number of collected face images. In other words, the face database 30 stores information of face recognition models.

Note that the face database 30 may store attribute information of the person (information such as genre such as "politician" and "actor") in addition to the person ID.

The face recognition processing unit 40 detects a face area portion in the image acquired by the image acquisition unit 10, and collates the feature amount calculated from the face area portion with the feature amount stored in the face database 30. to output the result of face recognition. The face recognition processing unit 40 outputs a person ID, which is information identifying a person, as a result of face recognition processing. Alternatively, the face recognition processing section 40 may output a score for each person ID corresponding to a specific face area in the video. This score represents the likelihood that the face included in the face area is the face of the person identified by the person ID. Note that the face recognition processing itself can be performed using existing technology.

In other words, the face recognition processing unit 40 extracts the feature amount of the face area included in the video, and performs face recognition processing of the face area based on the feature amount to obtain the position information of the face area (first position and a set (referred to as a first set) of person identification information, which is information for identifying a person, and a recognition score corresponding to the person identification information as a result of face recognition processing.

The tracking processing unit 50 uses the video analysis processing result from the video analysis processing unit 20 to track the face area in the frame image output by the face recognition processing unit 40 . As a result, the tracking processing unit 50 obtains temporally continuous face area tracks. Here, "temporally continuous" means continuous between adjacent frame images. This improves the accuracy of face recognition.

That is, the tracking processing unit 50 obtains a track in the time direction of the face area. Here, the track is data of tracking information (trajectory information). The track has position information of the face area (referred to as second position information) and a set of the person identification information of the face area and the recognition score (referred to as a second set). The tracking processing unit 50 obtains second position information after the facial region has moved based on the information on the motion prediction result of the facial region passed from the video analysis processing unit 20, and calculates the second position information passed from the face recognition processing unit. and the second set that the track originally has, and also based on the relationship (degree of overlap) between the first position information and the second position information. As a result, the tracking processing unit provides the track with new second position information and a new second set (person identification information and recognition score set).

The output shaping processing unit 60 outputs the tracking result data in a form shaped into a predetermined data structure. The structure of the data output by the output shaping processing unit 60 is a structure that is easy to use from a video interaction system such as video search processing, and includes information on face recognition results and information on video. Due to the data structure output by the output shaping processing unit 60, a function (application software, etc.) using the output data effectively realizes an operation such as a search using a person in the video as a key.

That is, the output shaping processing unit 60 shapes the data of the result of the tracking processing passed from the tracking processing unit 50 and outputs the data. As an example, the output shaping processing unit 60 shapes and outputs data of a set of tracks in a form in which information of a set of person identification information (person ID) and recognition score is added to track information. There may be a plurality of sets of personal identification information and recognition scores.

The face tracking device 1 produces effects that could not be achieved with the conventional technology through the processes of the video analysis processing unit 20, the tracking processing unit 50, and the output shaping processing unit 60. FIG. Next, more detailed processing contents of each of the video analysis processing unit 20, the tracking processing unit 50, and the output shaping processing unit 60 will be described.

The video analysis processing unit 20 performs processing for detecting scene changes in video, predicting global motion between frame images and local motion for each region, and determining whether or not a region is blurred. The results of these video analysis processes are supplied to the tracking processing section 50 and the output shaping processing section 60 together with the results of the face recognition processing section 40, and are utilized to improve the accuracy of face recognition.

The scene change detection performed by the video analysis processing unit 20 is a process of detecting a break where the content shown in the video changes significantly. When a scene change occurs, it is necessary to reset the face tracking processing in the video performed by the tracking processing unit 50 . To detect a scene change, the video analysis processing unit 20 calculates the degree of correlation between successive frame images. In a situation where there is no scene change, the degree of correlation between successive frame images maintains a level equal to or higher than a predetermined threshold. When a scene change occurs, the degree of correlation between successive frame images is significantly below the threshold. The video analysis processing unit 20 determines whether a scene change has occurred based on whether the degree of correlation is below the threshold. This scene change detection itself can be performed using existing technology. The video analysis processing unit 20 outputs information indicating whether or not a scene change occurred between the frame image at time (t−1) and the frame image at time t (where t is an integer value). The video analysis processing unit 20 notifies the tracking processing unit 50 of the scene change detection result by, for example, outputting the value of the variable is_sc_change. is_sc_change=1 represents "there is a scene change". is_sc_change=0 represents "no scene change".

The motion prediction processing performed by the video analysis processing unit 20 predicts the degree of motion of an object or person appearing in the image between the preceding and succeeding frame images. The video analysis processing unit 20 passes the predicted value of the degree of motion to the tracking processing unit 50 . As a result, the tracking processing unit 50 can perform motion compensation processing according to the prediction value passed from the video analysis processing unit 20, thereby improving the accuracy of tracking processing.

Note that the video analysis processing unit 20, as the motion prediction process, performs global motion (panning of a shooting camera, etc.), which is the motion of the entire screen, and local motion, which is the motion specific to the detected face area. Predict two types of motion. As global motion prediction, the video analysis processing unit 20 calculates parameters that model the global motion of the entire screen based on the degree of correlation between frame images. An affine model is a typical example of a global motion model. The affine model itself is a model that can be processed by existing technology. Parameters for global motion are represented by the matrix globalM. Motion compensation using this matrix globalM can be calculated using equation (1) below.

p(t)=glabalM・p(t−1) (1)

In this equation (1), p(t-1) is a vector of two-dimensional coordinates (horizontal and vertical coordinates within the frame image) representing the position on the frame image at time (t-1). . Similarly, p(t) is a vector of two-dimensional coordinates representing the position on the frame image at time t. The matrix globalM is a 2-row, 2-column matrix representing the global motion from time (t−1) to time t.

On the other hand, as prediction of local motion, the video analysis processing unit 20 predicts the motion of the face within the target face region. For local motion estimation, the video analytics processor 20 uses a Kalman filter framework. The processing procedure for local motion prediction is also disclosed in Non-Patent Document 2 mentioned above, for example. Models for local motion prediction differ from region to region. Denote the local motion model of region r as kf(r). Motion compensation for region r from time (t-1) to time t using the Kalman filter prediction function kf(r).pred can be calculated using equation (2) below.

r(t)=kf(r).pred(r(t-1)) (2)

In this embodiment, the video analysis processing unit 20 combines the above global motion prediction and local motion prediction to perform more precise motion prediction than the conventional method. Motion estimation from time (t−1) to time (t), which is a combination of global motion estimation and local motion estimation, is expressed by Equation (3) below.

r(t)=integrateM(r(t-1))=kf(r).pred(globalM.r(t-1))
... (3)

In Equation (3), integrateM represents a motion prediction model that integrates global motion prediction and local motion prediction.

In order to make the overall processing load of the face tracking device 1 appropriate, face recognition processing (processing of the face recognition processing unit 40) that takes a relatively long processing time may be performed at a low rate. In other words, the face tracking device 1 can perform motion prediction processing with relatively high frequency and perform face recognition processing with relatively low frequency. If the frequency of motion prediction processing is performed at a low rate in accordance with the frequency of face recognition processing, there is a possibility that the precision of motion prediction will be low. However, as described above, the face tracking device 1 of the present embodiment can perform motion prediction processing at a relatively high rate while performing face recognition processing at a relatively low rate. As a result, it is possible to maintain high motion prediction accuracy while reducing the load on the apparatus for face recognition processing. In other words, the face tracking device 1 of the present embodiment can maintain motion prediction accuracy while suppressing processing costs by making the rate of face recognition processing different from the rate of motion prediction processing.

In other words, the face tracking device 1 performs the image analysis processing by the image analysis processing unit 20 more frequently than the face recognition processing by the face recognition processing unit 40 . Then, the video analysis processing unit 20 accumulates motion prediction result information (amount of motion possessed by the motion prediction model) that is the result of motion prediction processing performed a plurality of times. Then, the tracking processing unit 50 performs processing based on the information on the accumulated motion prediction result passed from the video analysis processing unit 20 .

Note that the rate of tracking processing (processing by the tracking processing unit 50) and processing for data output (processing by the output shaping processing unit 60) is matched with the rate of face recognition processing.

FIG. 2 is a schematic diagram showing the temporal relationship between video analysis processing and tracking processing in the face tracking device 1. As shown in FIG. As shown, the video to be processed by the face tracking device 1 is a sequence of frame images. A video is, for example, a sequence of frame images at a rate such as 30 FPS or 60 FPS. Note that "FPS" is an abbreviation for "frames per second". The video analysis processing unit 20 performs video analysis processing on each of these frame images, and outputs video analysis results. On the other hand, the tracking processing unit 50 performs face tracking processing on a predetermined number of consecutive frame images that are appropriately determined. When the tracking processing unit 50 performs tracking processing, the video analysis result output by the video analysis processing unit 20 is used.

The video analysis processing in FIG. 2 includes the aforementioned motion prediction processing. Further, the tracking process is executed using the results of the face recognition process with the same frequency as the face recognition process. When processing is performed in the temporal relationship shown in FIG. 2, multiple motion prediction processes correspond to one face recognition process and tracking process. At this time, the face tracking device 1 accumulates motion parameters obtained by motion prediction processing by the video analysis processing unit 20 .

That is, the video analysis processing unit 20 obtains and outputs the video analysis result at time t based on the frame image at time (t−1) and the frame image at time t. Thereafter, similarly, the video analysis processing unit 20 obtains video analysis results based on two temporally adjacent frame images. That is, the video analysis processing unit 20 outputs the motion prediction parameter M1 (global motion prediction parameter) as a result of the first video analysis processing. Thereafter, similarly, the video analysis processing unit 20 outputs a motion prediction parameter Mk (global motion prediction parameter) as a result of the k-th (k is a positive integer) video analysis processing. The video analysis processing unit 20 accumulates the motion prediction parameters Mk (k=1, 2, . . . ) for each time and calculates a global motion prediction parameter GlobalM for tracking processing. The video analysis processing unit 20 also accumulates a local motion prediction model for each face region r, and calculates local motion prediction parameters for tracking processing. The video analysis processing unit 20 passes these accumulated motion prediction parameters to the tracking processing unit 50 at the timing of tracking processing.

Similarly, the video analysis processing unit 20 can also perform scene change detection processing at a high frequency. In this case, is_sc_change_1 (value is 0 (no scene change) or 1 (scene change) as described above) is obtained as a result of the first video analysis processing. Similarly, the video analysis processing unit 20 obtains is_sc_change_k as the k-th (k is a positive integer) video analysis processing result. Then, the video analysis processing unit 20 obtains is_sc_change for tracking processing based on the scene change determination result is_sc_change_k (k=1, 2, . . . ) as a result of each video analysis processing. Note that if is_sc_change_k=1 in any one of k=1, 2, . . . , the video analysis processing unit 20 sets is_sc_change=1. Also, if is_sc_change=0 for all k=1, 2, . . . , the video analysis processing unit 20 sets is_sc_change=0. The video analysis processing unit 20 passes the value of is_sc_change, which is the accumulated scene change determination result, to the tracking processing unit 50 at the timing of tracking processing. Note that when is_sc_change=1, the tracking processing unit 50 resets the motion prediction parameters. Once a scene change is detected and the motion prediction parameters are reset, the video analysis processing unit 20 suspends all motion prediction processing until the timing of the next face recognition processing (and tracking processing). good too.

The blur determination performed by the image analysis processing unit 20 is to determine whether or not each face area in the image to be processed is a blur area. Note that the image analysis processing unit 20 detects the face area even when the face area is blurred. If face recognition processing is performed based on an image in a blurred area, the recognition accuracy will be degraded. That is, even if face recognition processing is performed on a blurred face image, recognition accuracy is low, and there is a high possibility that an incorrect person will be recognized. This leads to deterioration of the face recognition accuracy of the face tracking device 1 as a whole. In order to avoid such an accuracy deterioration factor, the face tracking device 1 of the present embodiment performs blur determination (determines whether or not the area is blurred) for each area. No processing. However, the face tracking device 1 transfers the information of the detection result of the face region to the tracking processing unit 50 even when the face recognition processing is not performed.

The video analysis processing unit 20 determines whether or not the face area r is a blurred area, and sets the determination result as the value of the variable is_blur(r). is_blur(r)=1 indicates that the region r is determined to be a blurred image. is_blur(r)=0 indicates that the region r is determined not to be a blurred image. The image analysis processing unit 20 determines whether or not each face area is a blurred area.

As an example, the video analysis processing unit 20 determines the presence or absence of blur by analyzing edge components included in the image of the face region r. The method itself can be implemented using conventional techniques. Specifically, the video analysis processing unit 20 detects an edge component in the face region r, and the degree of sharpness of the edge component is equal to or lower than a predetermined level (the image of that portion does not have high frequency components). If so, it is determined that there is "blur" (is_blur(r)=1). In addition, the image analysis processing unit 20 determines that "no blur" (is_blur(r) = 0).

The video analysis processing unit 20 may determine a blurred image by another method. In this method, the video analysis processing unit 20 determines whether or not there is a blurred image at the same time when the face recognition processing unit 40 performs face recognition processing. That is, when using this technique, the face database 30 stores in advance the feature amount of the image extracted from a large number of blurred face images in association with the person ID of the blurred image. As a result, when the feature amount of the face area r to be subjected to face recognition processing is collated with the feature amount held in the face database 30, it is indicated that the face area r is a blurred area. That is, the determination of the presence or absence of a blurred image by the video analysis processing unit 20 and the face recognition processing by the face recognition processing unit 40 can be performed as an integrated process. That is, it is possible to reduce the amount of calculation of the face tracking device 1 as a whole.

FIG. 3 is a schematic diagram schematically showing a method of constructing a face database for blur determination processing integrated with face recognition processing. As illustrated, in this method, a large number of blurry face images are prepared. Then, the face tracking device 1 extracts feature amounts of these blurry face images. The process of extracting the feature amount of the image can be performed using the function of the face recognition processing section 40 . Then, the face tracking device 1 registers the obtained blurred face feature amount data in the face database 30 in association with the person ID representing the blurred face. Note that it is sufficient to give one person ID representing a blurred face to an unspecified number of people with blurred faces.

That is, the video analysis processing unit 20 determines whether or not the face area is a blurred area. Based on the determination result, the tracking processing unit 50 sets the value of the recognition score to the lowest value (worst score value) when updating the track of the face region, which is a blurred region. One method of blur judgment is as follows. The face recognition processing unit 40 accesses information (such as the face database 30) that associates a feature amount when the face area is a blurred area and special person ID identification information associated with the feature amount of the blurred area. Plan ahead if possible. When the face recognition processing unit 40 identifies a special person ID associated with the feature amount of the blurred area as a result of performing face recognition processing on the face area, the image analysis processing unit 20 detects that the face area is the blurred area. It can be determined that

The details of the tracking processing by the tracking processing unit 50 are as follows. The tracking processing unit 50 uses the result output by the face recognition processing unit 40 and the result output by the video analysis processing unit 20 to obtain the track of the face area of the same person from the corresponding relationship on the time axis. I do. If the track of the face region of the same person can be identified in the video by this processing of the tracking processing unit 50, even if the reliability of the face recognition processing result in the frame image at a certain time is low, the face region of the other time belonging to the same track can be identified. If a highly reliable result of face recognition processing can be obtained from a frame image, it is possible to achieve highly accurate face recognition as a result of replacement.

FIG. 4 is a flowchart showing the procedure of tracking processing by the tracking processing unit 50. As shown in FIG. Description will be made below along this flow chart. It should be noted that this flowchart shows the procedure of the tracking process at a certain point in time. That is, the tracking processing unit 50 executes the processing shown in FIG. 4 each time the tracking processing is performed.

Data on which the processing of the tracking processing unit 50 at time t is based is as follows. The tracking processing unit 50 acquires output signals (data) from the face recognition processing unit 40 and the video analysis processing unit 20 . From the face recognition processing unit 40 to the tracking processing unit 50, the detected face area (position information in the frame image), the recognition score that is the result of the face recognition process, the person ID that is the recognition result, and the face area. Features are passed. A scene change detection signal is_sc_change and a face area integrated motion prediction parameter integrateM are passed from the video analysis processing unit 20 to the tracking processing unit 50 .

Further, the tracking processing unit 50 holds a set of tracks that were tracked at the time of the previous tracking processing, that is, at time (t-1). The set of tracks at time (t-1) is given by equation (4) below. Note that n_Tr in equation (4) is the number of tracks included in the set of tracks.

Tr(t-1) = [tr[0](t-1), tr[1](t-1), …, tr[n_Tr-1](t-1)] (4)

Each track tr[k](t-1) (0≤k≤n_Tr-1) is a data structure, and the elements of the structure include the following bbox, feat, recog_ids, and recog_scores.
bbox is rectangular area information (position information within the frame image) of the face area corresponding to the track at time (t−1).
feat is the feature amount of the face area in the track at time (t-1).
recog_ids is a person ID candidate of the face recognition result corresponding to the track at time (t−1). There may be a plurality of person ID candidates. The plurality of person ID candidates are arranged in descending order of score, for example.
recog_scores is a score value corresponding to the person ID candidate in the track at time (t-1). They are sorted in the same order as the person IDs listed above (for example, in descending order of score).

The notation for accessing the elements of the structure is as follows. For example, bbox (rectangular area information) for the entire set of tracks at time (t-1) is expressed as Tr(t-1).bbox. For example, the bbox for an individual track at time (t-1) is represented as tr[k](t-1).bbox. where k is the index value of an individual track within the set of tracks, and 0≤k≤n_Tr-1.

Before the initial processing of the tracking processing unit 50 (that is, at time 0), initialization processing of the track set is performed. That is, initialization is performed with Tr(0)=[] (empty set). Note that the track set is reset to an empty set even when the track is reset due to a scene change.

The face detection area and face recognition result information at time t received by the tracking processing unit 50 from the face recognition processing unit 40 are given by the following equation (5). In Expression (5), n_Det is the number of face detection regions (that is, the number of face recognitions) in face recognition processing at time t.

Det(t) = [det[0](t), det[1](t), …, det[n_Det-1](t)] (5)

Each face detection area det[k](t) (0≦k≦n_Det−1) is a data structure, and the elements of the structure are the following bbox, feat, recog_ids, Includes recog_scores.
bbox is rectangular area information (position information within the frame image) of the face detection area at time t.
feat is the feature amount of the face detection area at time t.
recog_ids is a person ID candidate of the face recognition result corresponding to the face detection area at time t. There may be a plurality of person ID candidates. The plurality of person ID candidates are arranged in descending order of score, for example.
recog_scores is a score value corresponding to the person ID candidate in the face detection area at time t. They are sorted in the same order as the person IDs listed above (for example, in descending order of score).

The notation for accessing the elements of the structure is as follows. For example, bbox (rectangular area information) for the entire set of face detection areas at time t is expressed as Det(t).bbox. For example, the bbox for an individual face detection region at time t is represented as det[k](t-1).bbox. However, k is an index value of an individual face detection area within a set of face detection areas, and 0≦k≦n_Det−1.

The data handled by the tracking processing unit 50 further includes a matrix representing the degree of overlap between rectangular regions and a matrix representing the distance between feature quantities.

A matrix representing the degree of overlap between rectangular areas is expressed as IoU (bboxA, bboxB). IoU represents "Intersection over Union" (the ratio of the intersection of the regions to the union of the regions). The tracking processing unit 50 obtains the degree of overlap between the rectangular area of the track (bboxA above) and the rectangular area of the face detection area (bboxB above), and uses it for tracking processing. This matrix is a two-dimensional matrix with size n_bboxA×n_bboxB. However, n_bboxA is the number of regions in bboxA, and n_bboxB is the number of regions in bboxB. When the value of an element of this matrix is 1, the corresponding regions are perfectly matched. When the element value is a positive value (0 or more and 1 or less), the smaller the value, the smaller the degree of overlap between corresponding regions. If the value of the element is a negative value, the corresponding regions do not overlap at all, and the greater the absolute value of the value, the greater the distance between the two regions.

A matrix representing the distance between features is represented as Dis (FeatA, FeatB). The tracking processing unit 50 obtains the distance between the feature amount of the rectangular area of the track (FeatA described above) and the feature amount of the rectangular area of the face detection area (FeatB described above), and uses it for tracking processing. This matrix is a two-dimensional matrix with size n_FeatA×n_FeatB. However, n_FeatA is the number of feature amounts of FeatA (same as the number of rectangular areas held by the track), and n_FeatB is the number of feature amounts of FeatB (same as the number of rectangular areas held by the face detection area). The distance value is 0 or a positive value. If two features are the same, the distance between the two features is zero. The higher the degree of dissimilarity between two features, the greater the distance between the features.

The tracking processing unit 50 performs tracking processing based on the above data. In other words, the tracking process performed by the tracking processing unit 50 can be reduced to the problem of associating the track set Tr(t-1) at time (t-1) with the face detection area set Det(t) at time t. be done. The tracking processing unit 50 performs the processing shown in FIG. 4 once with the passage of time (for example, from time (t−1) to time t). We will update the data step by step.

The processing procedure at time t of the tracking processing unit 50 is as follows. First, in step S<b>11 , the tracking processing unit 50 determines whether or not there has been a scene change since the previous tracking processing, based on information passed from the video analysis processing unit 20 . Whether or not there is a scene change is indicated by the value of the variable is_sc_change. When there is a scene change, that is, when the value of is_sc_change is 1 (step S11: YES), the process jumps to step S21. When there is no scene change, that is, when the value of is_sc_change is 0 (step S11: NO), the track group at time (t-1) is set as the current track group (existTr=Tr(t-1)), and the next to step S12.

Next, in step S12, the tracking processing unit 50 performs motion correction processing. That is, the tracking processing unit 50 performs motion correction processing using the integrated motion prediction model integrateM passed from the video analysis processing unit 20 for each track of the current track set existTr.bbox.

Next, in step S13, the tracking processing unit 50 calculates the overlapping degree of the regions. Specifically, the tracking processing unit 50 expands the rectangular area obtained as a result of the motion correction processing in step S12 by sc times. This extension is to cover motion compensation errors that the motion compensation process may contain. The value of the expansion scale factor sc is set to a suitably determined value of 1.0 or more. For example, 1.5≦sc≦2.5 may be set. In this embodiment, sc=2.0 as an example. This extension increases the size of the rectangular area by sc while keeping the center position of the rectangular area unchanged. Then, the tracking processing unit calculates the degree of overlap between the extended rectangular area and Det(t).bbox passed from the face recognition processing unit 40 . Then, the tracking processing unit 50 performs threshold processing on each element of the degree-of-overlap matrix (IoU described above) with a predetermined threshold value to create a threshold-processed matrix GateMtx. That is, in the thresholded matrix GateMtx, regions with a degree of overlap below a predetermined threshold are treated as regions that do not overlap each other at all.

That is, the tracking processing unit 50 performs processing for updating the track based on the degree of overlap between the first position information and the second position information.

Next, in step S<b>14 , the tracking processing unit 50 compares the feature amount of the face area of the current track with the feature amount of the face area at time t passed from the face recognition processing unit 40 . In other words, the tracking processing unit 50 creates a matrix representing the distance between the feature amount of the current track, ExistTr.feat, and the feature amount, Det(t).feat, passed from the face recognition processing unit 40. Ask for DisMtx.

Next, in step S15, the tracking processing unit 50 performs area filtering. That is, the tracking processing unit uses GateMtx obtained in step S13 to filter out invalid components contained in the matrix DisMtx obtained in step S14. In other words, as a result of the threshold processing in step S13, when the face area of the track and the face area of the face detection area are treated as areas that do not overlap each other, the tracking processing unit 50 determines that the track and the face area are not overlapped. There is no association with the face detection area.

Next, in step S16, the tracking processing unit 50 determines the correspondence relationship between the face area track up to time (t−1) and the face detection area at time t, based on DisMtx obtained as a result of step S15. judge. In other words, the tracking processing unit 50 detects the track at time (t−1) and the face at time t based on the degree of overlap between regions (threshold processing has been performed) and the distance between the feature amounts of each region. Associate with the detection area. In addition, in this step, the tracking processing unit 50 obtains the above correspondence relationship using, for example, a Hungarian algorithm (Hungarian method). The Hungarian algorithm itself is based on existing technology. In other words, the tracking processing unit 50 attempts to associate the face area track number (tr_no) with the face detection area number (det_no). The tracking processing unit 50 branches the processing of the face area as follows according to the obtained correspondence relationship. If the face area track and the face detection area correspond (matched_pairs), the process proceeds to step S17. If there is no face area track corresponding to the face detection area (unmatched_dets), the process proceeds to step S18. If there is no face detection area corresponding to the face area track (unmatched_tracks), the process proceeds to step S19. The above matched_pairs is a set of combinations of face area track numbers and face detection area numbers (matched_pairsε{(tr_no, det_no)}). Also, the unmatched_dets is a set of face detection area numbers (unmatched_detsε{det_no}). Also, unmatched_tracks is a set of face area track numbers (unmatched_tracksε{tr_no}).

Note that the tracking processing unit 50 performs conditional branching in step S16 for each region (each matched_pair, each unmatched_det, and each unmatched_tracks). Then, the tracking processing unit 50 performs the processing of steps S17, S18, and S19 for each region classified in step S16.

Next, in step S17, the tracking processing unit 50 adds the information of the corresponding face detection area (Detection) to the corresponding existing face area track (Track).

Next, in step S18, the tracking processing unit 50 registers a new face region track (Track) based on the corresponding face detection region (unmatched detection). That is, the tracking processing unit 50 adds the face detection area to the existing track set as a new track.

Next, in step S19, the tracking processing unit 50 registers the corresponding face area track (unmatched track) as a lost track or a deleted track. Specifically, the tracking processing unit 50 puts the existing track into a lost state (lost state) when there is no face detection area corresponding to the existing track, and terminates the track when this lost state continues for a predetermined period of time. (deleted) may be excluded from the target of subsequent tracking processing.

When the processes of steps S17, S18, and S19 are completed for all areas, the tracking processing unit 50 updates Tracks in step S20. That is, the tracking processing unit 50 updates the tracks based on the correspondence determined in step S16, and generates the track set Tr(t) at time t.

Note that when the track is updated, if the area tr[i](t) is a blurred face area (is_blur=1), the tracking processing unit 50 forces the recognition score of the area to be the lowest. value (usually 0.0). That is, the tracking processing unit 50 sets tr[i](t).recog_score=0.0 for such a region.

On the other hand, when proceeding to step S21 (when there is a scene change in the determination of step S11), the tracking processing unit resets the track. That is, the track at time t is Tr(t)=[] (empty set). That is, in this case, track information up to Tr(t-1) is no longer used.

That is, the tracking processing section 50 resets the track to the initial state when a scene change is detected.

As a result of repeating the processing of the flowchart of FIG. 4 described above, the tracking processing unit 50 outputs a plurality of face area tracks having face recognition result information. The tracking processing unit 50 passes the data of the plurality of face area tracks to the output shaping processing unit 60 .

The output shaping processing unit 60 shapes the data of the result of the tracking processing passed from the tracking processing unit 50 and outputs the data. Specifically, the output shaping processing unit 60 converts the result of face recognition processing in each frame image of the track output by the tracking processing unit 50 (the track after being updated by the tracking processing in FIG. 4) into Format to output in order. Taking track tr[k] (k is an index value for a particular track) at time t as an example:

By time (t-1), the track contains the following data related to face recognition.
tr[k](t-1).recog_scores is an array of recognition scores up to time (t-1). Here, the recognition scores are arranged in ascending order (that is, in descending order).
tr[k](t-1).recog_ids is a sequence of person IDs corresponding to the above tr[k](t-1).recog_scores. That is, tr[k](t-1).recog_ids is person IDs (recognition results) arranged in descending order of recognition score up to time (t-1) of the track.

Here, it is assumed that the face detection area det[j] at time t is added to track tr[k]. Then, det[j].recog_scores and det[j].recog_ids, which are the face recognition results for the face detection area det[j], are also added to track tr[k] as information about the recognition results. At this time, in addition to the information of the recognition score and person ID (recognition result) that the track tr[k] originally had, newly added information is included, and sorting is performed in descending order of the recognition score. This sorting is performed by the output shaping processing unit 60 . This process is represented by equations (6) and (7) below.

tr[k](t).recog_scores =
sort([tr[k](t-1).recog_scores+det[j].recog_scores],topK) (6)
tr[k}(t).recog_ids =
argsort([tr[k](t-1).recog_ids+det[j].recog_ids],topK) (7)

Note that the process sort(listA, topK) shown in Expression (6) represents the process of rearranging the elements of the value list listA in the order of their values (here, in descending order) by the topK sentences. Note that the operator "+" in expression (6) is an operator for concatenating lists. In other words, Expression (6) represents sorting processing for a list formed by concatenating the recognition score list tr[k](t−1).recog_scores and the recognition score list det[j].recog_scores.

Also, the process argsort(listA, topK) shown in Expression (7) represents a process of sorting only topK sentences in listA in descending order using the corresponding recog_scores (recognition scores) as sort keys. Again, the operator "+" is an operator for concatenating lists. That is, equation (7) represents processing for sorting the person ID list tr[k](t−1).recog_ids and the person ID list det[j].recog_ids in the order corresponding to equation (6).

As described above, in the present embodiment, track update processing is repeated so that the maximum value of the recognition score for a certain person ID remains for a certain track over the time axis. As a result, the track can be adjusted based on the recognition results of the time period when the score of the face area is high, without being influenced by the recognition result of the time period when the score of the face area is low (including the time period when the face area is a blurred area). face recognition can be performed.

In other words, the face tracking device 1 retains and updates face recognition result data (person ID and recognition score) sorted in descending order of recognition score for each track. This makes it possible to obtain a stable recognition result for a person appearing in the video to be processed.

FIG. 5 is a schematic diagram showing an example of time transition of the recognition score for a certain track. As shown, this track has person IDs of person A and person B as candidates for recognition results. For that track, the score for person A is 0.5 at time t1 and 0.8 at time t2. On the other hand, the score for person B is 0.6 at time t1 and 0.6 at time t2.

Such a score is generated because, for example, the correct face recognition for this track is "person A", and at time t1, the recognition score for "person A" is 0 because the person A was looking sideways. 5, and at time t2, the recognition score increases to 0.8 because the person A was facing the front. That is, in this case, at time t1, the score of person A who should be correct is lower than the score of person B who is incorrect. Note that t1<t2 here.

In the above case, the data held by the track at time t1 are as follows.
tr(t1).recog_socres = [0.6,05]
tr(t1).recog_ids = [B,A]
At time t2, the data held by the track are as follows.
tr(t2).recog_scores = [0.8,0.5]
tr(t2).recog_ids = [A,B]

The recognition score at time t1 and the recognition score at time t2 are integrated as follows.
tr.recog_score = [0.8,0.6]
tr.recog_ids = [A,B]
In other words, the face tracking device 1 preferentially uses the state of the front face of the person A, which has a higher score, as the track recognition result over time. As a result, the face tracking device outputs the correct person A as a higher recognition result than the person B. FIG. In this way, the face tracking device 1 takes the person with the highest recognition score during the track over time as the recognition result. As a result, the face tracking device 1 can realize more reliable recognition processing.

As described above, when the blur detection result is_blur is 1, the face tracking device 1 forcibly sets the face recognition score of the track at that time to 0.0. That is, the recognition result when the face area of the track is blurred does not affect the calculation of the recognition score for the entire track over time.

As described above, the face tracking device 1 identifies the person ID of the face appearing in the video while tracking the face. As a result, the face tracking device 1 can obtain and output data of person ID candidates and scores associated with the person IDs for each track.

FIG. 6 is a schematic diagram showing an example of a data structure obtained as a result of tracking processing by the face tracking device 1. As shown in FIG. As shown, the face tracking device 1 can output a set of data, which are represented by tracks, for specific video content. The primary key of the tracks shown is, for example, the track ID. A video content to be processed can have a unique content ID. That is, one content ID is associated with multiple (many) tracks. Track attributes include start time (start frame), end time (end frame), recognition score (list), person ID (list). Each of the start time and end time is relative time in the video content. The start time corresponds to the start frame. The end time corresponds to the end frame. The accuracy of the start time and end time of each track depends on the frequency of tracking (see FIG. 2). That is, by performing the tracking process frequently enough (e.g., once every second, or once every 0.5 seconds, etc.), the accuracy of track start and end times is sufficient for practical use. . However, there is no constraint on the frequency of tracking processing. The recognition score is a list of score values corresponding to person IDs. Recognition scores are data in a list arranged in descending order of scores. The person ID is a list of persons appearing in the video. Person IDs are sorted in descending order of their corresponding recognition scores. In this way, by searching the data of the set of tracks using the person ID as a key, it is possible to obtain the scene (start time and end time) in the video in which the corresponding person appears with a predetermined accuracy.

As shown in FIG. 6, the person ID is identification information for uniquely identifying a person. A person ID is associated with one or more person name (notation) data. Also, the person ID is associated with one or more thumbnail image data. That is, it is possible to obtain the person ID from the person name notation. Also, when a thumbnail image is selected, a person ID can be obtained.

FIG. 7 is a schematic diagram showing an example of an information path for retrieving a video content scene in which a person appears, starting from a person's name or a thumbnail image. As described with reference to FIG. 6, for example, when a person's name notation is input as a character string, or when a specific thumbnail image is selected from a large number of presented thumbnail images (person's face images), the person ID is obtained. can be specified. When the person ID is obtained, by searching the data of the set of tracks shown in FIG. 6, the content ID of the video content corresponding to the person ID and its relative position (start time (start frame) and end time (end frame)) information may be specified in one or more sets. This makes it possible to present the video scene corresponding to the person ID to the user or the like.

FIG. 8 is a schematic diagram showing an example of a GUI (Graphical User Interface) for searching video scenes described in FIG. In other words, this figure shows an example of a GUI of a video retrieval system based on face recognition results. This video search system can search for video using track data obtained as a result of processing by the face tracking device 1 of the present embodiment. The illustrated screen is a display screen of a mobile terminal such as a PC, a tablet terminal, or a smart phone. In the figure, 701 is a video display area. 702 is a person name display area. 703 is a thumbnail image display area. 704 is a scene specific information display area. The user of this system selects one of the person names (character strings) displayed in the person name display area 702 or one of the thumbnail images (face images) displayed in the thumbnail image display area 703. can be clicked (or tapped). As a result, the video search system can extract tracks corresponding to the person, as shown in FIG. Then, the video search system displays the specific scene (relative position) of the specific video content in the video display area 701 based on the extracted track information. If there are multiple tracks associated with the selected person, for example, the video retrieval system may select the track with the highest score for the person based on the value of the recognition score.

It should be noted that the face tracking device 1 itself may be provided with a search processing unit to perform the image search shown in FIG. In this case, the search processing unit, from the GUI screen or the like, designates any of the person ID (person identification information), the person name description character string associated with the person ID, and the thumbnail image associated with the person ID. Get the search key for The search processing unit identifies one or more tracks associated with a specific person ID by searching the data of the set of tracks output by the output shaping processing unit 60 based on the acquired search key. The track identified by this has a content ID and relative position information within the video content. Therefore, the search processing unit can acquire a video scene specified by a track from a storage device or the like, decode it, and display it as a video on the screen (video display area 701).

FIG. 9 is a block diagram showing an example of the internal configuration of the face tracking device. The face tracking device 1 can be implemented using a computer. As shown, the computer includes a central processing unit 901 , RAM 902 , input/output ports 903 , input/output devices 904 and 905 and the like, and bus 906 . The computer itself can be implemented using existing technology. The central processing unit 901 executes instructions included in programs read from the RAM 902 or the like. The central processing unit 901 writes data to the RAM 902, reads data from the RAM 902, and performs arithmetic operations and logical operations according to each instruction. A RAM 902 stores data and programs. Each element contained in RAM 902 has an address and can be accessed using the address. Note that RAM is an abbreviation for "random access memory". The input/output port 903 is a port for the central processing unit 901 to exchange data with an external input/output device or the like. Input/output devices 904 and 905 are input/output devices. The input/output devices 904 and 905 exchange data with the central processing unit 901 via the input/output port 903 . Bus 906 is a common communication path used inside the computer. For example, central processing unit 901 reads and writes data in RAM 902 via bus 906 . Also, for example, central processing unit 901 accesses input/output ports via bus 906 .

At least part of the functions of the face tracking device 1 can be realized by a computer. In that case, a program for realizing this function may be recorded in a computer-readable recording medium, and the program recorded in this recording medium may be read into a computer system and executed. It should be noted that the "computer system" referred to here includes hardware such as an OS and peripheral devices. In addition, “computer-readable recording media” refers to portable media such as flexible discs, magneto-optical discs, ROMs, CD-ROMs, DVD-ROMs, USB memories, and storage devices such as hard disks built into computer systems. Say things. In other words, the "computer-readable recording medium" may be a non-transitory computer-readable recording medium. In addition, "computer-readable recording medium" means a medium that temporarily and dynamically retains a program, such as a communication line for transmitting a program via a network such as the Internet or a communication line such as a telephone line. , it may also include something that holds the program for a certain period of time, such as a volatile memory inside a computer system that serves as a server or client in that case. Further, the program may be for realizing part of the functions described above, or may be a program capable of realizing the functions described above in combination with a program already recorded in the computer system.

Although the embodiments have been described above, the present invention can also be implemented in the following modifications.

[First modification]
For example, the face recognition processing by the face recognition processing unit 40 may be performed using machine learning means such as a neural network. In that case, the image feature amount for each person ID held by the face database 30 is represented and held as a set of internal parameter values in the neural network. That is, information about the relationship between the image feature amount and the person ID is embedded as a parameter value of each node in the neural network.

[Second modification]
In the above embodiment, the process of tracking the face area from the past side of the image to the future side (in the forward direction) and updating the track was performed. This direction may be reversed. That is, the face tracking device 1 may track the face area from the future side of the image toward the past side (in the opposite direction) and update the track. Further, the face tracking device 1 may combine forward tracking and backward tracking. This makes it possible to specify a plurality of tracks in a moving image file from start to end, and to provide recognition results for those tracks.

As described above, according to the present embodiment (including its modifications; the same shall apply hereinafter), by combining face recognition processing of a person appearing in an image and processing for tracking a face region, It achieves both improved face recognition accuracy and improved tracking accuracy. The face tracking device 1 of the present embodiment can give consistent recognition results to regions in a video in which people appear over time.

In addition, in the present embodiment, the scene change (cut) detection result is used during the face tracking process. In other words, this embodiment initializes the tracks being tracked when a scene change is detected. As a result, even when a scene change is included in a long video, for example, appropriate tracking can be performed. In other words, it is possible to prevent processing confusion.

In addition, in the present embodiment, in order to deal with blurred images, which are factors that lower the accuracy of recognition processing, blur image detection processing is actively performed, and when a blurred image is detected, the recognition score is reduced to the lowest value. In addition, the influence of blurring on recognition is reduced.

Also, in this embodiment, the rate (frequency) of the two types of processing is changed and the processing is combined. Specifically, the video analysis processing by the video analysis processing unit 20 and the recognition and tracking processing by the face recognition processing unit 40 and the tracking processing unit 50 can be performed at different rates. As a result, the present embodiment accumulates motion predictions, which are the results of two types of motion prediction processing, and utilizes them during tracking processing. This makes it possible to execute necessary processing at a high rate (frequency) while reducing the processing load on the entire apparatus.

Further, in the present embodiment, data is output in units of loci (tracks) of the face area, and recognition result data is added to the tracks. By outputting data in such a format, the present embodiment can give the most probable recognition result (single or plural) on a track-by-track basis. Such output data is suitable for retrieving video scenes in which the person appears, using the person ID as a key. As a result, it is possible to facilitate video retrieval with a person as a starting point. Also, this embodiment presented an example of a GUI for such video search.

In other words, this embodiment makes it possible to carry out face recognition of a person appearing in a video in a consistent manner in response to changes over time. By devising a data structure that allows the recognition results to be easily used to add metadata to the target video, it becomes easier to develop a search system that can effectively search for characters.

Although the embodiments of the present invention have been described in detail above with reference to the drawings, the specific configuration is not limited to these embodiments, and designs and the like are included within the scope of the gist of the present invention.

INDUSTRIAL APPLICABILITY The present invention can be used, for example, in video content production business, video content distribution business, video content management business, video content search service business, and the like. However, the scope of application of the present invention is not limited to those exemplified here.

1 face tracking device 10 image acquisition unit 20 image analysis processing unit 30 face database 40 face recognition processing unit 50 tracking processing unit 60 output shaping processing unit 701 image display area 702 person name display area 703 thumbnail image display area 704 scene specific information display area 901 central processing unit 902 RAM
903 input/output ports 904, 905 input/output device 906 bus

Claims (9)

a video analysis processing unit that performs motion prediction processing for a face region included in a video and outputs information on a motion prediction result of the face region;
extracting a feature amount of the face area included in the video, and performing face recognition processing of the face area based on the feature amount; a face recognition processing unit that outputs a first set that is a set of person identification information that is information that identifies a person as a result of processing and a recognition score that corresponds to the person identification information;
a tracking processing unit that obtains a track in the time direction of the face area;
with
the track has second position information, which is position information of the face region, and a second set, which is a set of person identification information and a recognition score of the face region;
The tracking processing unit obtains the second position information after the movement of the face region based on the information of the motion prediction result of the face region passed from the video analysis processing unit, and passes the second position information from the face recognition processing unit. By updating the track based on the first set and the second set possessed by the track, and also based on the relationship between the first position information and the second position information, Giving the new second position information and the new second set to the track;
face tracker. The face tracking device performs processing by the video analysis processing unit more frequently than processing by the face recognition processing unit,
The video analysis processing unit accumulates information of the motion prediction result, which is the result of the motion prediction processing performed a plurality of times,
The tracking processing unit performs processing based on the accumulated information of the motion prediction result passed from the video analysis processing unit.
A face tracking device according to claim 1. The video analysis processing unit further detects a scene change in the video,
The tracking processing unit resets the track to an initial state when the scene change is detected.
3. A face tracking device according to claim 1 or 2. The image analysis processing unit further determines whether or not the face area is a blurred area,
With respect to the face region that is the blurred region, the tracking processing unit sets the value of the recognition score to be the lowest value when updating the track of the face region.
4. A face tracking device according to any one of claims 1-3. The face recognition processing unit is capable of accessing information in which the feature amount when the face area is a blurred area and the special person identification information associated with the feature amount of the blurred area can be accessed in advance. keep,
When the face recognition processing unit identifies the special person identification information associated with the feature amount of the blurred region as a result of the face recognition processing performed on the face region, the video analysis processing unit determining that the face area is the blurred area;
5. A face tracking device according to claim 4. The tracking processing unit updates the track based on the degree of overlap between the first position information and the second position information.
6. A face tracking device according to any one of claims 1-5. an output shaping processing unit for shaping and outputting the data of the set of tracks in a form in which information of the set of the person identification information and the recognition score is added to the information of the track;
7. A face tracking device according to any one of claims 1 to 6, further comprising: obtaining a search key for one of the personal identification information, a character string in which a person's name is associated with the personal identification information, and a designation of a thumbnail image associated with the personal identification information, and obtaining the obtained search key; a search processing unit that identifies the track associated with specific person identification information by searching the data of the set of tracks output by the output shaping processing unit based on
8. The face tracking device of claim 7, further comprising: a video analysis processing unit that performs motion prediction processing for a face region included in a video and outputs information on a motion prediction result of the face region;
extracting a feature amount of the face area included in the video, and performing face recognition processing of the face area based on the feature amount; a face recognition processing unit that outputs a first set that is a set of person identification information that is information that identifies a person as a result of processing and a recognition score that corresponds to the person identification information;
a tracking processing unit that obtains a track in the time direction of the face area;
with
the track has second position information, which is position information of the face region, and a second set, which is a set of person identification information and a recognition score of the face region;
The tracking processing unit obtains the second position information after the movement of the face region based on the information of the motion prediction result of the face region passed from the video analysis processing unit, and passes the second position information from the face recognition processing unit. By updating the track based on the first set and the second set possessed by the track, and also based on the relationship between the first position information and the second position information, Giving the new second position information and the new second set to the track;
A program that makes a computer act as a face-tracking device.

Images