JP7340992B2 - Image management device and program - Google Patents

Description

The present invention relates to an image management device and a program.

It is desirable to be able to store metadata related to video content at low cost by automatically recognizing faces included in video images.

For example, claim 1 of Patent Document 1 describes a face detection device that transmits facial data to a face recognition device for recognizing a face. In the face detection device, the classification unit performs classification for each person by comparing the feature data of the face images included in the frames constituting the video image, so that the feature data of the same person is grouped into one group. ing.

JP2017-182210A

However, in order to accumulate a large amount of information on people for a wide range of video content, it is necessary to improve the accuracy of models for face recognition processing. Normally, as the number of target persons increases, it becomes difficult to correctly identify, for example, similar faces. If the accuracy of the model for face recognition processing cannot be improved, there is a problem in that the error rate in face recognition processing will increase.

The present invention was made based on the above-mentioned problem recognition, and it is possible to improve the accuracy of face recognition processing while accumulating feature amount information of facial images of multiple people, and to improve the accuracy of face recognition processing. An object of the present invention is to provide an image management device and a program that can store information on identification results.

[1] In order to solve the above problems, an image management device according to one aspect of the present invention includes a clustering unit that clusters facial images extracted from video content based on image features, and a clustering unit that clusters facial images extracted from video content based on image features, and a clustering unit that clusters facial images extracted from video content based on image features; By performing machine learning processing based on a temporary cluster storage unit that stores the image feature amount for each cluster and information representing the relationship between the face image and the cluster, the face images included in the video content are a face image learning unit for specific content that generates a face recognition model for the video content in order to determine the cluster based on the face recognition model; A face recognition processing unit that performs recognition processing and outputs cluster information as a recognition result, a person database that stores image features of face images and clusters in association with each other, and clusters output from the face recognition processing unit, A person data registration unit that registers the image feature amount for each cluster read from the temporary cluster storage unit in the person database.

[2] Further, in one aspect of the present invention, in the image management device described above, the clustering unit has a plurality of stage clustering units for performing clustering processing in a plurality of stages, and the stage clustering after the second stage is performed. The clustering section performs clustering processing on the facial images of the cluster output from the previous stage clustering section.

[3] Further, in one aspect of the present invention, the image management device described above further includes a name matching processing unit that integrates a plurality of clusters into one cluster based on the image feature amount registered in the person database. It is also something to be prepared for.

[4] Further, in one aspect of the present invention, in the image management device described above, the person database stores tags to be assigned to clusters in association with the clusters, and for the clusters to which no tags are set. , further comprising a tag setting section that performs a process of setting a new tag.

[5] Further, in one aspect of the present invention, in the above-described image management device, the face recognition processing unit detects each time interval divided by a cut point at which the amount of change in pixel value between frames of the video content reaches a peak. The facial image is tracked between frames, and based on the tracking result, an error in the cluster determination is detected, the detected error is corrected, and cluster information is output as the recognition result.

[6] Moreover, one aspect of the present invention is that in the image management device described above, the person database further stores information associating the time interval with the cluster.

[7] Further, one aspect of the present invention includes a clustering unit that clusters face images extracted from video content based on image features, and a clustering unit that clusters face images extracted from video content based on image features, and the image features for each cluster obtained by processing of the clustering unit. determining the cluster based on the face image included in the video content by performing machine learning processing based on a stored temporary cluster storage unit and information representing a relationship between the face image and the cluster; A face image learning unit for specific content that generates a face recognition model for the video content, performs recognition processing on the face image included in the video content based on the generated face recognition model, and generates a cluster of clusters as a recognition result. A face recognition processing unit that outputs information, a person database that stores clusters in association with image feature amounts of face images, and information about each cluster read from the temporary cluster storage unit for the clusters output from the face recognition processing unit. A program for causing a computer to function as an image management device, comprising: a person data registration unit that registers the image feature amount of , in the person database.

According to the present invention, it is possible to accumulate a large amount of information regarding face images included in video content while increasing the recognition rate of faces in videos.

FIG. 1 is a block diagram showing a schematic functional configuration of an image management device according to an embodiment of the present invention. FIG. 3 is a schematic diagram illustrating an example of a data configuration of a temporary cluster storage unit that stores information on temporary clusters in the same embodiment. It is a schematic diagram showing an outline of processing of a cluster selection part by the same embodiment. FIG. 2 is a schematic diagram illustrating a configuration example of person data (before name matching) held by a person database according to the embodiment. It is a schematic diagram showing an example of composition of person data (after name matching) held by a person database according to the same embodiment. FIG. 2 is a schematic diagram showing a configuration example of person data (after adding a new tag) held by the person database according to the embodiment. FIG. 3 is a schematic diagram showing a configuration example (another form) of person data (before name matching) held by the person database according to the embodiment. FIG. 3 is a schematic diagram showing a configuration example (another form) of person data (after name matching) held by the person database according to the embodiment. FIG. 7 is a schematic diagram showing a configuration example (another form) of person data (after adding a new tag) held by the person database according to the same embodiment. It is a schematic diagram showing an outline of two-stage clustering processing in the same embodiment. FIG. 3 is a schematic diagram illustrating an example of facial images classified by two-stage clustering processing in the same embodiment. FIG. 3 is a schematic diagram for explaining processing of a cut-by-cut face recognition processing section in the same embodiment. It is a schematic diagram showing an example of composition of appearance information in the same embodiment. It is a flowchart which shows the processing procedure of the image management apparatus by the same embodiment.

Next, one embodiment of the present invention will be described with reference to the drawings.

FIG. 1 is a block diagram showing a schematic functional configuration of an image management device according to this embodiment. Reference numeral 1 indicates an image management device. As shown in the figure, the image management device 1 includes a video content acquisition unit 11, a face detection processing unit 12, a clustering unit 13, a temporary cluster management unit 16, a temporary cluster storage unit 17, and face image learning for specific content. unit 18, specific content face recognition model 19, cut unit face recognition processing unit 20, cluster selection unit 21, person data registration unit 22, person database 23, performer data storage unit 24, tag setting 25. Note that the clustering section 13 includes a first clustering section 14 and a second clustering section 15.

Each of these functional units can be realized using, for example, an electronic circuit. It is also possible to implement each functional section using a computer and a program. For example, when implementing the image management device 1 using a computer, one computer may have all the functions of the image management device 1, or the functions may be distributed among multiple computers. Multiple computers are allowed to communicate with each other. Further, some of the functions of the image management device 1 may be realized by a so-called cloud server. Each functional unit has storage means, if necessary. As the storage means, for example, a semiconductor memory or a magnetic hard disk device can be used. The functions of each part are as explained below.

The video content acquisition unit 11 acquires video content. The video content includes time-series frame images. For example, the video content acquisition unit 11 receives a broadcast signal and acquires video content included in the broadcast signal. Further, even if the video content acquisition unit 11 acquires video content from an external distribution server device via a communication network, or reads video content recorded on a recording medium such as a magnetic hard disk device or an optical disk, good.

The face detection processing unit 12 detects a face image included in a frame image of the video content acquired by the video content acquisition unit 11. For example, the face detection processing unit 12 cuts out an image of the detected face image area and passes it to the clustering unit 13. The face detection processing unit 12 may normalize the size (the number of vertical and horizontal pixels) of the face image to be cut out, if necessary. Note that existing technology can be used as the face image detection technology itself. The face detection processing unit 12 detects a face in an image by referring to a model representing face-likeness.

The clustering unit 13 performs clustering based on the feature amount of the face image passed from the face detection processing unit 12. The feature amount of the face image is based on the pattern of arrangement of pixel values in the face image or in a partial region of the face image, or the arrangement pattern of the range to which the pixel values belong. The features of a face image reflect the outline of the face, the shape of the parts included in the face, their relative positions, the size of each part, the distribution of colors (pixel values), and other factors. It is something. The feature amount can be expressed as a multidimensional vector, for example. In this embodiment, multi-stage clustering is performed. Specifically, the clustering section 13 includes a first clustering section 14 and a second clustering section 15. That is, the first clustering section 14 first clusters the face image group passed from the face detection processing section 12, and the second clustering section 15 further clusters the image group output from the first clustering section 14. This multi-stage clustering process has the effect of removing noise contained in clusters, increasing the accuracy of clusters. Note that the number of stages of the clustering process is not limited to 2, and may be 1 or 3 or more. Details of the clustering process will be described later.

Note that the first clustering section 14 and the second clustering section 15 may be realized by operating the same circuit or program module with different parameters.

The temporary cluster management section 16 manages information on clusters (herein referred to as "temporary clusters") output from the clustering section 13. Specifically, the temporary cluster management unit 16 writes information identifying the temporary cluster and information regarding the feature amount of each temporary cluster to the temporary cluster storage unit 17. The feature amount information may include feature amount statistical information (for example, average value and variance value).

The temporary cluster storage unit 17 stores information on the above-mentioned temporary clusters. The temporary cluster storage unit 17 stores information for identifying a temporary cluster and information on the feature amount of the temporary cluster in association with each other. The structure of the data stored in the temporary cluster storage section 17 will be explained later with reference to another diagram. Note that the temporary cluster storage unit 17 may store the face image group itself belonging to the temporary cluster instead of storing information on the feature amount of the temporary cluster. Further, the temporary cluster storage unit 17 may store, for each temporary cluster, both information on the feature amount of the temporary cluster and a group of facial images belonging to the temporary cluster.

The specific content face image learning unit 18 reads temporary cluster identification information and feature amount information from the temporary cluster storage unit 17 and performs machine learning using them as training data, thereby obtaining the temporary cluster identification information and image feature amount. Build a model that represents the relationship between The model constructed here is a model for specific content (video content currently being processed). The specific content face image learning unit 18 performs machine learning using a neural network, for example.

The specific content face recognition model 19 is a model constructed by the specific content face image learning unit 18. In the case of a model using a neural network, the specific content face recognition model 19 specifically stores values of calculation parameters at each node in the neural network. As described above, the specific content face recognition model 19 is a model for specific content (video content currently being processed).

The cut-by-cut face recognition processing unit 20 performs processing for recognizing faces included in the video content in units of cuts regarding the video content passed from the video content acquisition unit 11. A cut is a unit in which a video camera is switched or a scene is changed. The cut-by-cut face recognition processing unit 20 determines which category the face image included in the video content belongs to by referring to the above-mentioned specific content face recognition model 19, and generates information for identifying the cut. The information for that category is output in a form that is associated with it.

The cluster selection unit 21 selects and outputs the cluster (referred to as “used cluster”) that the cut unit face recognition processing unit 20 outputs as a recognition result from among the clusters (temporary clusters) registered in the temporary cluster storage unit 17. do. The cluster selection unit 21 passes information on the selected cluster to the person data registration unit 22. The cluster information passed by the cluster selection unit 21 includes at least information for identifying the cluster and information on the feature amount of the cluster.

The person data registration unit 22 registers the cluster information passed from the cluster selection unit 21 into the person database 23.

The person database 23 is a database for accumulating information regarding people appearing in video content. The person database 23 mutually exchanges information that identifies clusters, information on image features of clusters, tags assigned to clusters (for example, person names, etc.), and other attribute information of clusters as necessary. Store in association. Furthermore, the person database 23 may include a group of facial images belonging to the cluster itself in association with information for identifying the cluster. A specific example of a method for holding a group of facial images in the person database 23 or in association with information in the person database 23 is as follows. That is, in the person database 23, several (for example, 2 to 3) representative face images are held in association with information for identifying clusters. In addition, information on locations where more facial image groups are stored is held in association with the cluster. This "location information" is, for example, a name that specifies a folder in a file system, a URL (uniform resource locator) equivalent to the name, or the like. In the folder or the like, the data of the facial image group may be held as a group of image files such as JPEG format, or the image file group may be held within a compressed file such as ZIP format. Note that if the person database 23 directly holds several representative facial images as described above, the user can quickly view those facial images when searching the person database 23.

Furthermore, the person database 23 may further accumulate appearance information. The appearance information is information indicating which person appeared in which scene (cut) of which video content. In other words, the appearance information is stored in such a manner that information for identifying content, information for identifying scenes (cuts), and information for identifying performers (clusters) are correlated with each other.

The person database 23 has a function of performing name matching processing. When multiple clusters registered in a person database are actually the same cluster (that is, they represent the same person's features), the name matching process integrates the multiple clusters into one cluster. This is a cluster process. Name matching processing is typically performed when there is a newly registered cluster and an existing cluster, in order to integrate data representing the same person from these clusters into one cluster. The person database 23 performs cluster name matching processing based on the similarity of feature amounts, for example.

Details of the person database 23 will be further explained later with reference to another figure.

The performer data storage unit 24 stores information about people who appear in the video content that is processed by the image management device 1. The performer data storage unit 24 stores, for example, the names of performers (personal names) for each video content. The performer data for each video content is not information extracted from the video (image) of the video content, but is information provided separately from the outside.

The tag setting unit 25 sets a tag to a cluster in the person database 23 to which a tag (person's name, etc.) has not yet been attached, while referring to the information on the person's name stored in the performer data storage unit 24. The tag setting unit 25 may determine which tag is assigned to which cluster based on the user's operation. That is, the tag setting section 25 has a user interface. The tag setting section 25 presents the list of performers stored in the performer data storage section 24 to the user through its user interface. Then, the name of a specific performer selected by the user or the name of a performer newly input by the user is set as a tag in a specific cluster in the person database 23.

FIG. 2 is a schematic diagram showing an example of the data structure of the temporary cluster storage unit 17 that stores information on temporary clusters obtained as a result of processing by the clustering unit 13. As illustrated, the temporary cluster storage unit 17 stores, for example, tabular data. The temporary cluster storage unit 17 stores temporary cluster identification information and feature amount information in association with each other. Temporary cluster identification information is information for identifying individual temporary clusters. Note that when the temporary cluster storage section 17 also has information on the facial image group itself as described above, the temporary cluster storage section 17 associates the temporary cluster identification information with the information on the facial image group. Remember. In the illustrated example, the temporary cluster identification information is, for example, "A", "B", "C", etc. The feature amount information is information representing the image-related features of each cluster. The feature amount information may be data obtained by statistically processing a feature amount (for example, an average value, a variance value, etc. of a certain amount). That is, the temporary cluster storage unit 17 stores information representing the characteristics of each temporary cluster.

FIG. 3 is a schematic diagram showing an outline of processing by the cluster selection unit 21. As shown in the figure, the cluster selection unit 21 selects a set of clusters used as a result of face recognition by the cut unit face recognition processing unit 20 from a set of extracted temporary clusters for a certain specific content. select. The cluster selection unit 21 passes information on a set of clusters, which is the selection result, to the person data registration unit. In the illustrated example, the cluster selection unit 21 reads temporary clusters A, B, and C from the temporary cluster storage unit 17. Further, the cluster selection unit 21 receives information on clusters A and B, which are clusters extracted by performing face recognition processing on video content, from the cut-by-cut face recognition processing unit 20. Then, the cluster selection unit 21 selects only the information of clusters A and B, which are the clusters passed from the cut unit face recognition processing unit 20, from among the temporary clusters A, B, and C read out from the temporary cluster storage unit 17. do. The cluster selection unit 21 passes information on the selected clusters A and B to the person data registration unit 22. That is, in this example, cluster C was extracted as a temporary cluster, but since cluster C was not recognized in face recognition, the cluster selection unit 21 discards cluster C without selecting it. Note that the cluster output by the cut-by-cut face recognition processing section 20 may be referred to as a "used cluster." The specification cluster is a cluster detected to have been used in the video.

4, FIG. 5, and FIG. 6 are schematic diagrams showing configuration examples of person data (cluster data) held by the person database 23. As shown in the figure, the person data is tabular data, and includes items such as cluster identification information, feature information, tags, and attribute information (person attributes, etc.).

FIG. 4 shows person data in a state where new clusters have been registered and the name matching process between these new clusters and existing clusters has not yet been performed. As shown in the figure, this person data includes A, B, X, and Y as cluster identification information. Among these, clusters X and Y are existing clusters. Clusters A and B are newly registered clusters. All clusters have feature information. The information on the feature amount includes, for example, statistical information such as the average value and variance value of the feature amount. Further, as described above, the person database 23 may have data on the facial image itself as well as data on the feature amount of the facial image. Clusters X and Y, which are existing clusters, have already been tagged. Here, the tag is a person's name corresponding to each cluster. Clusters A and B have not yet been tagged.

FIG. 5 shows the person data in a state after the name matching process has been performed from the state shown in FIG. 4. Since cluster A in FIG. 4 has been integrated with existing cluster X through the name matching process, there is no row for cluster A in the data in FIG. 5. Cluster B was not integrated with an existing cluster by the name matching process (there was no existing cluster to be matched), so the row for cluster B remains in the data in FIG. 5 as well. In this state, the tag is still unset in the cluster B row.

FIG. 6 shows the person data in a state after the tagging process has been performed from the state shown in FIG. As described above, the tag setting unit 25 sets tags for new clusters.

As described above, tag information (person name information) is also associated with newly registered clusters through the name matching process (FIG. 5) and tag setting process (FIG. 6).

The data shown in FIGS. 4, 5, and 6 may be configured in another format.
7, FIG. 8, and FIG. 9 are schematic diagrams showing other configuration examples of person data (cluster data) held by the person database 23. In the configurations shown in these figures, data related to tags (corresponding to people) and data related to image clusters are stored in separate tables, respectively, so that they can be associated with each other.

FIG. 7 shows person data in a state where new clusters have been registered and the name matching process between these new clusters and existing clusters has not yet been performed. The illustrated data includes clusters A, B, X, and Y. Among these, clusters X and Y are existing clusters. Clusters A and B are newly registered clusters. All clusters have feature information. Even in this form of data, the person database 23 may have data on the facial image itself as well as data on the feature amount of the facial image. Note that tag information is associated with each of the clusters X and Y. The tag information includes information about the tag itself (eg, name) and attribute information (personal attributes, etc.). Further, as an example, the association between the tag information and the cluster information is realized by the tag information having cluster identification information, as shown in the figure.

FIG. 8 shows the person data in a state after the name matching process has been performed from the state shown in FIG. The tag information that was associated only with cluster X in FIG. 7 is also associated with cluster A in FIG. 8 through the name matching process. Cluster B has no existing cluster to be merged, so no tag information is associated with cluster B in the data of FIG. 8 as well. That is, in this state, cluster B still has no tag set. In the data representation shown in FIG. 8, clusters A and X are not integrated as data on the table. In other words, the information of clusters A and X is not lost even in the data shown in FIG. This is a point in which the data representation in FIG. 8 differs from the data representation in FIG. 5.

FIG. 9 shows the person data in a state after the tagging process has been performed from the state shown in FIG. In FIG. 9, tag information is also associated with cluster B. As described above, the tag setting unit 25 sets a tag for the new cluster (cluster B).

As described above, tag information (person name information) is also associated with newly registered clusters through the name matching process (FIG. 8) and tag setting process (FIG. 9).

Next, details of the characteristic processing that the image management device 1 has will be further explained.

FIG. 10 is a schematic diagram showing an overview of two-stage clustering processing by the clustering unit 13. As shown in the figure, the group of images to be clustered is classified into a plurality of clusters by first-stage clustering and second-stage clustering processing. The first clustering unit 14 performs first-stage clustering, and the second clustering unit 15 performs second-stage clustering. In the figure, reference numeral 301 indicates a group of images before clustering. Reference numeral 302 is the result of the first stage clustering. As a result of the first stage clustering, it is classified into clusters 1, 2, and 3. Reference numeral 303 is the result of the second stage clustering. As a result of the second stage clustering, the original cluster 1 has been classified into two clusters, clusters 1 and 4. Each of the original clusters 2 and 3 remains intact as clusters 2 and 3, respectively.

Both the first-stage and second-stage clustering processes use, for example, DBSCAN (Density-based spatial clustering of applications with noise). Different parameters are used in the first and second stages. In the first stage of clustering, the search range is set wide and classification is performed with coarse granularity. In addition, in the second stage of clustering, accuracy is improved by narrowing the search range and reclassifying. By performing clustering in two stages in this way, it is possible to reduce the mixing of noise. Noise here means that a certain cluster is not composed only of facial images of a specific person, but includes facial images of other people. Improving the accuracy of face image clusters in this way leads to an improvement in the accuracy of the face recognition model for face recognition processing (specific content face recognition model 19). In other words, the accuracy of face recognition processing by the cut-by-cut face recognition processing unit 20 is improved.

FIG. 11 is a schematic diagram showing an example of facial images classified by two-stage clustering processing. In the figure, (A) shows a group of images belonging to cluster 1 obtained as a result of the first stage clustering. Further, (B) shows a group of images belonging to cluster 1 obtained as a result of the second stage clustering. Further, (C) shows a group of images belonging to cluster 4 obtained as a result of the second stage clustering. As described above (FIG. 10), cluster 1 (A) in the first stage is classified into cluster 1 (B) and cluster 4 (C) in the second stage. That is, in (A), images 311 to 315 are face images of the same person, and images 316 and 317 are mixed into cluster 1 as noise. As a result of the second-stage clustering process, cluster 1 in (B) includes images 311 to 315, and cluster 4 in (C) includes images 316 and 317. In other words, images 316 and 317, which are noise, are separated from cluster 1 as cluster 4 by the second stage of clustering.

FIG. 12 is a schematic diagram for explaining processing by the cut unit face recognition processing section 20. As shown in FIG. The figure shows a comparison of the recognition results when face recognition processing is performed in units of cuts and the recognition results when face recognition processing is performed at fixed time intervals (for example, every second). There is. In the figure, (A) is the result of face recognition processing in units of cuts (processed by the cut unit face recognition processing section 20). (B) is a comparison target and is the result of face recognition processing in units of seconds (every second).

As already mentioned, the cut-by-cut face recognition processing unit 20 divides a video into cut units and performs face recognition processing while tracking a person in each cut. For cut detection, the cut unit face recognition processing section 20 refers to the amount of change in the color histogram, for example. Specifically, the cut-by-cut face recognition processing unit 20 calculates, for example, a color histogram of each frame of a video to be recognized. Therefore, the cut-by-cut face recognition processing unit 20 calculates the pixel frequency for each value range of each primary color of RGB in each frame. As an example, the cut-by-cut face recognition processing unit 20 determines that in one frame, the value of the R (red) channel is 0% or more and less than 25%, 25% or more and less than 50%, 50% or more and less than 75%, 75 The number of pixels belonging to each of four ranges of % or more and 100% or less is counted. Similar processing is performed for the G (green) channel and the B (blue) channel. The cut-by-cut face recognition processing unit 20 calculates the amount of change in the temporal direction of the color histogram for each frame thus calculated between adjacent frames. A point where the amount of change temporarily increases appears in the video, and the cut unit face recognition processing section 20 detects that point as a cut point.

The cut-by-cut face recognition processing unit 20 performs recognition processing on a face image in each frame while tracking a person on the premise that there is a tendency for very few people to be replaced within a cut. For example, if face recognition processing is performed at fixed time intervals (comparison targets, e.g. 1 second intervals), recognition errors may occur due to changes in the direction of the face of the person in the video, changes in lighting, etc. There may be cases where the face image is recognized as a facial image. However, by tracking the person within the cut, it becomes possible to determine such recognition errors as noise. In other words, the cut-by-cut face recognition processing unit 20 can exclude noise (a recognition result in which facial images that are supposed to be of the same person are recognized as different people). In other words, the cut unit face recognition processing section 20 can eliminate recognition errors.

In FIG. 12, regarding face recognition in units of cuts in (A), information on cut numbers, time intervals (the length of time is variable for each cut), and recognized clusters is shown. Furthermore, regarding face recognition in units of seconds in (B), time intervals (the length of time is, for example, 1 second) and information on recognized clusters are shown. Also, for convenience, line numbers are attached near the center (between (A) and (B)). Note that the time interval is expressed using relative time with the start time of the content as zero.

In the illustrated example, cut number 98 in face recognition in cut units corresponds to the time interval 0:30:01 to 0:30:11, and corresponds to lines 1 to 10 in face recognition in seconds. do. Cut number 99 in face recognition in cut units corresponds to the time interval 0:30:11 to 0:30:18, and corresponds to lines 11 to 17 in face recognition in seconds. Cut number 100 in face recognition in cut units corresponds to the time interval 0:30:18 to 0:30:27, and corresponds to lines 18 to 26 in face recognition in seconds. Cut number 101 in face recognition in cut units corresponds to the time interval 0:30:27 to 0:30:31, and corresponds to lines 27 to 30 in face recognition in seconds. On the other hand, each line from the 1st line to the 30th line of face recognition in seconds represents 30 time intervals from 1 second starting at 0:30:01 to 1 second starting at 0:30:30. This corresponds to

In the cut-by-cut face recognition processing performed by the cut-by-cut face recognition processing unit 20, for cut number 98, two clusters, A and B, are output as recognition results. Furthermore, for cut number 99, there is no recognized cluster. Furthermore, for cut number 100, only cluster A is output as the recognition result. Furthermore, for cut number 101, only cluster E is output as the recognition result. Note that the cut-by-cut face recognition processing unit 20 may output cluster information as a recognition result for a specific time period within a cut. For example, the cut-by-cut face recognition processing unit 20 outputs cluster A as the recognition result in the time interval 0:30:01 to 0:30:07 in the cut 98. In addition, the cut-by-cut face recognition processing unit 20 outputs cluster B as a recognition result in the time interval 0:30:01 to 0:30:11 in cut 98 (cluster B is continuous in the time interval ). In addition, the cut-by-cut face recognition processing unit 20 generates a cluster A as a recognition result in the time interval 0:30:18 to 0:30:21 and the time interval 0:30:24 to 0:30:27 in the cut 100. Output. Further, the cut-by-cut face recognition processing unit 20 outputs cluster E as the recognition result in the time interval 0:30:28 to 0:30:31 in the cut 101.

When performing second-by-second face recognition processing (B) on the same video content, the set of clusters recognized in each time interval will be the same as the corresponding cut-by-cut recognition processing results, unless misrecognition is taken into account. It is a subset of the cluster set (including the case where it is an empty set). However, when performing face recognition on a second-by-second basis, erroneous recognition may occur due to not tracking a person on a cut-by-cut basis. In the illustrated example, cluster C included in the recognition results in the fifth and sixth rows is a cluster that has been misrecognized. Furthermore, cluster D included in the recognition result in the 24th line is a cluster that has been misrecognized.

As described above, in this embodiment, the cut-by-cut face recognition processing unit 20 performs face recognition processing in cut units. In other words, the cut-by-cut face recognition processing unit 20 performs face recognition processing while tracking a person on the premise that there is no (or only a small) change of people within a cut. Thereby, the cut-by-cut face recognition processing section 20 removes noise from the face recognition result. In other words, the cut-by-cut face recognition processing section 20 can reduce misrecognition. In other words, the cut-by-cut face recognition processing unit 20 tracks the face image between frames for each time interval divided by the cut point, detects an error in cluster determination based on the tracking result, and corrects the detected error. (revise).

FIG. 13 is a schematic diagram showing an example of the configuration of a table of appearance information held in the person database 23. As shown in the figure, the tabular data representing appearance information is data in which content identification information, time interval identification information, and performer identification information are comprehensively associated. Content identification information is information for identifying video content. The time section identification information is information that identifies a time section within the content. Specifically, the time interval identification information is, for example, a number assigned to a time interval (for example, the cut number shown in FIG. 9), a set of a start time and an end time of a time interval, and the like. Performer identification information is information for identifying performers. The performer identification information may be, for example, the cluster identification information shown in FIG. 4, etc., or a person's name (tag shown in FIG. 4, etc.). The appearance information according to the illustrated example indicates that each performer corresponding to categories A and B appeared in the video in the time section identified as "Cut 98" of the content identified as "Content X". represent. By accumulating such appearance information, it is possible to manage which performer appeared in which scene of which video content out of a large amount of video content. In this way, the person database 23 of the image management device 1 manages information on performers who have appeared in video content by associating it with cuts (scenes), for example, based on automatically recognized (identified) facial images. be able to.

FIG. 14 is a flowchart showing the processing procedure of the image management device 1. The operating procedure will be explained below along with this flowchart.

First, in step S11, the face detection processing unit 12 detects a face in the video content. Here, the content whose face is to be detected is specific content. The face detection processing unit 12 passes the image of the area including the detected face to the clustering unit 13.
Next, in step S12, the clustering unit 13 performs clustering processing on the face images passed from the face detection processing unit 12. Specifically, as already explained, the first clustering unit 14 performs the first stage clustering process, and the second clustering unit 15 performs the second stage clustering process. In other words, the clustering unit 13 performs two-stage clustering.

Next, in step S13, the temporary cluster management unit 16 receives the results of the clustering process from the clustering unit 13. Then, the temporary cluster management unit 16 registers information about each temporary cluster in the temporary cluster storage unit 17, regarding all of these clusters as “temporary clusters”. Here, the information on the temporary cluster includes at least information for identifying the temporary cluster and information regarding the feature amount (image feature amount) for the temporary cluster. Further, the information on the feature amount may be, for example, statistical information such as numerical values regarding the image.

Next, in step S14, the specific content face image learning unit 18 performs learning processing for each of the temporary clusters registered in step S13. Specifically, the specific content face image learning unit 18 performs a learning process using face images narrowed down for each temporary cluster, and constructs a model for face recognition processing. The model that is constructed has information about the shape, color, size, and other characteristics of the entire face or each part of the face. The model constructed in this step is a model for determining clusters based on facial images. Note that the learning process itself can be realized using existing machine learning technology. As an example, a neural network or the like can be used for the learning process. The trained model obtained in this step is written to the storage medium as the specific content face recognition model 19.

Next, in step S15, the cut-by-cut face recognition processing unit 20 performs face recognition processing for the specific video content in cut units. The cut-by-cut face recognition processing unit 20 passes information on a set of clusters, which is the recognition result for each cut, to the cluster selection unit 21. The clusters obtained in this step are clusters used in the video, and may be called "used clusters."

Next, in step S16, the cluster selection unit 21 selects only the information on the used clusters output by the cut unit face recognition processing unit 20 in step S15 from among the information on the temporary clusters stored in the temporary cluster storage unit 17. select. The cluster selection unit 21 passes information on the selected cluster to the person data registration unit 22.
Next, in step S17, the person data registration section 22 uses the data passed from the cluster selection section 21 to register the used cluster in the person database.

Next, in step S18, the person database 23 performs name matching processing on the data it holds. In other words, the person database 23 performs name matching of data on newly registered clusters that match existing clusters (clusters that can be determined to be the same person). A specific example of the name matching process is as described with reference to FIGS. 4 and 5.

Through the above series of processes, the image management device 1 constructs a face recognition model for specific content, performs face recognition processing based on the face recognition model for specific content, and identifies clusters (persons) obtained as recognition results. Can be registered in the person database.

Note that at least some of the functions of the image management apparatus in the embodiments described above can be realized by a computer. In that case, a program for realizing this function may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read into a computer system and executed. Note that the "computer system" herein includes hardware such as an OS and peripheral devices. Furthermore, "computer-readable recording media" refers to portable media such as flexible disks, magneto-optical disks, ROM, CD-ROM, DVD-ROM, and USB memory, and storage devices such as hard disks built into computer systems. Say something. Furthermore, a "computer-readable recording medium" refers to a medium that temporarily and dynamically stores a program, such as a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In that case, it may also include something that retains a program for a certain period of time, such as a volatile memory inside a computer system that is a server or client. Further, the program may be one for realizing a part of the above-mentioned functions, or may be one that can realize the above-mentioned functions in combination with a program already recorded in the computer system.

Although the embodiments have been described above, the present invention can be further implemented in the following modifications. For example, the cut-by-cut face recognition processing unit 20 performed face recognition processing in cut units. However, the face recognition process may be performed in units of time other than cuts. Further, for example, the name matching process in the person database 23 may not be performed automatically, but may be performed based on human judgment. Furthermore, for example, the image management device 1 may be configured without the performer data storage section 24. Furthermore, the image management device 1 may be configured without the tag setting section 25.

The embodiment described above and its modifications are summarized as follows.

The image management device 1 includes a clustering unit 13, a temporary cluster storage unit 17, a specific content face image learning unit 18, a face recognition processing unit (cut unit face recognition processing unit 20), a person database 23, and person data. and a registration section 22. The clustering unit 13 clusters facial images extracted from video content based on image features. The temporary cluster storage unit 17 stores image feature amounts for each cluster obtained through the processing of the clustering unit 13. The specific content face image learning unit 18 determines the cluster based on the face image included in the specific video content by performing machine learning processing based on information representing the relationship between the face image and the cluster. Generate a facial recognition model for video content. The face recognition processing unit performs recognition processing on a face image included in the video content based on the generated face recognition model, and outputs cluster information as a recognition result. The person database 23 stores image features of facial images and clusters in association with each other. The person data registration unit 22 registers, in the person database 23, information on image feature amounts for each cluster read from the temporary cluster storage unit 17 for the clusters (used clusters) output from the face recognition processing unit.

Thereby, the face recognition processing unit can perform face recognition processing, that is, processing for determining a cluster corresponding to a face image, based on a model dedicated to the video content. Since a model specific to the video content is used, the number of target clusters (number of people) is relatively limited, and the probability of misrecognition is low. In other words, clusters can be correctly determined with high accuracy.

The clustering unit 13 may have multiple stage clustering units for performing clustering processing at multiple stages. Specifically, the multiple stage clustering units are the first clustering unit 14 and the second clustering unit 15, which have already been described. The stage clustering unit after the second stage (that is, the second clustering unit 15 in this embodiment) uses the clusters output from the previous stage clustering unit (that is, the first clustering unit 14 in this embodiment). Clustering processing is performed on facial images. Note that the number of stages of clustering may be three or more.

Thereby, as already explained, it is possible to improve the accuracy of clustering.

The image management device 1 may further include a name matching processing section (not shown). The name matching processing unit may be a function provided within the person database 23, for example. The name matching processing unit integrates a plurality of clusters into one cluster (name matching) based on image feature amounts registered in the person database 23.

This makes it easy to integrate a newly registered cluster with an existing cluster.

As already explained, the person database 23 may store tags assigned to clusters in association with the clusters. The tag represents, for example, a person's name. Further, a tag setting section 25 may be provided. The tag setting unit 25 performs a process of setting new tags for clusters for which no tags have been set.

This makes it possible to manage a newly registered cluster by associating it with a person's name, etc.

The face recognition processing section may perform face recognition processing for each time interval divided by a cut point. The cut point is a point at which the amount of change in pixel values between frames of video content (for example, the overall evaluation value of pixel values in the entire frame) reaches a peak. The face recognition processing unit tracks the face image between frames for each time interval separated by the cut point, detects errors in cluster determination based on the tracking results, corrects the detected errors, and outputs the results as recognition results. It may be something that outputs cluster information. Face recognition in units of cuts and error detection and correction using assumptions within a cut are as described with reference to FIG. 12.

The person database 23 may further store information (appearance information) associating the time intervals with clusters. This makes it possible to easily manage which person appeared in which time section.

Although the embodiments of the present invention have been described above in detail with reference to the drawings, the specific configuration is not limited to these embodiments, and includes designs within the scope of the gist of the present invention.

The present invention can be used, for example, in businesses that manage and distribute video content (including broadcast programs). However, the scope of use of the present invention is not limited to what is exemplified here.

1 Image management device 11 Video content acquisition unit 12 Face detection processing unit 13 Clustering unit 14 First clustering unit 15 Second clustering unit 16 Temporary cluster management unit 17 Temporary cluster storage unit 18 Specific content face image learning unit 19 Specific content face Recognition model 20 Cut unit face recognition processing section 21 Cluster selection section 22 Person data registration section 23 Person database 24 Performer data storage section 25 Tag setting section

Claims (7)

a clustering unit that clusters face images extracted from video content based on image features;
a temporary cluster storage unit that stores the image feature amount for each cluster obtained by the processing of the clustering unit;
a face recognition model for the video content for determining the cluster based on the face image included in the video content by performing machine learning processing based on information representing the relationship between the face image and the cluster; a specific content face image learning unit that generates
a face recognition processing unit that performs recognition processing on a face image included in the video content based on the generated face recognition model and outputs cluster information as a recognition result;
a person database that stores image features of facial images and clusters in association with each other;
a person data registration unit that registers the image feature amount for each cluster read from the temporary cluster storage unit with respect to the cluster output from the face recognition processing unit in the person database;
An image management device comprising: The clustering unit has a plurality of stage clustering units for performing clustering processing in a plurality of stages, and the stage clustering unit after the second stage uses the face of the cluster output from the previous stage clustering unit. Perform clustering processing on images,
The image management device according to claim 1. a name matching processing unit that integrates a plurality of clusters into one cluster based on the image feature amount registered in the person database;
The image management device according to claim 1 or 2, further comprising:. The person database stores tags assigned to clusters in association with the clusters,
further comprising a tag setting unit that performs processing to set a new tag for the cluster for which no tag is set;
An image management device according to any one of claims 1 to 3. The face recognition processing unit tracks the face image between frames for each time interval separated by a cut point at which the amount of change in pixel value between frames of the video content peaks, and identifies the cluster based on the tracking result. detecting an error in the determination, correcting the detected error, and outputting cluster information as the recognition result;
An image management device according to any one of claims 1 to 4. The person database further stores information associating the time interval with the cluster.
The image management device according to claim 5. a clustering unit that clusters face images extracted from video content based on image features;
a temporary cluster storage unit that stores the image feature amount for each cluster obtained by the processing of the clustering unit;
a face recognition model for the video content for determining the cluster based on the face image included in the video content by performing machine learning processing based on information representing the relationship between the face image and the cluster; a specific content face image learning unit that generates
a face recognition processing unit that performs recognition processing on a face image included in the video content based on the generated face recognition model and outputs cluster information as a recognition result;
a person database that stores image features of facial images and clusters in association with each other;
a person data registration unit that registers, in the person database, the image feature amount for each cluster read from the temporary cluster storage unit for the cluster output from the face recognition processing unit;
A program that allows a computer to function as an image management device.

Images