JP6793169B2 - Thumbnail output device, thumbnail output method and thumbnail output program - Google Patents

Description

The present invention relates to a thumbnail output device for outputting thumbnails of video data, a thumbnail output method, and a thumbnail output program.

Generally, one frame data of video data is published as a thumbnail to introduce the video data. Therefore, as the thumbnail, it is preferable to use frame data that captures the characteristics of the video data.

Thumbnails can be manually selected, but there is also a technique for extracting thumbnails from video data using AI (Artificial Intelligence) (see Non-Patent Document 1 and Non-Patent Document 2). The technology disclosed in Non-Patent Document 1 mainly targets genres such as dramas and movies, and creates thumbnails based on past know-how. The technology disclosed in Non-Patent Document 2 includes a large number of thumbnails set by the user for video data, which are presumed to have a high number of playbacks and high quality, and thumbnails which are presumed to have a low number of playbacks and low quality. To generate thumbnails.

Madeline, 6 outsiders, "AVA: The Art and Science of Image Discovery at Netflix", [online], February 8, 2018, Netflix Technology Blog, [Search October 30, 2018], Internet <URL: https://medium.com/netflix-techblog/ava-the-art-and-science-of-image-discovery-at-netflix-a442f163af6> Munenori Taniguchi, "Google that automates everything explains amazing YouTube thumbnail automatic generation, selects the optimum frame with artificial neural network technology", [online], October 14, 2015, engadget, [October 30, 2018 Search], Internet <URL: https://japanese.engadget.com/2015/10/14/google-youtube/>

However, since all the documents create thumbnails based on past know-how or selection results, the intention of the user who creates thumbnails of video data is not taken into consideration. For example, it is not possible to create a thumbnail in consideration of the user's intention such as changing the extraction point of the thumbnail according to the genre, content, etc. of the video data.

Therefore, an object of the present invention is to provide a thumbnail output device, a thumbnail output method, and a thumbnail output program for outputting thumbnails of video data, reflecting the intention of the user.

In order to solve the above problems, the first feature of the present invention relates to a thumbnail output device that outputs thumbnail data of video data. In the thumbnail output device according to the first feature of the present invention, for each person recognized by the frame data constituting the video data, the time until the next frame data to be processed of the frame data recognized by the person is added. The appearance time is calculated, the character score, which is the ratio of the appearance time of the person to the maximum value of the appearance time of each person, is calculated for each person, and the character score of the person with the largest face area in the frame data is calculated. It includes a person importance score calculation unit that calculates as a person importance score of frame data, and a thumbnail output unit that outputs frame data with a high character score as thumbnail data.

The second feature of the present invention relates to a thumbnail output device that outputs thumbnail data of video data. The thumbnail output device according to the second feature of the present invention calculates the frame data constituting the video data and the area of the face area of the person recognized by the frame data.

The face area area score calculation unit for calculating the face area area score, which is higher for the frame data having the area of the face area close to the optimum area and lower for the frame data having the area of the face area far from the optimum area, and the face area area score It is provided with a thumbnail output unit that outputs high frame data as thumbnail data.

A third feature of the present invention relates to a thumbnail output device that outputs thumbnail data of video data. The thumbnail output device according to the third feature of the present invention calculates a facial expression value for each facial expression type of a person recognized by the frame data constituting the video data, and the facial expression value of each facial expression type of the person in the frame data. It is provided with a facial expression score calculation unit that calculates the maximum value of each facial expression value with respect to the total of the above as the facial expression score of the frame data, and a thumbnail output unit that outputs the frame data having a high facial expression score as the facial expression data.

A fourth feature of the present invention relates to a thumbnail output device that outputs thumbnail data of video data. The thumbnail output device according to the fourth feature of the present invention calculates a volume score that calculates a volume score that increases for frame data corresponding to a time when the volume of video data is high and decreases for frame data corresponding to a time when the volume is low. It is provided with a unit and a thumbnail output unit that outputs frame data having a high volume score as thumbnail data.

A fifth feature of the present invention relates to a thumbnail output device that outputs thumbnail data of video data. The thumbnail output device according to the fifth feature of the present invention has the frame data of the video data as the person importance score described in the first feature, the facial area area score described in the second feature, and the third feature. An integrated score calculation unit that calculates an integrated score obtained by multiplying a plurality of scores including one or more of the described facial expression score and the volume score described in the fourth feature by weights, respectively, and a high integrated score. It is provided with a thumbnail output unit that outputs frame data as thumbnail data.

A sixth feature of the present invention relates to a thumbnail output method for outputting thumbnail data of video data. In the thumbnail output method according to the sixth feature of the present invention, for each person recognized by the frame data constituting the video data, the time until the frame data to be processed next to the frame data recognized by the person is reached. In addition to calculating the appearance time by adding, the computer calculates the character score, which is the ratio of the appearance time of the person to the maximum value of the appearance time of each person, for each person, and the computer is the most in the frame data. It includes a step of calculating a character score of a person having a large face area as a person importance score of frame data, and a step of a computer outputting frame data having a high character score as thumbnail data.

A seventh feature of the present invention relates to a thumbnail output method for outputting thumbnail data of video data. In the thumbnail output method according to the seventh feature of the present invention, the computer calculates the area of the face area of the person recognized by the frame data constituting the video data and the frame data, and obtains the area of the face area close to the optimum area. The step of calculating the face area area score that becomes higher for the frame data that has and becomes lower for the frame data that has the area of the face area far from the optimum area, and the writing computer outputs the frame data with the high face area area score as thumbnail data. Have steps to do.

The eighth feature of the present invention relates to a thumbnail output method for outputting thumbnail data of video data. In the thumbnail output method according to the eighth feature of the present invention, the computer calculates the facial expression value for the facial expression type of the person recognized by the frame data constituting the video data, and each facial expression type of the person in the frame data. It includes a step of calculating the maximum value of each facial expression value with respect to the total of the facial expression values of the above as a facial expression score of frame data, and a step of the computer outputting frame data having a high facial expression score as thumbnail data.

A ninth feature of the present invention relates to a thumbnail output method for outputting thumbnail data of video data. In the thumbnail output method according to the ninth feature of the present invention, the computer calculates a volume score that increases for the frame data corresponding to the time when the volume of the video data is high and decreases for the frame data corresponding to the time when the volume is low. It includes a step and a step in which the computer outputs frame data having a high volume score as thumbnail data.

A tenth feature of the present invention relates to a thumbnail output method for outputting thumbnail data of video data. In the thumbnail output method according to the tenth feature of the present invention, the computer performs the frame data of the video data with respect to the person importance score described in the sixth feature, the facial area area score described in the seventh feature, and the eighth. A computer integrates a step of calculating an integrated score obtained by multiplying and adding weights to a plurality of scores including one or more of the facial expression score described in the feature and the volume score described in the ninth feature. It includes a step of outputting frame data having a high score as thumbnail data.

The eleventh feature of the present invention relates to a thumbnail output program for causing a computer to function as the thumbnail output device according to the first to fifth features of the present invention.

According to the present invention, it is possible to provide a thumbnail output device, a thumbnail output method, and a thumbnail output program for outputting thumbnails of video data, reflecting the intention of the user.

It is a figure explaining the hardware composition and the functional block of the thumbnail output device which concerns on embodiment of this invention. It is a figure explaining an example of the data item of the condition data of the thumbnail output apparatus which concerns on embodiment of this invention. It is a figure explaining an example of the data item of the score data of the thumbnail output apparatus which concerns on embodiment of this invention. It is a flowchart explaining the person importance score calculation process by the person importance score calculation part which concerns on embodiment of this invention. It is a figure explaining an example of the person recognition result referred to in the person importance score calculation part which concerns on embodiment of this invention. It is a figure explaining an example of the processing target frame data referred to in the person importance score calculation part which concerns on embodiment of this invention. It is a flowchart explaining the face area area score calculation process by the face area area score calculation part which concerns on embodiment of this invention. It is a flowchart explaining the facial expression score calculation process by the facial expression score calculation unit which concerns on embodiment of this invention. It is a figure explaining an example of the facial expression recognition result referred to in the facial expression score calculation part which concerns on embodiment of this invention. It is a flowchart explaining the volume score calculation process by the volume score calculation part which concerns on embodiment of this invention.

Next, an embodiment of the present invention will be described with reference to the drawings. In the description of the drawings below, the same or similar parts are designated by the same or similar reference numerals.

(Thumbnail output device)
The thumbnail output device 1 according to the embodiment of the present invention will be described with reference to FIG. The thumbnail output device 1 outputs thumbnail data 15 that reflects the user's intention from the video data 11.

The thumbnail output device 1 is a general computer including a storage device 10 and a processing device 20. The function shown in FIG. 1 is realized by executing a thumbnail output program on a general computer.

The storage device 10 is a ROM (Read Only Memory), a RAM (Random access memory), a hard disk, or the like, and stores various data such as input data, output data, and intermediate data for the processing device 20 to execute processing. .. The processing device 20 is a CPU (Central Processing Unit) that reads and writes data stored in the storage device 10 and executes processing in the thumbnail output device 1.

Further, although not shown in FIG. 1, an input / output device such as a keyboard, a mouse, and a display, an input / output interface serving as an interface between the input / output device and the processing device, and the like may be provided.

The storage device 10 stores the thumbnail output program, and also stores the video data 11, the frame data 12 to be processed, the condition data 13, the score data 14, and the thumbnail data 15.

The video data 11 is the content expressed by the thumbnail data 15 output by the thumbnail output device 1. The video data 11 includes a plurality of frame data such as 30 images per second, and time-series audio data. The audio data is data on the transition of audio and volume in the video data 11.

The processing target frame data 12 is frame data to be processed by the score calculation unit 23, which will be described later, among the plurality of frame data of the video data 11. The number of frame data 12 to be processed may be plural. The frame data 12 to be processed may be data thinned out from a plurality of frame data included in the video data 11 at a predetermined frequency, such as one frame per second, or may be randomly thinned out data. You may. Further, the processing target frame data 12 may be extracted from the video data 11 through predetermined processing such as the frame data extracted from each scene by dividing the video data into scenes.

The condition data 13 is the data of the conditions for creating the thumbnail data 15. The condition data 13 includes an optimum area, the number of thumbnails, and weights, as shown in FIG. The optimum area is referred to in the face area area score calculation unit 25 described later. In the embodiment of the present invention, the optimum area is expressed by the area ratio with respect to the frame data, but may be expressed by the number of pixels in the frame data. The number of thumbnails is the number of thumbnail data 15 output by the thumbnail output unit 29, which will be described later, and a natural number is set.

The weight is referred to when calculating the integrated score considering a plurality of scores including one or more of the person importance score, the face area area score, the facial expression score, and the volume score for the integrated score calculation unit 28 described later. .. Weights are set for one or more of the person importance score, face area score, facial expression score and volume score, and for each of the other scores used to calculate the integrated score. ..

The score data 14 is the data of the calculation result by the score calculation unit 23. As shown in FIG. 3, the score data 14 associates a person importance score, a face area area score, a facial expression score, a volume score, and an integrated score with the frame data identifier. The frame data identifier shown in FIG. 3 is an identifier of the processing target frame data 12. The person importance score, face area area score, facial expression score, volume score, and integrated score are the person importance score calculation unit 24, face area area score calculation unit 25, facial expression score calculation unit 26, and volume of the score calculation unit 23, respectively. It is the calculation result of each of the score calculation unit 27 and the integrated score calculation unit 28.

In the score data 14, it is not necessary to associate the person importance score, the face area area score, the facial expression score, the volume score, and the integrated score with each frame data, and the score referred to when outputting the thumbnail data 15 is set. It should be done. For example, when the thumbnail data 15 is output based only on the person importance score, only the person importance score may be set for each frame data. Further, when the thumbnail data 15 is output based on the integrated score that integrates the person importance score and the face area area score, if only the person importance score, the face area area score, and the integrated score are set for each frame data. good.

The thumbnail data 15 is the thumbnail data 15 output by the thumbnail output device 1. The thumbnail data 15 is determined based on the score calculated for each of the processing target frame data 12 shown in the score data 14. The storage device 10 may store the thumbnail data 15 of the number of thumbnails of the condition data 13.

The processing device 20 includes a processing target frame extraction unit 21, a condition data acquisition unit 22, a score calculation unit 23, and a thumbnail output unit 29.

The processing target frame extraction unit 21 extracts the frame data that is a candidate for the thumbnail data 15 from the frame data constituting the video data 11 as the processing target frame data 12. The processing target frame extraction unit 21 may extract the processing target frame data 12 by thinning out from a plurality of frame data included in the video data 11 such as one per second at a predetermined frequency, or randomly extracts the processing target frame data 12. You may. Further, the processing target frame data 12 may be obtained by dividing the video data into scenes and extracting the processing target frame data 12 through predetermined processing such as frame data extracted from each scene. Further, the processing target frame extraction unit 21 may extract each frame data of the video data 11 as the processing target frame data 12.

The condition data acquisition unit 22 acquires each item of the condition data 13 described with reference to FIG. 2, for example, by input of a user, and stores the condition data 13 in the storage device 10. These items may be stored in the storage device 10 in advance. Further, it is not necessary to set each item of the condition data 13, and when the integrated score is not calculated, a necessary condition such that the weight item is not set may be set.

Further, the condition data acquisition unit 22 has an index indicating which of the person importance score, the face area area score, the facial expression score, the volume score, and the integrated score is used to output the thumbnail data 15. You may.

The score calculation unit 23 calculates a score as an index for determining the thumbnail data 15 for each of the processing target frame data 12. The score calculation unit 23 includes a person importance score calculation unit 24, a face area area score calculation unit 25, a facial expression score calculation unit 26, a volume score calculation unit 27, and an integrated score calculation unit 28.

In the embodiment of the present invention, the person importance score calculation unit 24, the face area area score calculation unit 25, the facial expression score calculation unit 26, the volume score calculation unit 27, and the integrated score calculation unit 28 are the processing target frame data 12, respectively. The case of calculating the person importance score, the face area area score, the facial expression score, the volume score, and the integrated score will be described for each, but the present invention is not limited to this. For example, when the thumbnail data 15 is output based only on the person importance score, only the person importance score calculation unit 24 needs to process it. Further, when the thumbnail data 15 is output based on the person importance score and the face area area score, the person importance score calculation unit 24, the face area area score calculation unit 25, and the integrated score calculation unit 28 may process the thumbnail data 15. In this way, the calculation unit that processes the processing target frame data 12 may be limited according to the index that outputs the thumbnail data 15.

The person importance score calculation unit 24 calculates the person importance score of the processing target frame data 12. A person who appears large and long in the video data 11 is estimated to be an important person, and the person importance score is set to be higher for the frame data in which the important person is shown.

The face area area score calculation unit 25 calculates the face area area score of the processing target frame data 12. The face area area score indicates the approximation of the area of the face portion of the person recognized in the processing target frame data 12 to the optimum area set as the condition data 13. The optimum area is an index of the face area area to be displayed in the thumbnail data 15.

For example, in a movie, a documentary program closely related to one person, or the like, when it is desired to select a close-up image of a person as thumbnail data 15, a value of 100% or a value close to 100% is set for the optimum area. When it is desired to select the landscape as the thumbnail data 15 in a program introducing the landscape, a value of 0% or a value close to 0% is set for the optimum area. Further, in a comedy program or the like, when it is desired to select a scene in which a plurality of people perform one performance as thumbnail data 15, a value corresponding to the area of the face per person is set as the optimum area.

The facial expression score calculation unit 26 calculates the facial expression score of the processing target frame data 12. The facial expression score of the processing target frame data 12 is calculated. The facial expression score is set to be high for the frame data in which the facial expression of the person recognized by the processing target frame data 12 is rich.

The volume score calculation unit 27 calculates the volume score of the processing target frame data 12. The volume score is set to be high when the volume corresponding to the time of the processing target frame data 12 is high. For example, in a movie or a variety show, it can be estimated that the volume rises during a loud time, so the volume score is set to be higher for the frame data during the loud time. The volume score is suitable when it is desired to output frame data, which is presumed to have a large excitement because the volume is high, as thumbnail data 15.

The integrated score calculation unit 28 calculates the integrated score of the processing target frame data 12. The integrated score of the processing target frame data 12 is calculated. The integrated score is calculated based on the weight defined in the condition data 13 for a plurality of scores including one or more of the person importance score, the face area area score, the facial expression score, and the volume score. The integrated score is suitable when it is desired to output thumbnail data 15 based on a plurality of viewpoints.

Each process of the person importance score calculation unit 24, the face area area score calculation unit 25, the facial expression score calculation unit 26, the volume score calculation unit 27, and the integrated score calculation unit 28 will be described in detail later.

The thumbnail output unit 29 outputs the processing target frame data 12 of the number of thumbnails specified in the condition data 13 as the thumbnail data 15 in descending order of the score based on each score calculated by the score calculation unit 23. The thumbnail output unit 29 outputs the processing target frame data 12 having a high score as the thumbnail data 15 based on one of the person importance score, the face area area score, the facial expression score, and the volume score. Alternatively, the thumbnail output unit 29 outputs thumbnail data 15 in consideration of a plurality of scores based on the integrated score.

(Person importance score calculation department)
The person importance score calculation unit 24 processes, for each person recognized by the processing target frame data 12, the appearance time obtained by adding the time until the next processing target frame data of the processing target frame data recognized by the person. It is calculated for each person recognized by the target frame data, and the character score, which is the ratio of the appearance time of the person to the maximum value of the appearance time of each person, is calculated for each person. The person importance score calculation unit 24 further calculates the character score of the person having the largest proportion of the face area in the processing target frame data 12 as the person importance score of the processing target frame data 12. The person importance score calculation unit 24 calculates the person importance score for each of the processing target frame data 12 and stores it in the score data 14.

When the thumbnail data 15 is output based only on the person importance score, the thumbnail output unit 29 outputs the frame data having a high character score as the thumbnail data 15.

The person importance score calculation process by the person importance score calculation unit 24 will be described with reference to FIG.

First, in step S101, the person importance score calculation unit 24 recognizes the person in the image for each of the processing target frame data 12. In person recognition, for example, API (Application Programming Interface) such as Azure, AWS, Google may be used. As a result, as shown in FIG. 5, the relationship between the persons recognized in each frame data is specified for each frame identifier. In the example shown in FIG. 5, the persons P1 and P2 are recognized in the processing target frame data 12 of the frame identifier f1. Persons P1 and P3 are recognized in the processing target frame data 12 of the frame identifier f2, and persons P1, P2 and P3 are recognized in the processing target frame data 12 of the frame identifier f3.

In step S102, the person importance score calculation unit 24 identifies the frame data in which each person is recognized. In the example of FIG. 5, the person P1 is recognized by the processing target frame data 12 of the frame identifiers f1, f2, and f3. The person P2 is recognized in each of the processing target frame data 12 of the frame identifiers f1 and f3. The person P3 is recognized in each of the processing target frame data 12 of the frame identifiers f2 and f3.

In step S103, the person importance score calculation unit 24 calculates the appearance time for each person. The appearance time is the time obtained by adding the time until the next processing target frame data of the processing target frame data 12 in which the person is recognized.

In the case of a movie, the appearance time of the person recognized in the processed frame data tends to be long for the main character. Also, in the case of location programs such as travel programs, the appearance time of the reporter becomes longer. For example, the appearance time of the store clerk who visited the location program and the person who passed the reporter during the location tends to be shorter than the appearance time of the reporter. is there.

As shown in FIG. 6, it is assumed that the processing target frame data 12 is frame data extracted from the video data 11 every time Δt. In this case, the appearance time of a certain person is the number of times the person is recognized in the processing target frame data 12 × Δt. Here, the case where the processing target frame data 12 is uniformly extracted from the video data 11 will be described, but similarly, when the processing target frame data 12 is randomly extracted, the processing target frame data next to the processing target frame data in which the person is recognized is also described. The appearance time of the person is calculated by adding the time until.

When the appearance time of each person is calculated, the person importance score calculation unit 24 calculates the character score of each person in step S104. The character score is the ratio of the appearance time of the person to the maximum value of the appearance time of each person in the video data 11. The character score Sp (Pi) of the person Pi (i = 1,2, ..., n: n is the number of people recognized in the processing target frame) is expressed by the equation (1). Here, Ti is the appearance time of the person Pi.
Sp (Pi) = Ti / max (Ti) ・ ・ ・ Equation (1)

Here, the appearance time T1 of the person P1 is 10 seconds, the appearance time T2 of the person P2 is 15 seconds, and the appearance time T3 of the person P3 is 20 seconds. The maximum value of the appearance time of each person in the video data 11 is 20 seconds of the appearance time T3 of the person P3. Therefore, the character score of person P1 is 10/20. The character score of person P2 is 15/20. The character score of person P3 is 20/20.

When the character score is calculated for each person, the person importance score calculation unit 24 repeats the process of calculating the person importance score for each processing target frame data 12. First, in step S105, the person importance score calculation unit 24 identifies the person with the largest face region in the processing target frame data 12. In step S106, the character score of the person having the largest face area is output as the person importance score of the processing target frame data 12.

When the person having the maximum area of the face area in a certain processing target frame data f is the person Px, the character score S1 (f) is calculated by, for example, the equation (2).
S1 (f) = Sp (Px) ・ ・ ・ Equation (2)

For example, when the face area of the person P2 is larger than the face area of the person P1 in the processing target frame data 12 of the frame identifier f1, the person importance score of the processing target frame data 12 of the frame identifier f1 is the character score of the person P2. It will be 15/20.

When the person importance score is calculated for each processing target frame data 12 and output to the score data 14, the person importance score calculation unit 24 ends the processing.

(Face area area score calculation unit)
The face area area score calculation unit 25 calculates the area of the face area of the person recognized by the processing target frame data 12 and the processing target frame data 12. The face area area score calculation unit 25 increases the processing target frame data 12 having the area of the face area close to the optimum area and decreases the processing target frame data 12 having the area of the face area far from the optimum area. Is calculated. Here, the optimum area is set in the condition data 13.

When a plurality of persons are recognized in the processing target frame data 12, the face area area score calculation unit 25 can consider several methods for calculating the face area area score. For example, the face area area score calculation unit 25 may calculate the face area area score based on the comparison between the area of the face area of the person having the largest area of the face area and the optimum area. The face area area score calculation unit 25 may calculate the face area area score based on the comparison between the average value of the area of the face area of each person having the area of the face area equal to or larger than a predetermined threshold value and the optimum area. .. The face area area score calculation unit 25 calculates the face area area score based on the comparison between the area of the smallest face area among the areas of the face area of each person having the area of the face area equal to or larger than a predetermined threshold and the optimum area. You may calculate.

When the thumbnail data 15 is output based only on the face area area score, the thumbnail output unit 29 outputs the frame data having a high face area area score as the thumbnail data 15.

The face area area score calculation process by the face area area score calculation unit 25 will be described with reference to FIG. 7. FIG. 7 is based on the area of the face area of the person having the largest face area in the processing target frame data 12 and the optimum area set in the condition data 13 when a plurality of people are recognized in the processing target frame data 12. An example of calculating the face area area score will be described. The face area area score calculation unit 25 repeats the process of calculating the face area area score for each of the processing target frame data 12.

In step S201, the face area area score calculation unit 25 recognizes the person shown in the processing target frame data 12. Similar to step S101 of FIG. 4, in person recognition, for example, an existing API (Application Programming Interface) such as Azure, AWS, or Google may be used. In step S202, the area of the face area of the person having the largest face area among the persons recognized in step S201 is calculated.

When the face area score of the frame identifier f is S2 (f) and the optimum area is Space_best, the face area score S2 (f) is calculated by, for example, the equation (3). Here, w is a parameter indicating how far from the optimum area Space_best is allowed. A default value may be given to w, or the condition data 13 may be set in advance. Space (Px, f) is the ratio of the number of pixels in the face area of the person having the largest area of the face area to the number of pixels in the frame data 12 to be processed.
S2 (f) = exp (-1 * w * abs (1-sqrt (Space (Px, f) / Space_best))) ・ ・ ・ Equation (3)

In step S203, the face area area score calculation unit 25 calculates the face area area score according to the proximity to the optimum area. For example, when the optimum area is 0.2, the face area score of the processing target frame data 12 in which the area of the face area calculated in step S202 is 0.2 is such that the area of the face area calculated in step S202 is 0. It is higher than the face area area score of the processing target frame data 12 in 5.5.

(Facial expression score calculation unit)
The facial expression score calculation unit 26 calculates the facial expression value for the type of facial expression for the person recognized by the processing target frame data 12. The facial expression score calculation unit 26 calculates the maximum value of each facial expression value as the facial expression score of the frame data with respect to the total of the facial expression values of each facial expression type of the person in the processing target frame data 12.

When a plurality of persons are recognized in the processing target frame data 12, the facial expression score calculation unit 26 can consider several methods for calculating the facial expression score. For example, the facial expression score calculation unit 26 may calculate the facial expression score based on the facial expression value of the person having the largest area of the face area. The facial expression score calculation unit 26 calculates the facial expression score of each person based on the facial expression value of each person having an area of the face area equal to or larger than a predetermined threshold, and calculates the calculated facial expression score as the average value, and the processed frame data. It may be a facial expression score of 12. The facial expression score calculation unit 26 may calculate the facial expression score based on the facial expression value of the person having the smallest face area of the face area of each person having the area of the face area equal to or larger than a predetermined threshold. good.

When the thumbnail data 15 is output based only on the facial expression score, the thumbnail output unit 29 outputs the frame data having a high facial expression score as the thumbnail data 15.

The facial expression score calculation process by the facial expression score calculation unit 26 will be described with reference to FIG. The facial expression score calculation unit 26 repeats the process of calculating the facial expression score for each processing target frame data 12.

In step S301, the facial expression score calculation unit 26 identifies the person having the largest face region in the processing target frame data 12. In step S302, the facial expression score calculation unit 26 calculates the facial expression value for each facial expression type for the person with the largest facial expression region identified in step S301. An existing API may be used in calculating the facial expression value.

As shown in FIG. 9 in the embodiment of the present invention, there are joy, anger, sadness, and surprise as types of facial expressions, and a case where a facial expression value is set for each of them will be described. Here, it is assumed that the facial expression value 5 of joy, the facial expression value 0 of anger, the facial expression value 0 of sadness, and the facial expression value 1 of surprise are calculated for the person P1 having the largest facial expression in the frame identifier f1.

In step S303, the facial expression score calculation unit 26 calculates the facial expression score from the facial expression values calculated in step S302. The facial expression score is the ratio of the maximum value of the facial expression value to the total of the facial expression values. The facial expression score S3 (f) of the processing target frame data 12 is calculated by, for example, the equation (4) using the facial expression value Sej (f) for each facial expression type of the person recognized in the processing target frame data 12.
S3 (f) = max (Sej (f)) / Σ (Sej (f)) ・ ・ ・ Equation (4)

When the frame identifier f1 shown in FIG. 9 is calculated as a joy facial expression value 5, an angry facial expression value 0, a sad facial expression value 0, and a surprise facial expression value 1, the total facial expression value is 6, which is the maximum facial expression. Since the value is 5, the facial expression score is 5/6.

When the facial expression score is calculated for each processing target frame data 12 and output to the score data 14, the facial expression score calculation unit 26 ends the processing.

In the embodiment of the present invention, the facial expression score describes the case where the processing target frame data having a high facial expression value of some kind of facial expression is selected as the thumbnail data 15, regardless of the type of facial expression, but the present invention is limited to this. Absent. For example, the facial expression type may be set in the condition data 13, and the frame data having a high facial expression value of the set facial expression type may be output as the thumbnail data 15.

(Volume score calculation unit)
The volume score calculation unit 27 calculates a volume score that increases for the frame data corresponding to the time when the volume of the video data 11 is high and decreases for the frame data corresponding to the time when the volume is low.

When the thumbnail data 15 is output based only on the volume score, the thumbnail output unit 29 outputs the frame data having a high volume score as the thumbnail data 15.

The volume score calculation process by the volume score calculation unit 27 will be described with reference to FIG.

First, in step S401, the volume score calculation unit 27 converts the transition of the volume of the video data 11 into a smooth transition. For example, the volume score calculation unit 27 calculates the convolution integral twice with Gaussian every minute time with respect to the time of the video data 11, and the volume score S4 (f) of the frame identifier f is calculated by the equation (5). The transition is calculated. V (f) in the equation (5) is an integral value corresponding to the time of the frame identifier f in the video data 11.
S4 (f) = 4 * (V (f) -0.5) ^ 2 ・ ・ ・ Equation (5)

In step S402, the volume score calculation unit 27 repeats the process of calculating the volume score for each processing target frame data 12. The volume score calculation unit 27 acquires a value with respect to the time of the processing target frame data 12 as the volume score of the processing target frame data 12 from the change in the volume calculated in step S401.

When the volume score is calculated for each processing target frame data 12 and output to the score data 14, the volume score calculation unit 27 ends the processing.

(Integrated score calculation department)
The integrated score calculation unit 28 added the frame data 12 to be processed by multiplying a plurality of scores including one or more of the person importance score, the face area area score, the facial expression score, and the volume score by weights. Calculate the integrated score.

For example, when calculating the integrated score by integrating the four scores of the person importance score, the face area area score, the facial expression score, and the volume score, it is calculated by the equation (6). In equation (6), Sall (f), S1 (f), S2 (f), S3 (f) and S4 (f) are the integrated score and person importance in the frame data 12 to be processed of the frame identifier f, respectively. Score, face area score, facial expression score and volume score. a1, a2, a3 and a4 are weights of the person importance score, the face area area score, the facial expression score and the volume score set in the condition data 13, respectively.

Sall (f) = Σai * Si (f) (i = 12.3.4) ・ ・ ・ Equation (6)

The integrated score calculation unit 28 calculates the integrated score based on one or more of the person importance score, the face area area score, the facial expression score, and the volume score, and two or more scores including the other scores. To do. The integrated score calculation unit 28 calculates the integrated score by multiplying each score by a weight and adding them.

When the thumbnail data 15 is output based on the integrated score, the thumbnail output unit 29 outputs the frame data having a high integrated score as the thumbnail data 15.

As described above, the thumbnail output device 1 according to the embodiment of the present invention can output thumbnail data 15 that reflects the user's intention.

For example, when the video data 11 is a movie, drama, or the like, it is required that the thumbnail data 15 shows the main cast. Therefore, the desired thumbnail data 15 is output by extracting the thumbnail data 15 based only on the person importance score or based on the integrated score calculated by setting the weight of the person importance score high. Can be done. On the other hand, in a landscape location program, it is required to extract the image at the moment when the landscape is taken rather than the person as a thumbnail. Therefore, for the video data 11 of the location program, the desired thumbnail data is output by extracting the thumbnail data 15 based on the integrated score calculated by setting the weight of the person importance score lower than the other scores. can do.

Further, even when a person is shown in the video data 11, the index for selecting the thumbnail data 15 desired by the user may be different. For example, in movies, dramas, etc., thumbnail data 15 showing the main cast is required, while when a reporter introduces a product in a variety show, the thumbnail data 15 requires that the product appears large and the reporter appears small. Be done. Therefore, for the video data 11 of such a variety show or the like, it is desired to extract the thumbnail data 15 based on the integrated score calculated by setting the weight of the person importance score lower than the other scores. Thumbnail data 15 can be output.

When making a thumbnail of a program that is closely related to one person, such as a documentary program, not only the person to be closely related is reflected in the thumbnail data, but also that person's facial expression is smiling or angry. It is required to include facial expressions suitable for the program. Therefore, the desired thumbnail data 15 can be output by extracting the thumbnail data 15 based only on the facial expression score or based on the integrated score calculated by setting the weight of the facial expression score high.

In a variety show, not only the facial expression of the cast but also the moment when the customer is excited is the highlight of the program, so it is required to extract the frame data at the moment when the customer is excited as thumbnail data 15. Therefore, the desired thumbnail data 15 can be output by extracting the thumbnail data 15 based only on the volume score or based on the integrated score calculated by setting the weight of the volume score high. On the other hand, in the location program, it is required to extract the frame data at the moment when the scenery is photographed quietly as the thumbnail data 15 rather than the moment when the reporter is talking. Therefore, the desired thumbnail data 15 is output by extracting the thumbnail data 15 based on the integrated score calculated by setting the weight of the volume score lower than the other scores for the video data 11 of the location program. be able to.

As described above, the thumbnail output device 1 according to the embodiment of the present invention can output the thumbnail data 15 of the video data 11 reflecting the intention of the user.

(Other embodiments)
As mentioned above, although described by embodiments of the invention, the statements and drawings that form part of this disclosure should not be understood to limit the invention. This disclosure reveals to those skilled in the art various alternative embodiments, examples and operational techniques.

For example, the thumbnail output device described in the embodiment of the present invention may be configured on one hardware as shown in FIG. 1, or may be configured on a plurality of hardware according to its function and the number of processes. May be done. Further, it may be realized on an existing information processing system.

It goes without saying that the present invention includes various embodiments not described here. Therefore, the technical scope of the present invention is defined only by the matters specifying the invention relating to the reasonable claims from the above description.

1 Thumbnail output device 10 Storage device 11 Video data 12 Processing target frame data 13 Condition data 14 Score data 15 Thumbnail data 20 Processing device 21 Processing target frame extraction unit 22 Condition data acquisition unit 23 Score calculation unit 24 Person importance score calculation unit 25 Face area area score calculation unit 26 Facial expression score calculation unit 27 Volume score calculation unit 28 Integrated score calculation unit 29 Thumbnail output unit

Claims (11)

A thumbnail output device that outputs thumbnail data of video data.
For each person recognized by the frame data constituting the video data, the appearance time is calculated by adding the time until the frame data to be processed next to the frame data recognized by the person.
The character score, which is the ratio of the character's appearance time to the maximum value of each person's appearance time, is calculated for each person, and at the same time.
A person importance score calculation unit that calculates the character score of the person having the largest face area in the frame data as the person importance score of the frame data.
A thumbnail output device including a thumbnail output unit that outputs frame data having a high character score as thumbnail data. The area of the face area of the person recognized by the frame data constituting the video data is calculated.
A face area area score calculation unit for calculating a face area area score that is high for frame data having the area of the face area close to the optimum area and low for frame data having the area of the face area far from the optimum area is further provided.
The third aspect of claim 1, wherein the thumbnail output unit outputs frame data having a high integrated score , which is obtained by multiplying the person importance score and the face area area score by weights and adding them , as thumbnail data. Thumbnail output device. For the person recognized by the frame data constituting the video data, the facial expression value for the facial expression type is calculated.
Further provided with a facial expression score calculation unit that calculates the ratio of the maximum value of each facial expression value to the total facial expression value of each facial expression type of the person in the frame data as the facial expression score of the frame data.
The thumbnail according to claim 1, wherein the thumbnail output unit outputs frame data having a high integrated score obtained by multiplying the person importance score and the facial expression score by weights, respectively, and adding them as thumbnail data. Output device. A volume score calculation unit for calculating a volume score that increases the frame data corresponding to the time when the volume of the video data is high and decreases the frame data corresponding to the time when the volume is low is further provided.
The thumbnail according to claim 1, wherein the thumbnail output unit outputs frame data having a high integrated score obtained by multiplying the person importance score and the volume score by weights, respectively, and adding them as thumbnail data. Output device. A thumbnail output device that outputs thumbnail data of video data.
Of the frame data of the video data, the person importance score according to claim 1, the face area area score according to claim 2, the facial expression score according to claim 3, and the volume score according to claim 4. An integrated score calculation unit that calculates an integrated score by multiplying multiple scores including one or more by weights and adding them.
A thumbnail output device including a thumbnail output unit that outputs frame data having a high integrated score as thumbnail data. It is a thumbnail output method that outputs thumbnail data of video data.
For each person recognized by the frame data constituting the video data, the computer calculates the appearance time by adding the time until the frame data to be processed next to the frame data recognized by the person.
A step to calculate the character score, which is the ratio of the character's appearance time to the maximum value of each person's appearance time, for each person, and
A step in which the computer calculates the character score of the person having the largest face area in the frame data as the person importance score of the frame data.
A thumbnail output method, wherein the computer includes a step of outputting frame data having a high character score as thumbnail data. The computer calculates the area of the face area of the person recognized by the frame data constituting the video data.
A step of calculating a face area area score that is high for frame data having the area of the face area close to the optimum area and low for frame data having the area of the face area far from the optimum area.
6. The computer further comprises a step of outputting as thumbnail data frame data having a high integrated score obtained by multiplying the person importance score and the face area area score by weights, respectively. Thumbnail output method described in. The computer calculates a facial expression value for a type of facial expression for a person recognized by the frame data constituting the video data.
A step of calculating the ratio of the maximum value of each facial expression value to the total of the facial expression values of each facial expression type of the person in the frame data as the facial expression score of the frame data.
The sixth aspect of claim 6 is characterized in that the computer further includes a step of outputting as thumbnail data frame data having a high integrated score obtained by multiplying the person importance score and the facial expression score by weights and adding them. Thumbnail output method. A step in which the computer calculates a volume score that increases for frame data corresponding to a time when the volume of the video data is high and decreases for frame data corresponding to a time when the volume is low.
The sixth aspect of claim 6 is characterized in that the computer further includes a step of outputting frame data having a high integrated score , which is obtained by multiplying the person importance score and the volume score by a weight, respectively , as thumbnail data. Thumbnail output method. It is a thumbnail output method that outputs thumbnail data of video data.
The computer uses the frame data of the video data as the person importance score according to claim 6, the face area area score according to claim 7, the facial expression score according to claim 8, and the volume score according to claim 9. A step of calculating an integrated score by multiplying a plurality of scores including one or more of them by weights and adding them.
A thumbnail output method, wherein the computer includes a step of outputting frame data having a high integrated score as thumbnail data. A thumbnail output program for causing a computer to function as the thumbnail output device according to any one of claims 1 to 5.

Images