JP7441107B2 - Learning device, representative image extraction device and program - Google Patents

Description

The present invention relates to a learning device, a representative image extraction device, and a program used in the field of video analysis for extracting representative images from videos.

BACKGROUND ART Conventionally, broadcasting stations have been enhancing their program homepages (homepages) with the aim of improving viewer contact rates. Program homepages often display a plurality of representative images extracted from program videos in order to give viewers a rough idea of the program content.

However, a lot of effort is required to extract representative images from program videos. For this reason, methods have been proposed for automatically extracting representative images from program videos (see, for example, Patent Document 1 and Non-Patent Document 1).

For example, the method disclosed in Patent Document 1 calculates the degree of association between images based on the results of identifying human faces, scenes, and objects from a set of images, GPS (Global Positioning System) information, and similarity. , a representative image is extracted based on the degree of association and the date of photography.

Furthermore, the method disclosed in Non-Patent Document 1 uses a GoogleLeNet neural network that has been trained in advance to determine the level of artistic quality of an image.

Patent No. 6149015

Xin Jin, et al., “ILGNet: Inception modules with connected local and global features for efficient image aesthetic quality classification using domain adaptation.” IET Computer Vision 13.2 (2018): 206-212.

However, when extracting a representative image from a program video, the method disclosed in Patent Document 1 described above requires special information such as GPS information and shooting date. Furthermore, it focuses only on some elements such as objects and faces included in the image, and does not take into account the artistic quality of the image as a whole. Furthermore, the method of Non-Patent Document 1 mentioned above does not take into account program production know-how.

For this reason, program homepages created using representative images are not necessarily effective, and there have been problems in that they may not be able to effectively present the program content to viewers. .

SUMMARY OF THE INVENTION The present invention has been made to solve the above problems, and its purpose is to provide a learning device, a representative image extraction device, and a program that can extract representative images from program videos in consideration of program production know-how. It's about doing.

In order to solve the problem, a learning device according to a first aspect of the present invention is a learning device for learning a neural network, in which a frame image obtained by sampling a learning program video is taken as a program image, and a plurality of frames added to the program image are provided. The score of any one of the stages is set as a first correct score, the score of any one of the plural stages given to a predetermined image is set as a second correct score, and the neural network is set to the program image and As a model in which the predetermined images are input alternately and a one-dimensional score is output, program learning data consisting of the program image and the first correct score, and predetermined learning data consisting of the predetermined image and the second correct score. and a learning unit that reads the program learning data and the predetermined learning data from the memory and learns the neural network using the program learning data and the predetermined learning data, using the neural network to calculate a one-dimensional score of the program image as a first score from the program image included in the program learning data; a neural network unit that calculates a one-dimensional score of the predetermined image as a second score from the predetermined image, and the first score calculated by the neural network unit and the first correct score included in the program learning data. an error calculation unit that calculates an error between the second score and the second correct score included in the predetermined learning data as a second error; The method further includes a parameter updating section that updates parameters of the neural network so that the sum of the first error and the second error calculated by the section becomes smaller.

The learning device according to claim 2 is the learning device according to claim 1, further comprising a program learning data generation unit that generates the program learning data, and wherein the program learning data generation unit a sampling processing unit that samples the program image into the program image; a download processing unit that accesses the URL of the home page of the program corresponding to the learning program video and downloads a still image of the program; a similarity calculation unit that calculates the similarity between the program image and the still image downloaded by the download processing unit; and a similarity calculation unit that calculates the similarity between the program image and the still image downloaded by the download processing unit; and a first correct score assigning unit that assigns the first correct score to the program image and stores the program learning data including the program image and the first correct score in the memory.

The learning device according to claim 3 is the learning device according to claim 2, further comprising a predetermined learning data generation section that generates the predetermined learning data, and wherein the predetermined learning data generation section generates the predetermined image and the predetermined learning data. By inputting open data consisting of a label of one of a plurality of stages assigned to an image in advance, and converting the label into the second correct answer score, the second correct score is applied to the predetermined image. The present invention is characterized by comprising a second correct score assigning unit that assigns a correct score and stores the predetermined learning data including the predetermined image and the second correct score in the memory.

The learning device according to claim 4 is the learning device according to any one of claims 1 to 3, wherein the number of the program learning data is A (A is a positive integer), and the number of the predetermined learning data is A (A is a positive integer). When the number is B (B is a positive integer), A<B, and the result of subtracting A from B is (B-A), the learning section obtains A pieces of the program learning data and the program learning data. (B-A) pieces of data that are insufficient in the program learning data with respect to predetermined learning data, the program learning data supplemented using any or all of the A pieces of the program learning data, and B The method is characterized in that the neural network is trained using the predetermined learning data.

Further, a representative image extraction device according to a fifth aspect of the present invention is a representative image extraction device that extracts a representative image from a program video, and further includes a sampling processing section that samples the program video into a frame image and outputs the frame image as a program image. A score calculation that calculates a one-dimensional score of the program image from the program image output by the sampling processing unit using a neural network trained by the learning device according to any one of claims 1 to 4. and a selection unit that selects the representative image from all the program images in which the program video is sampled and output by the sampling processing unit, based on the score calculated by the score calculation unit. It is characterized by being equipped.

Furthermore, the program according to claim 6 causes a computer to function as the learning device according to any one of claims 1 to 4.

Moreover, the program according to claim 7 is characterized in that it causes a computer to function as the representative image extraction device according to claim 5.

As described above, according to the present invention, a representative image can be extracted from a program video in consideration of program production know-how.

1 is a block diagram showing a configuration example of a learning device according to an embodiment of the present invention. FIG. FIG. 2 is a block diagram showing a configuration example of a program learning data generation section. 7 is a flowchart illustrating an example of processing by a program learning data generation unit. It is a flowchart which shows another example of a structure of a program learning data generation part. FIG. 2 is a block diagram showing a configuration example of an artistic learning data generation section. FIG. 2 is a block diagram showing a configuration example of a learning section. 7 is a flowchart illustrating an example of processing by a learning unit. FIG. 2 is a block diagram showing a configuration example of a NN section. FIG. 3 is a diagram illustrating a specific example of the configuration of the NN section. 1 is a block diagram showing a configuration example of a representative image extraction device according to an embodiment of the present invention. FIG. 1 is a diagram illustrating a program homepage creation system of a first embodiment using a representative image extraction device. It is a figure explaining the program DVD sales HP creation system of 2nd Example using a representative image extraction device. It is a figure explaining the effect of learning processing in an embodiment of the present invention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments for carrying out the present invention will be described in detail using the drawings.
[Learning device]
First, a learning device according to an embodiment of the present invention will be described. FIG. 1 is a block diagram showing a configuration example of a learning device according to an embodiment of the present invention.

This learning device 1 includes a program learning data generation section 10, memories 11 and 13, an artistic learning data generation section 12, and a learning section 14. The learning device 1 is a representative image extracting device so that a representative image extracting device 2 (described later) can extract representative images from program videos in consideration of program production know-how by using learning program videos, artistic evaluation open data, etc. 2 is a device for learning the neural network used.

The program learning data generation unit 10 inputs a learning program video and a URL (Uniform Resource Locator) of a program HP corresponding to the program of the learning program video. Then, the program learning data generation unit 10 generates a plurality of frame images obtained by accessing the URL of the program HP for each of the plurality of frame images (hereinafter referred to as "program images") obtained by sampling the learning program video. The degree of similarity between each of the still images is calculated.

The program learning data generation unit 10 assigns a correct score to the program image based on the degree of similarity, and stores the program learning data including the program image and the correct score in the memory 11.

As a result, program learning data consisting of program images and correct scores is stored in the memory 11 for all program images obtained by sampling the learning program video.

Here, the still images posted on the program homepage can be said to be representative screens selected from the program video by the program production staff utilizing their know-how. Therefore, the similarity between the program image and the still image is a value that reflects the know-how of the program production staff, and as a result, the correct score for the program image is a value that reflects the know-how of the program production staff. .

The artistic learning data generation unit 12 sequentially inputs artistic evaluation open data. This artistic evaluation open data is data that anyone can obtain and use without any restrictions, and images are given correct labels evaluated from an artistic perspective. There is. Artistic evaluation open data consists of images (hereinafter referred to as ``artistic evaluation images'') and correct answer labels (indicating high or low artistic quality) that reflect the multi-level evaluation given in advance to the artistic evaluation images. label).

For each piece of input artistic evaluation open data, the artistic learning data generation unit 12 converts the correct label included in the artistic evaluation open data into a correct answer score according to a predetermined rule. Then, the artistic learning data generation unit 12 stores artistic learning data consisting of the artistic evaluation image and the correct score in the memory 13. The correct answer label is a label indicating high or low as described above, and the correct answer score is a numerical value.

As a result, the memory 13 stores artistic learning data consisting of artistic evaluation images and correct scores for a plurality of artistic evaluation open data.

The learning unit 14 includes a neural network to be learned. The learning unit 14 reads program learning data consisting of program images and correct scores from the memory 11, and reads artistic learning data consisting of artistic evaluation images and correct scores from the memory 13. The learning unit 14 then uses the program learning data and the artistic learning data to learn the neural network. This neural network is a model in which program images and artistic evaluation images are alternately input, and a one-dimensional score (importance) is output.

As a result, optimal parameters (weighting coefficients, etc.) for use in the neural network can be obtained. This parameter is a value that reflects the know-how of the program production staff, and is used in the neural network provided in the representative image extraction device 2 shown in FIG. 10, which will be described later.

(Program learning data generation unit 10)
Next, the program learning data generation section 10 shown in FIG. 1 will be explained in detail. FIG. 2 is a block diagram showing a configuration example of the program learning data generation section 10, and FIG. 3 is a flowchart showing an example of processing of the program learning data generation section 10. The program learning data generation section 10 includes a sampling processing section 20, a download processing section 21, a similarity calculation section 22, and a correct answer scoring section 23.

The program learning data generation unit 10 inputs the learning program video stored in a hard disk recorder or the like and the URL of the program HP corresponding to the program of the learning program video (step S301). The sampling processing section 20 inputs the learning program video, and the download processing section 21 inputs the URL of the corresponding program homepage.

The sampling processing unit 20 samples program images, which are frame images, from the learning program video at regular intervals (step S302). Let all sampled program images be P ₁ , . . . , P _N . N is an integer of 2 or more. The sampling processing section 20 outputs the program images P ₁ , . . . , P _N to the similarity calculation section 22 .

The download processing unit 21 accesses the URL of the program HP and downloads all still images posted on the program HP (step S303). Let all downloaded still images be P' ₁ , . . . , P' _M. M is an integer of 2 or more. The download processing unit 21 outputs the still images P′ ₁ , . . . , P′ _M to the similarity calculation unit 22 .

The similarity calculation unit 22 receives program images P ₁ , . . . , P _N from the sampling processing unit 20 and still _images P′ ₁ , . Then, the similarity calculation unit 22 calculates the similarity S n, _m between the program image P _n and the still image P' _m (step S304). n=1,...,N, and m=1,...,M.

The similarity calculating unit 22 outputs the program image P _n and the similarity S _n,m (S _n,1 , . . . , S _n,M ) of the program image P _n to the correct score assigning unit 23 .

The correct score assigning unit 23 inputs the program image P _n and the similarity S _n,m (S _n,1 , . . . , S _n,M ) of the program image P _n from the similarity calculation unit 22 . Then, the correct score assigning unit 23 obtains the maximum value B=max _m S _n ,m of the degrees of similarity S _n,1 , . . . , S _{n, M} for the program image P n (step S305).

The correct score assigning unit 23 determines whether the maximum value B is greater than or equal to a preset threshold (step S306). If it is determined in step S306 that the maximum value B is greater than or equal to the threshold (step S306: Y), the correct score assigning unit 23 assigns a correct score (=1) of a positive example to the program image P _n ( Step S307).

On the other hand, if the correct score assigning unit 23 determines in step S306 that the maximum value B is not equal to or greater than the threshold (step S306: N), it assigns a negative example correct score (=0) to the program image P _n . (Step S308).

The correct score assigning unit 23 assigns a two-level correct score (positive example (=1) or negative example (=0)) to the program image P _n in a score range of 0 to 1. However, three or more levels of correct scores may be given. For example, in the case of a three-level correct score, the correct score assigning unit 23 thresholds the maximum value B to give the program image P _n a three-level correct score (for example, 0.0, 0.5, 1). .0).

In this case, the stages of the correct score do not necessarily have to be at equal intervals in the range of 0 to 1, and may be, for example, 0.0, 0.7, 1.0, as long as they are at appropriate intervals. Further, the correct score is set in stages in the same range as the correct score of the artistic evaluation image in FIG. 5, which will be described later, for example, in the range of 0 to 1.

Further, the correct score assigning unit 23 may assign the similarity input from the similarity calculating unit 22 to the program image as the correct score. The range of similarity in this case is 0-1.

The correct score assigning unit 23 moves from step S307 or S308 and stores the program learning data consisting of the program image and the correct score in the memory 11 (step S309). The processes of steps S304 to S309 are performed on N program images P _n (n=1, . . . , N), and N program learning data are stored in the memory 11.

As a result, a correct score that reflects the degree of similarity S _{n ,m} between the program image P n and the still image downloaded from the program HP is assigned, and the program learning data is stored in the memory 11 . The higher the similarity S _n,m (the more appropriate the image is to be a representative image), the correct score will be 1 or a value close to 1, and the lower the similarity S _n,m (the less appropriate the image is to be a representative image), the correct score will be. is a value of 0 or a step close to 0.

FIG. 4 is a flowchart showing another example of the configuration of the program learning data generation section 10. The program learning data generation section 10 includes a sampling processing section 20 and a correct answer scoring section 24. This program learning data generation section 10 receives only the learning program video and does not input the URL of the program homepage.

The sampling processing unit 20 inputs the learning program video, performs the same processing as the sampling processing unit 20 shown in FIG. 2, and outputs the program image to the correct score assigning unit 24.

The correct score assigning unit 24 inputs the program image from the sampling processing unit 20 and displays the program image on a display device (not shown). A user who is a program production staff member evaluates a program image displayed on a display device and determines a correct score for the program image. For example, in the case of a two-level correct score, the correct score (=1) is determined when the evaluation of the program image is high, and the correct score (=0) is determined when the evaluation of the program image is low.

The correct score assigning unit 24 inputs the correct score for the program image according to the operation of the program production staff. Then, the correct score assigning unit 24 generates program learning data consisting of the program image and the correct score by assigning a correct score to the program image, and stores this in the memory 11 .

As a result, a correct score that reflects the know-how of the program production staff is assigned to the program image, and the program learning data is stored in the memory 11. The higher the evaluation of the program image (the more suitable it is as a representative image), the correct score will be 1 or a value close to 1, and the lower the evaluation of the program image (the more unsuitable it is as a representative image), the correct score will be 0 or 0. The value is close to .

(Artistic learning data generation unit 12)
Next, the artistic learning data generation section 12 shown in FIG. 1 will be explained in detail. FIG. 5 is a block diagram showing an example of the configuration of the artistic learning data generation section 12. As shown in FIG. The artistic learning data generating section 12 includes a correct answer scoring section 25.

The correct score assigning unit 25 sequentially inputs artistic evaluation open data consisting of artistic evaluation images and correct answer labels.

The correct score assigning unit 25 converts the correct label included in the artistic evaluation open data into a correct score according to a predetermined rule for each piece of artistic evaluation open data. A correct score is assigned to the evaluation image. Then, the correct score assigning unit 25 stores the artistic learning data consisting of the artistic evaluation image and the correct scores in the memory 13.

Generally, the correct label cannot be used for learning processing because it is not digitized. Therefore, the correct answer score giving unit 25 converts the correct answer label into a correct answer score expressed as a numerical value. Thereby, the correct answer score, which reflects the correct answer label and is expressed numerically, can be used for the learning process.

The predetermined rule is a preset rule that converts a p-level correct answer label into a q-level correct answer score. p and q are integers of 2 or more, and may be p≠q or p=q.

According to a predetermined rule, for example, three-level correct labels "great," "good," and "bad" are converted into two-level correct scores. The three levels of correct labels are "great" > "good" > "bad" in order of artistic quality. The correct label (=great) is converted into a correct score (=1), the correct label (=good) is converted into a correct score (=1), and the correct label (=bad) is converted into a correct score (=0).

Further, according to a predetermined rule, for example, a three-level correct answer label is converted into a three-level correct answer score. The correct label (=great) becomes the correct score (=1.0), the correct label (=good) becomes the correct score (=0.5), and the correct label (=bad) becomes the correct score (=0.0). converted.

Note that the stages of the correct score do not necessarily have to be at equal intervals within the range of 0 to 1, and may be, for example, 0.0, 0.7, 1.0, as long as they are at appropriate intervals. Further, the correct score may have more than three levels, and the level is set in the same range as the correct score of the program image described above, for example, in the range of 0 to 1.

Instead of converting the correct label into a correct score, the correct score assigning unit 25 assigns the correct score to the artistic evaluation image by inputting the correct score determined by the program production staff in accordance with the program production staff's operation. It may be given. Similar to the above, the stages of correct scores do not necessarily have to be equally spaced.

Specifically, the correct score assigning unit 25 displays the artistic evaluation image and the correct answer label included in the artistic evaluation open data on a display device (not shown). A user who is a program production staff member evaluates the artistic evaluation image with reference to the correct answer label displayed on the display device, and determines the correct answer score for the artistic evaluation image.

The correct score assigning unit 25 inputs the correct score for the artistic evaluation image according to the operation of the program production staff. Then, the correct score assigning unit 25 generates artistic learning data consisting of the artistic evaluation image and the correct score by assigning a correct score to the artistic evaluation image, and stores this in the memory 13 .

As a result, a correct score reflecting the know-how of the program production staff is assigned to the artistic evaluation image, and the artistic learning data is stored in the memory 13. The higher the evaluation for the artistic evaluation image (the more suitable it is as a representative image), the correct score will be 1 or a value close to 1, and the lower the evaluation for the artistic evaluation image (the more unsuitable it is for a representative image), the correct answer score will be. is a value of 0 or a step close to 0.

(Learning part 14)
Next, the learning section 14 shown in FIG. 1 will be explained in detail. FIG. 6 is a block diagram showing a configuration example of the learning unit 14, and FIG. 7 is a flowchart showing an example of processing by the learning unit 14.

The learning section 14 includes a switching section 30, a neural network (NN) section 31, an error calculation section 32, and a parameter updating section 33. The learning unit 14 performs the processes of steps S701 to S706 for each set of program learning data and artistic learning data until the end condition is satisfied in the process of step S707.

The switching unit 30 receives a switching signal indicating program learning data or artistic learning data from the parameter updating unit 33. Then, when the switching signal indicates program learning data, the switching unit 30 reads program learning data consisting of a program image and a correct score from the memory 11. On the other hand, when the switching signal indicates artistic learning data, the switching unit 30 reads artistic learning data consisting of an artistic evaluation image and a correct score from the memory 13 (step S701).

As a result, new program learning data is read out from the memory 11 every time a switching signal indicating program learning data is input, and new program learning data is read out from the memory 13 every time a switching signal indicating artistic learning data is input. Artistic learning data is read out.

When the switching signal indicates program learning data, the switching unit 30 outputs the program image included in the program learning data to the NN unit 31, and also outputs the correct score corresponding to the program image to the error calculation unit 32.

On the other hand, when the switching signal indicates the artistic learning data, the switching unit 30 outputs the artistic evaluation image included in the artistic learning data to the NN unit 31, and also outputs the correct score corresponding to the artistic evaluation image. It is output to the error calculation section 32.

The NN section 31 receives the tensor of the program image or the artistic evaluation image from the switching section 30 . Then, the NN section 31 calculates a one-dimensional score from the program image or the artistic evaluation image using a neural network whose parameters are set by the parameter update section 33, and outputs the score to the error calculation section 32.

The error calculation section 32 receives the score from the NN section 31 and the correct score from the switching section 30, calculates an error between the two, and outputs the result to the parameter update section 33. For example, as a function for calculating the error, a function such as MSE (mean square error) that outputs a larger value as the error becomes larger is used.

Specifically, when the tensor of the program image is input from the switching unit 30, the NN unit 31 calculates a score from the program image using a neural network (step S702). This neural network is a feature extraction NN 40 and a score calculation NN 41 shown in FIG. 8, which will be described later.

Then, the error calculation unit 32 calculates the error between the score of the program image and the correct score of the program image included in the program learning data (step S703).

On the other hand, when the tensor of the artistic evaluation image is input from the switching section 30, the NN section 31 calculates a score from the artistic evaluation image using a neural network (step S704).

Then, the error calculation unit 32 calculates the error between the score of the artistic evaluation image and the correct score of the artistic evaluation image included in the artistic learning data (step S705).

The parameter update unit 33 inputs the error of the program image and the error of the artistic evaluation image from the error calculation unit 32, and updates the held parameters so that the sum of these errors becomes small (step S706). Then, the parameter update section 33 sets the updated parameters in the NN section 31.

Here, it is assumed that the parameter update section 33 holds the parameters set in the NN section 31.

Note that the parameter updating unit 33 uses a general neural network optimization method such as Adam, SGD (Stochastic Gradient Descent), and error backpropagation learning method (Backpropagation) to update the parameters.

Further, the parameter updating unit 33 may update the parameters for each set of a predetermined number of sets (for example, 30 sets) of a program image and an artistic evaluation image. Specifically, the parameter updating unit 33 inputs a predetermined number of sets of errors, calculates the sum of the predetermined number of sets of errors, and updates the parameters so that the sum of the errors becomes smaller.

When the parameter update unit 33 receives the error of the program image from the error calculation unit 32, it outputs a switching signal indicating artistic learning data to the switching unit 30 in order to input the error of the artistic evaluation image next.

On the other hand, when the error of the artistic evaluation image is input from the error calculation unit 32, the parameter update unit 33 outputs a switching signal indicating program learning data to the switching unit 30 in order to input the error of the program image next. .

The parameter update unit 33 moves from step S706 and determines whether the parameter update termination condition is satisfied (step S707).

If the parameter update unit 33 determines in step S707 that the end condition is not satisfied (step S707: N), the parameter update unit 33 moves to step S701 and updates the next set of program learning data and artistic learning data in steps S701 to S706. Process. That is, the processes of steps S701 to S706 are performed for each set of program learning data and artistic learning data until the termination condition is satisfied.

On the other hand, if the parameter updating unit 33 determines in step S707 that the termination condition is satisfied (step S707: Y), it outputs the parameters updated in the process of step S706 as optimal parameters (step S708). The optimal parameters output by the parameter updating section 33 are set in the neural network of the score calculation section 51 provided in the representative image extraction device 2 shown in FIG. 10, which will be described later.

Here, the termination conditions in step S707 include, for example, whether the parameters have been updated a preset number of times, and whether the amount of parameter updates is smaller than a preset threshold.

(NN section 31)
Next, the NN unit 31 shown in FIG. 6 will be explained in detail. FIG. 8 is a block diagram showing an example of the configuration of the NN section 31. As shown in FIG. This NN unit 31 is configured to include a feature extraction NN 40 and a score calculation NN 41.

The feature extraction NN 40 receives a program image or an artistic evaluation image as input data, and calculates output data of a 1024-dimensional image feature vector by calculation of a neural network whose parameters are set by the parameter updating unit 33.

The score calculation NN 41 uses the 1024-dimensional image feature vector obtained by the feature extraction NN 40 as input data, and calculates one-dimensional score output data through neural network calculations whose parameters are set by the parameter update unit 33.

FIG. 9 is a diagram illustrating a specific example of the configuration of the NN section 31, and shows the configuration of the NN section 31 shown in FIG. 8 in detail. In FIG. 9, "Conv" indicates a convolution layer, "MaxPool" indicates a pooling layer that extracts the maximum value, and "LocalResponseNorm" indicates a normalization layer. Furthermore, "Inception Module" is a technology included in "GoogLeNet" and indicates a convolution layer and a pooling layer. Furthermore, "AveragePool" indicates a pooling layer that calculates the average value, "FC" indicates a fully connected layer, "Concat" indicates a connected layer, and "Sigmoid" indicates a layer using a sigmoid function. Furthermore, "Kernel" indicates the filter size, and "dim" indicates the number of dimensions.

The feature extraction NN 40 is configured by each layer from the pooling layer α1 of “Conv” where program images or artistic evaluation images are input to the connection layer α2 of “Concat” where a 1024-dimensional image feature vector is output. .

In addition, each layer from the fully connected layer α3 of “FC” where a 1024-dimensional image feature vector is input to the output layer of the sigmoid function α4 of “Sigmoid” where a one-dimensional score is output, the score calculation NN41 is configured.

In this way, the NN unit 31 uses the feature extraction NN 40 that calculates the 1024-dimensional image feature vector of the image from the program image or the artistic evaluation image, and the NN 40 that calculates the 1-dimensional score from the 1024-dimensional image feature vector of the image. It is composed of a score calculation NN 41 to be calculated.

The NN unit 31 calculates a one-dimensional score regardless of the number of correct score stages assigned to the program image or artistic evaluation image. That is, as the NN section 31, there is no need to prepare different neural networks depending on the number of stages of the program image or artistic evaluation image, and it is sufficient to prepare a neural network with a fixed configuration that does not depend on the number of stages.

As described above, according to the learning device 1 according to the embodiment of the present invention, the program learning data generation unit 10 accesses the URL of the program homepage and obtains the program image obtained by sampling the learning program video. The degree of similarity between the still image and the captured still image is calculated. Then, the program learning data generation unit 10 assigns a correct score to the program image based on the degree of similarity, and generates program learning data including the program image and the correct score.

The artistic learning data generation unit 12 converts the correct label included in the artistic evaluation open data into a correct score according to a predetermined rule, and generates artistic learning data including the artistic evaluation image and the correct score.

The learning unit 14 uses the program learning data and the artistic learning data to learn the neural network. Specifically, the NN section 31 uses a neural network to calculate a one-dimensional score from the program image included in the program learning data, and the error calculation section 32 calculates the score of the program image and the program image included in the program learning data. Calculate the error between the correct score and the correct score. The NN unit 31 also calculates a one-dimensional score for the artistic evaluation image, and the error calculation unit 32 calculates the error between the score of the artistic evaluation image and the correct score included in the artistic learning data. do.

The parameter updating unit 33 updates the parameters of the neural network so that the sum of the error in the program image and the error in the artistic evaluation image becomes small, and outputs the parameters when a predetermined termination condition is satisfied as the optimal parameters. .

Here, the still images on the program homepage are images generated by making use of the know-how of the program production staff, so the correct score of the program image calculated from the similarity between the program image and the still image is based on the know-how of the program production staff. The score takes into account the

As a result, the neural network trained using the correct scores of program images also takes into account program production know-how. Therefore, by using the neural network learned by the learning device 1, the representative image extraction device 2, which will be described later, can extract a representative image from the program video taking into account the know-how of program production. Furthermore, since the representative image can be extracted without using special data other than the program video, the processing load can be reduced. Further, when creating a program homepage using representative images, the amount of work can be significantly reduced.

Furthermore, even if the number of program learning data stored in the memory 11 is smaller than the artistic learning data stored in the memory 13, by repeatedly using the same program learning data, the shortage of program learning data can be compensated for. Can be replenished. Thereby, the same number of program learning data as the artistic learning data can be prepared, and the neural network can be trained using the same number of program learning data and artistic learning data.

Specifically, the number of program learning data stored in the memory 11 is A (A is a positive integer), the number of artistic learning data stored in the memory 13 is B (B is a positive integer), If A<B, the difference between the program learning data and the artistic learning data, that is, the result of subtracting A from B is set as (BA). The (B-A) pieces of program learning data corresponding to the difference between the program learning data and the artistic learning data (the shortfall of the program learning data with respect to the artistic learning data) are obtained by combining any or all of the A pieces of program learning data. Replenishes with use. In other words, the learning unit 14 performs program learning that is supplemented by using any or all of the A program learning data and the missing (B-A) data. A neural network is trained using the data and B pieces of artistic training data. In this case, the total number of A program learning data and the missing (B−A) program learning data is B, which is the same as the number of artistic learning data.

For example, assume that the number of program learning data is A=6,000 and the number of artistic learning data is B=10,000. In this case, the shortage of program learning data (B−A)=4,000 pieces of program learning data with respect to the artistic learning data is supplemented by using a part of the program learning data of A=6,000 pieces. As a result, the missing 4,000 pieces of program learning data can be supplemented using the original A=6,000 pieces of program learning data. The learning unit 14 trains the neural network using the original A=6,000 program learning data, the missing 4,000 program learning data, and 10,000 artistic learning data.

Further, assume that the number of program learning data is A=6,000 and the number of artistic learning data is B=25,000. In this case, the lack of program learning data (B-A) = 19,000 pieces of program learning data for the artistic learning data is obtained by using A = 6,000 pieces of program learning data three times, and the remaining For 1,000 pieces of program learning data, part of A=6,000 pieces of program learning data is used. As a result, the missing 19,000 pieces of program learning data can be supplemented using the original A=6,000 pieces of program learning data. The learning unit 14 trains the neural network using the original A=6,000 program learning data, the missing 19,000 program learning data, and 25,000 artistic learning data.

Furthermore, even if the number of stages of the correct score of the program image and the number of stages of the correct score of the artistic evaluation image are the same or different, the NN unit 31 provides a unified one-dimensional score for the program image and the artistic evaluation image. is calculated. In other words, the neural network unit 31 can use a neural network with a fixed configuration that does not depend on the number of stages of program images and artistic evaluation images, so there is no need to prepare different neural networks in advance depending on the number of stages. . Therefore, highly accurate learning processing can be achieved with a simple configuration.

FIG. 13 is a diagram illustrating the effect of learning processing in the embodiment of the present invention. (1) shows the learning process of Non-Patent Document 1, in which a data set of image a is given a correct score of two classes (2 classes) using a feature extraction NN and a class classification NN. From this, the probability distribution of the two classes is calculated.

(2) shows a general multi-data set learning process. Using the feature extraction NN and the class classification NN shown above, the probability distribution of two classes is calculated from the data set of image a to which the correct scores of two classes have been assigned. Further, using the feature extraction NN and the class classification NN shown below, the probability distribution of the three classes is calculated from the data set of the image b to which the correct scores of the three classes have been assigned.

(3) shows the learning process in the embodiment of the present invention, and corresponds to the process by the feature extraction NN 40 and score calculation NN 41 of the NN unit 31 shown in FIG. Using the feature extraction NN 40 and the score calculation NN 41, a one-dimensional score is calculated from a data set of images a (for example, program images) to which two classes of correct scores have been assigned. Further, using the feature extraction NN 40 and the score calculation NN 41, a one-dimensional score is calculated from the data set of the image b (for example, an artistic evaluation image) to which three classes of correct scores have been assigned.

In (3), in the case of a two-class data set, for example, the correct score for the first class is 0.0, and the correct score for the second class is 1.0. Further, in the case of a data set of three classes, for example, the correct score of the first class is 0.0, the correct score of the second class is 0.5, and the correct score of the third class is 1.0.

In (2), for the first class in the 2-class data set and the first class in the 3-class data set, the degree of evaluation of images implied by these correct scores is similar but not the same. For example, assume that the correct score of the first class in the two-class data set is 0.0, and the correct answer score of the first class in the three-class data set is also 0.0. In this case, since the numbers of classes in both datasets are different, the range of evaluation for images with a correct answer score of 0.0 will also be different.

Therefore, as shown in (2), it is necessary to use two different NNs, one for class classification for a two-class data set and one for class classification for a three-class data set.

However, for example, if the number of three-class datasets is smaller than the two-class dataset, the overall accuracy of the feature extraction NN, the class classification NN shown in the upper part, and the class classification NN shown in the lower part will be reduced. Unable to achieve high learning.

Therefore, as shown in (3), in the embodiment of the present invention, by using the score calculation NN41 common to both datasets, which calculates a one-dimensional score, it is possible to calculate a one-dimensional score depending on the number of classes in the dataset. It is possible to realize learning processing without having to do so.

In this way, as shown in (2), conventionally, when learning a neural network using multiple types of data sets, it was necessary to prepare a different neural network for each data set. On the other hand, as shown in (3), in the embodiment of the present invention, there is no need to prepare different neural networks, and it is sufficient to use a single score calculation NN 41. In other words, highly accurate learning processing can be achieved with a simple configuration.

The embodiment of the present invention shown in (3) is particularly effective when the classes of datasets have an order relationship (for example, "great" > "good" > "bad", etc.).

[Representative image extraction device]
Next, a representative image extraction device that extracts representative images from program video using the neural network trained by the learning device 1 shown in FIG. 1 will be described. FIG. 10 is a block diagram showing a configuration example of a representative image extraction device according to an embodiment of the present invention.

The representative image extraction device 2 includes a sampling processing section 50, a score calculation section 51, and a selection section 52. The sampling processing unit 50 inputs the program video, samples the program image, which is a frame image, from the program video at regular intervals, and outputs the program image to the score calculation unit 51.

Note that the sampling processing unit 50 selects in advance a predetermined number of program images from all the program images obtained by sampling the program video, and outputs only the selected predetermined number of program images to the score calculation unit 51. You can do it like this. Thereby, the processing load on the score calculation section 51 and the selection section 52 in the subsequent stages can be reduced.

The score calculation unit 51 includes a trained neural network trained by the learning device 1 shown in FIG. That is, the score calculation unit 51 inputs the optimal parameters output by the learning device 1 and sets them in the neural network.

The score calculation unit 51 inputs the tensor of the program image from the sampling processing unit 50 and calculates a score from the program image using a neural network. Then, the score calculation unit 51 outputs the program image and the score of the program image to the selection unit 52.

As a result, the program image and the score of the program image are calculated for each of the plurality of program images obtained by sampling the program video, and are output to the selection unit 52.

The selection unit 52 inputs program images and scores from the score calculation unit 51 for each of all program images sampled by the sampling processing unit 50. Then, the selection unit 52 sorts the program images in descending order of scores, and selects C program images with high scores from all the program images as representative images. C is an integer of 1 or more and is set in advance.

The selection unit 52 sorts the C representative images in chronological order and outputs the C representative images in chronological order.

Note that the selection unit 52 inputs all program images and their corresponding scores, classifies all program images into, for example, three classes based on the scores by threshold processing, and selects program images of higher classes. The image may be selected as a representative image. The selection unit 52 does not necessarily need to classify the program images by dividing the scores into equal intervals.

For example, when 0.00≦score≦threshold 0.25, the selection unit 52 selects the program image with the score by threshold processing using a preset threshold (for example, 0.25, 0.75). Classify into class 1. Further, when threshold value 0.25<score<threshold value 0.75, the selection unit 52 classifies the program image with the score into the second stage class, and when threshold value 0.75≦score≦1.00, The program image with the score is classified into a third stage class. Then, the selection unit 52 selects the program image of the third stage class as the representative image.

As described above, according to the representative image extraction device 2 of the embodiment of the present invention, the score calculation unit 51 uses the neural network learned by the learning device 1 to calculate the program image obtained by sampling the program video. and calculate the score.

The selection unit 52 sorts all the program images obtained by sampling the program video in descending order of scores, selects C program images with high scores as representative images, and sorts the C representative images in chronological order. Sort and output in order.

Here, the neural network learned by the learning device 1 is a model generated in consideration of program production know-how. Therefore, by using this neural network, it is possible to extract representative images from program videos in consideration of program production know-how. Furthermore, since the representative image can be extracted without using special data other than the program video, the processing load can be reduced. Further, when creating a program homepage using representative images, the amount of work can be significantly reduced.

[Example using representative image extraction device 2]
Next, an example using the representative image extraction device 2 shown in FIG. 10 will be described. FIG. 11 is a diagram illustrating a program HP creation system according to the first embodiment using the representative image extraction device 2. As shown in FIG. This program HP creation system 3 is a system that creates a program HP using program EPG (Electronic Programming Guide) information and program video regarding the program for which the program HP is to be created.

The program HP creation system 3 includes a representative image extraction device 2, a summary video generation section 100, and an automatic placement processing section 101. The summary video generation unit 100 is a component that generates a summary video from a program video using conventional processing, and the representative image extraction device 2 is a device according to the embodiment of the present invention shown in FIG. For example, three representative images are extracted.

The automatic placement processing unit 101 places the program EPG information, the summary video, and the three representative images at preset positions, and creates a program HP as shown in FIG. 11.

FIG. 12 is a diagram illustrating a program DVD sales HP creation system according to the second embodiment using the representative image extraction device 2. As shown in FIG. This program DVD sales HP creation system 4 is a system that creates a program DVD sales HP using DVD advertising comments, DVD package images, and program DVD videos about the program DVD for which the program DVD sales HP is created.

The program DVD sales HP creation system 4 includes a representative image extraction device 2 and an automatic placement processing section 102. The representative image extraction device 2 is a device according to the embodiment of the present invention shown in FIG. 10, and extracts, for example, six representative images from a program video.

The automatic placement processing unit 102 places the DVD promotion comment, DVD package image, and six representative images at preset positions, and creates a program DVD sales website as shown in FIG. 12.

Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the embodiments described above, and can be modified in various ways without departing from the technical concept thereof.

For example, the learning device 1 shown in FIG. 1 uses artistic evaluation images from artistic evaluation open data in addition to program images to learn the neural network, but it is also possible to use only program images. good. Further, the learning device 1 may use open data other than artistic evaluation open data in addition to program images. The open data used for learning may be any data as long as it consists of an image and a correct label that has been evaluated from a predetermined viewpoint for the image.

Further, the feature extraction NN 40 of the NN unit 31 shown in FIGS. 8 and 9 obtains output data of a 1024-dimensional image feature vector, and the score calculation NN 41 handles the 1024-dimensional image feature vector as input data. I made it. This 1024-dimensional image vector is an example, and the output data of the feature extraction NN 40 and the input data of the score calculation NN 41 in the present invention are not limited to the 1024-dimensional image vector.

Note that a normal computer can be used as the hardware configuration of the learning device 1 according to the embodiment of the present invention. The learning device 1 is configured by a computer including a CPU, a volatile storage medium such as a RAM, a non-volatile storage medium such as a ROM, an interface, and the like. The same applies to the representative image extraction device 2 according to the embodiment of the present invention.

The functions of the program learning data generation unit 10, memories 11, 13, artistic learning data generation unit 12, and learning unit 14 provided in the learning device 1 are realized by having the CPU execute a program in which these functions are written. be done.

Further, the functions of the sampling processing section 50, score calculation section 51, and selection section 52 provided in the representative image extraction device 2 are also realized by causing the CPU to execute a program in which these functions are written.

These programs are stored in the storage medium, and are read and executed by the CPU. Additionally, these programs can be stored and distributed in storage media such as magnetic disks (floppy (registered trademark) disks, hard disks, etc.), optical disks (CD-ROMs, DVDs, etc.), semiconductor memories, etc., and can be distributed via networks. You can also send and receive messages.

1 Learning device 2 Representative image extraction device 3 Program HP creation system 4 Program DVD sales HP creation system 10 Program learning data generation units 11, 13 Memory 12 Artistic learning data generation unit 14 Learning unit 20 Sampling processing unit 21 Download processing unit 22 Similarity Degree calculation units 23, 24, 25 Correct score assigning unit 30 Switching unit 31 NN (neural network) unit 32 Error calculation unit 33 Parameter updating unit 40 Feature extraction NN
41 NN for score calculation
50 Sampling processing unit 51 Score calculation unit 52 Selection unit 100 Summary video generation unit 101, 102 Automatic placement processing unit P ₁ , ..., P _N , P _n program image P' ₁ , ..., P' _M , P ' _m still image S _n,m similarity B maximum value

Claims (7)

In a learning device for learning neural networks,
A frame image obtained by sampling a learning program video is defined as a program image, a score of one of the multiple stages assigned to the program image is defined as a first correct score, and a plurality of stages assigned to a predetermined image is defined as a first correct score. The score at any one of the stages is a second correct score, and the neural network is a model in which the program image and the predetermined image are alternately input and a one-dimensional score is output.
a memory storing program learning data consisting of the program image and the first correct score, and predetermined learning data consisting of the predetermined image and the second correct score;
a learning unit that reads the program learning data and the predetermined learning data from the memory and learns the neural network using the program learning data and the predetermined learning data,
The learning department is
Using the neural network, calculate a one-dimensional score of the program image from the program image included in the program learning data as a first score; a neural network unit that calculates a one-dimensional score of the predetermined image from the image as a second score;
An error between the first score calculated by the neural network unit and the first correct score included in the program learning data is calculated as a first error, and an error between the first score and the first correct score included in the program learning data is calculated as a first error. an error calculation unit that calculates an error between the second correct answer score and the second correct answer score as a second error;
A learning device comprising: a parameter updating section that updates parameters of the neural network so that the sum of the first error and the second error calculated by the error calculating section becomes smaller. The learning device according to claim 1,
further comprising a program learning data generation unit that generates the program learning data;
The program learning data generation unit includes:
a sampling processing unit that samples the learning program video into the program image;
a download processing unit that accesses a URL of a homepage of a program corresponding to the learning program video and downloads a still image of the program;
a similarity calculation unit that calculates a degree of similarity between the program image sampled by the sampling processing unit and the still image downloaded by the download processing unit;
The first correct score is assigned to the program image based on the similarity calculated by the similarity calculation unit, and the program learning data consisting of the program image and the first correct score is stored in the memory. A learning device comprising: a first correct score assigning unit that stores a first correct answer score. The learning device according to claim 2,
comprising a predetermined learning data generation unit that generates the predetermined learning data,
The predetermined learning data generation unit includes:
By inputting open data consisting of the predetermined image and a label of one of a plurality of stages assigned in advance to the predetermined image, and converting the label into the second correct answer score, the predetermined A learning device comprising a second correct score assigning unit that assigns the second correct score to an image and stores the predetermined learning data including the predetermined image and the second correct score in the memory. . The learning device according to any one of claims 1 to 3,
The number of program learning data is A (A is a positive integer), the number of predetermined learning data is B (B is a positive integer), A<B, and the result of subtracting A from B is ( As B-A),
The learning department is
A pieces of the program learning data and (B-A) pieces of data that are short of the program learning data with respect to the predetermined learning data, using any or all of the A pieces of the program learning data. The learning device is characterized in that the neural network is trained using the program learning data supplemented by the program learning data and the B pieces of the predetermined learning data. In a representative image extraction device that extracts a representative image from a program video,
a sampling processing unit that samples the program video into a frame image and outputs the frame image as a program image;
A score calculation that calculates a one-dimensional score of the program image from the program image output by the sampling processing unit using a neural network trained by the learning device according to any one of claims 1 to 4. Department and
a selection unit that selects the representative image from all the program images sampled and output from the program video by the sampling processing unit, based on the score calculated by the score calculation unit; A representative image extraction device characterized by: A program for causing a computer to function as the learning device according to any one of claims 1 to 4. A program for causing a computer to function as the representative image extraction device according to claim 5.

Images