JP7486871B1 - Scene extraction system, scene extraction method, and scene extraction program - Google Patents

Abstract

【assignment】
To provide a new technique for extracting scenes that match a user's preferences.
SOLUTION
The scene extraction system 0 includes a storage unit, a division unit, a similarity calculation unit, a preferred scene extraction unit, and a similar scene extraction unit. The division unit divides a video into scenes. The storage unit stores the video importance level for the video feature, the sound importance level for the sound feature, and the speech importance level for the speech feature for each user in the divided split scenes, linked to the user ID. The similarity calculation unit calculates the video similarity, sound similarity, and speech similarity between the split scenes based on the video feature, sound feature, and speech feature of the divided split scenes. The preferred scene extraction unit extracts preferred scenes preferred by the user from the multiple split scenes based on the user's viewing history. The similar scene extraction unit extracts similar scenes from the split scenes based on the preferred scene, the similarity, and the importance level linked to the user ID.
[Selected Figure] Figure 1

Description

The present invention relates to a scene extraction system, a scene extraction method, and a scene extraction program.

Conventionally, there exists technology that extracts scenes that match a user's preferences based on multiple features in the scene.

For example, Patent Document 1 discloses a technology that extracts scenes that match a user's preferences based on the characteristics of the words, images, sounds, etc. in the scene.

Patent No. 5338450

Each user has a different preference for which feature to prioritize when watching a video. Therefore, in order to provide scenes that match the user's preferences, it is preferable to set for each user which feature to prioritize when extracting a scene. However, while the technology in Patent Document 1 can extract scenes based on multiple features, it cannot set the feature that each user prioritizes for each user and extract scenes.

The present invention was made in consideration of the above-mentioned circumstances, and the problem to be solved is to provide a new technology for extracting scenes that match the user's preferences.

In order to solve the above problems, the present invention provides a scene extraction system for extracting scenes from a video, comprising:
the scene extraction system includes a storage unit, a division unit, a similarity calculation unit, a preference scene extraction unit, and a similar scene extraction unit;
The division unit divides the video into scenes,
the storage unit stores, for each user, a video importance level for a video-related feature, a sound importance level for a sound-related feature, and an utterance importance level for an utterance-related feature in the divided split scenes, in association with a user ID;
the similarity calculation unit calculates a video similarity, a sound similarity, and a speech similarity between the split scenes based on a video feature, a sound feature, and a speech feature of the split scenes;
the preference scene extraction unit extracts preference scenes preferred by the user from a plurality of split scenes based on a viewing history of the user;
The similar scene extraction unit extracts similar scenes from the split scenes based on the preferred scene, the video similarity, the sound similarity, the speech similarity, and an importance level linked to the user ID.

The present invention also provides a scene extraction method executed by a scene extraction system that extracts scenes from a moving image, comprising the steps of:
the scene extraction system includes a storage unit, a division unit, a similarity calculation unit, a preference scene extraction unit, and a similar scene extraction unit;
A step of dividing the video into scenes by the dividing unit;
storing, by the storage unit, a video importance level for a video feature, a sound importance level for a sound feature, and an utterance importance level for an utterance feature, for each user, in the divided split scenes, in association with a user ID;
a step of calculating a video similarity, a sound similarity, and a speech similarity between the split scenes based on a video feature, a sound feature, and a speech feature of the split scenes by the similarity calculation unit;
a step of the preference scene extraction unit extracting preference scenes preferred by the user from a plurality of split scenes based on a viewing history of the user;
The similar scene extraction unit extracts similar scenes from the split scenes based on the preferred scene, the video similarity, the sound similarity, the speech similarity, and the importance level linked to the user ID.

The present invention also provides a scene extraction program for extracting a scene from a video, comprising:
A computer is caused to function as a storage unit, a division unit, a similarity calculation unit, a preferred scene extraction unit, and a similar scene extraction unit;
The division unit divides the video into scenes,
the storage unit stores, for each user, a video importance level for a video-related feature, a sound importance level for a sound-related feature, and an utterance importance level for an utterance-related feature in the divided split scenes, in association with a user ID;
the similarity calculation unit calculates a video similarity, a sound similarity, and a speech similarity between the split scenes based on a video feature, a sound feature, and a speech feature of the split scenes;
the preference scene extraction unit extracts preference scenes preferred by the user from a plurality of split scenes based on a viewing history of the user;
The similar scene extraction unit extracts similar scenes from the split scenes based on the preferred scene, the video similarity, the sound similarity, the speech similarity, and an importance level linked to the user ID.

This configuration makes it possible to extract scenes that are preferred by the user based on the features that each user values, and to extract similar scenes that match the user's preferences.

In a preferred embodiment of the present invention, the scene extraction system includes a change importance level creation unit,
the change importance level creation unit creates a change importance level by changing all or a part of the importance level based on the importance level and a predetermined condition;
The similar scene extraction unit extracts scenes with similar change importance from the split scenes based on the preferred scene, the video similarity, the sound similarity, the speech similarity, and the change importance.

This configuration makes it possible to extract change importance similar scenes, which are similar scenes based on the change importance obtained by changing the importance associated with the user ID, and it is possible to find an importance that matches the user's preferences using the change importance and the importance associated with the user ID.

In a preferred embodiment of the present invention, the scene extraction system includes an importance updating unit,
The importance level update unit updates the change importance level related to the change importance level similar scene as the importance level of the user by linking it to a user ID based on a viewing history of the user for the change importance level similar scene.

This configuration makes it possible to use viewing histories of similar scenes with various levels of importance, and update the level of importance to match the user's preferences.

In a preferred embodiment of the present invention, the scene extraction system includes a presentation unit,
The presentation unit presents scenes having a similar change importance level at a lower rate than the similar scenes.

This configuration makes it possible to simultaneously present similar scenes and scenes with similar importance with small changes to the user, and by comparing the viewing histories of each, the user can find an importance level that suits their preferences.

In a preferred embodiment of the present invention, the scene extraction system includes a digest movie creation unit,
The digest movie creation unit creates a digest movie using a plurality of the similar scenes.

This configuration makes it possible to create a digest video that uses multiple similar scenes, providing additional content to users.

In a preferred embodiment of the present invention, the scene extraction system includes a frame extraction unit and a frame similarity calculation unit,
The frame extraction unit extracts the first and last frames of the plurality of similar scenes;
the frame similarity calculation unit calculates frame similarity between the extracted first frame and last frame;
The digest movie creation section creates the digest movie based on the frame similarities.

This configuration makes it possible to create a digest video based on the similarity between the first and last frames of similar scenes, making it possible to create a digest video in which the user does not notice the gaps between similar scenes.

In a preferred embodiment of the present invention, the scene extraction system includes a preference score calculation unit,
the preference score calculation unit calculates a preference score based on the preference scene, the video similarity, the sound similarity, the speech similarity, and the importance level;
The digest video creation unit creates the digest video by designating the similar scene with the highest preference score as the beginning of the digest video, and by designating, among the first frames of similar scenes excluding the beginning similar scene, the similar scene with the highest frame similarity with the last frame of the beginning similar scene as the next scene after the beginning similar scene.

By configuring it in this way, it is possible to start the digest video with a similar scene that best suits the user's preferences, making it possible to create a digest video that will capture the user's interest.

The present invention provides a new technology for extracting scenes that match a user's preferences.

FIG. 1 is a block diagram showing a configuration of a scene extraction system according to an embodiment of the present invention. FIG. 2 is a diagram showing a hardware configuration according to the present embodiment. 4 shows an example of a data configuration stored in a storage unit in the present embodiment. 4 shows an example of a data configuration stored in a storage unit in the present embodiment. 4 is a flowchart of a scene extraction process in the present embodiment. 11 is a flowchart of an importance level update process according to the present embodiment. 5 is a flowchart of a digest movie creation process according to the embodiment.

The scene extraction system of the present invention will be described below with reference to the drawings. The drawings show a preferred embodiment. However, the present invention can be embodied in many different forms and is not limited to the embodiment described in this specification.

For example, in this embodiment, the configuration, operation, etc. of the scene extraction system are described, but similar effects can be achieved by the executed method (steps), device, computer program, etc. The program in this embodiment may be provided as a non-transient computer-readable recording medium, may be provided so as to be downloadable from an external server, or the program may be started on an external computer to implement its functions on a client terminal (so-called cloud computing).

In addition, in this embodiment, a "unit" may include, for example, a combination of hardware resources implemented by a circuit in the broad sense and software information processing that can be specifically realized by these hardware resources. In this embodiment, "information" is represented, for example, by the physical value of a signal value representing a voltage or current, the high or low value of a signal value as a binary bit collection consisting of 0 or 1, or a quantum superposition (so-called quantum bit), and communication and calculations can be performed on a circuit in the broad sense.

In the broad sense, a circuit is a circuit that is realized by appropriately combining a circuit, circuitry, processor, memory, etc. In other words, it includes a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), etc.

<System Overview>
Fig. 1 is a block diagram showing the configuration of a scene extraction system according to the present embodiment. As shown in Fig. 1, the scene extraction system 0 includes a scene extraction device 1 and a user terminal 2. The scene extraction device 1 is configured to be able to communicate with the user terminal 2 via a network NW.

The scene extraction device 1 extracts scenes that match the preferences of a user from a video, based on information about the video, the scene, the user, etc. The scene extraction device 1 may extract a similar scene from another split scene in one video, based on a user's preferred scene extracted from the one video. The scene extraction device 1 may also extract similar scenes from multiple split scenes of multiple videos, based on the extracted preferred scene.

A general-purpose server computer, a personal computer, or the like can be used as the scene extraction device 1. It is also possible to configure the scene extraction device 1 using multiple computers.

The user watches videos and scenes via the user terminal 2. Furthermore, the user may input the importance level and the like via the user terminal 2 and transmit it to the scene extraction device 1. As the user terminal 2, a terminal device such as a smartphone, tablet terminal, or personal computer can be used.

In this embodiment, the network NW is an IP (Internet Protocol) network, but there are no restrictions on the type of communication protocol, and there are also no restrictions on the type or size of the network.

<Hardware Configuration>
2 is a hardware configuration diagram. As shown in Fig. 2A, the information processing device 10 (scene extraction device 1) has a control unit 101, a storage unit 102, and a communication unit 103, which are used to perform the functions of each unit and each process.

The control unit 101 includes one or more processors such as a CPU (Central Processing Unit), and controls the overall operation and processing of the information processing device 10 by executing the scene extraction program according to the present invention, an OS (Operating System), browser software, and other applications.

The storage unit 102 is a hard disk drive (HDD), a solid state drive (SSD), a read only memory (ROM), a random access memory (RAM), etc., and stores the scene extraction program according to the present invention and data used when the control unit 101 executes processing based on the program. The control unit 101 executes processing based on the scene extraction program stored in the storage unit 102, thereby realizing the functional configuration described below.

The communication unit 103 controls communication with the network NW, and performs input necessary to operate the information processing device 10 and outputs related to the results of the operation.

As shown in FIG. 2(b), the terminal device 9 (user terminal 2) has a control unit 91, a memory unit 92, a communication unit 93, an input unit 94, and an output unit 95, and is used to perform the functions of each unit and each process.

The control unit 91 of the terminal device 9 includes one or more processors such as a CPU, and controls the overall operation and processing of the terminal device 9. The storage unit 92 of the terminal device 9 is an HDD, SSD, ROM, RAM, etc., and stores the above-mentioned applications and data used when the control unit 91 executes processing based on a program.

The communication unit 93 of the terminal device 9 controls communication with the network NW. The input unit 94 of the terminal device 9 is a mouse, keyboard, or the like, and inputs operation requests from the user/provider to the control unit 91. The output unit 95 of the terminal device 9 is a display, or the like, and displays the results of processing by the control unit 91, etc.

<Functional components>
As shown in FIG. 2, the scene extraction device 1 includes a division unit 11, a feature generation unit 12, a similarity calculation unit 13, a preference scene extraction unit 14, a preference score calculation unit 15, a similar scene extraction unit 16, a presentation unit 17, a change importance creation unit 18, an importance update unit 19, a frame extraction unit 1a, a frame similarity calculation unit 1b, and a digest movie creation unit 1c.

The arrangement of these functional components is an example, and some of the functional components of the scene extraction device 1 may be arranged in one or more devices configured to be able to communicate with the scene extraction device 1 and the user terminal 2.

<Data structure>
3 and 4 are examples of data configurations stored in the storage unit in this embodiment. The storage unit of the scene extraction device 1 stores split scene information, video similarity information, sound similarity information, speech similarity information, viewing history information, importance information, similarity score information, preference score information, similar scene information, and similar scene similarity information.

The arrangement of each piece of data is also an example, and some or all of the data stored in the memory unit of the scene extraction device 1 may be stored in one or more devices configured to be able to communicate with the scene extraction device 1 and the user terminal 2.

<Creating split scenes>
5 is a flowchart of the scene extraction process in this embodiment. First, in step S501, the division unit 11 divides a video into scenes to create divided scenes (for example, by chapter, by batter in baseball, or by shots and other scenes in a soccer game). The number of videos divided by the division unit 11 may be one or more. In this embodiment, the division unit 11 divides all videos that it manages (stored in the storage unit, received via a user terminal) into scenes.

The division unit 11 may divide the video at equal time intervals (e.g., 5 minutes), or may detect change points in the video footage and divide it automatically. The division unit 11 may also divide the video footage into similar images as one cluster by clustering the video footage in the time direction, or, if a tag for dividing the video (e.g., chapter) is attached, the division unit 11 may divide the video footage by the tag.

The division unit 11 stores in the storage unit split scene information relating to which video the split scene was split from and which section of the original video the scene belongs to. As shown in FIG. 3(a), the split scene information includes a video ID, a split scene ID, and a video section. This allows the scene extraction device 1 to refer to, for example, that split scene ID "M1_C2" is a split scene of a video with video ID "M1", and is a scene from the 10-20 minute section of that video.

<Feature Generation>
In step S502, the feature generator 12 generates a video feature, a sound feature, and a speech feature of the split scene created by the splitter 11. For example, a video or a scene can be decomposed into a plurality of elements, and a feature for each element can be generated. In this embodiment, a video or a scene is decomposed into three elements, namely, video, sound, and speech content (meaning of the conversation), and the feature generator 12 generates each feature as a video feature, a sound feature, and a speech feature.

In this embodiment, video features are features related to things that appear in a video or scene and their movements (e.g., the color, shape, size, movement, and positional relationship of an object). In this embodiment, sound features are features related to sounds in a video or scene (e.g., volume, speed, frequency, wavelength, etc.). In this embodiment, speech features are features related to the content of speech (audio) in a video or scene (meaningful content that can be understood by humans), and are generated, for example, by extracting human speech and performing natural language processing.

The feature generator 12 may generate features by vectorizing the video (moving and still) and sound (audio and non-audio) contained in the split scene. As shown in FIG. 3(a), the feature generator 12 associates the generated features with the split scene ID and stores them in the storage unit.

For example, the feature generator 12 generates video features by breaking down the split scenes into frames (still images) and creating a group of image files.

To generate video features from a group of image files, a proprietary trained encoding model may be used, or a trained model published as open source software may be used. For example, a publicly available model based on Vision Transformer may be used, which can generate 400-dimensional features.

Additionally, trained models that explain the patterns in images in natural language are also publicly available, and features can also be generated by embedding natural language information extracted from images. A model that embeds natural language information can be trained independently, or a publicly available trained model can be used. For example, one publicly available encoding published by OpenAI as an API (Application Programming Interface) can be applied, in which case 1,536-dimensional features can be generated.

For example, the feature generator 12 generates sound features by extracting only sound information related to sound from the split scenes and creating a sound file. The extracted sound information may or may not include human voice.

To generate features from audio files, a proprietary trained encoding model may be used, or a trained model published as open source software may be used. For example, a publicly available model based on the Vision Transformer may be used, which can generate 383232-dimensional features.

For example, the feature generation unit 12 extracts speech information related to human speech (voice) from the split scenes to generate speech features. The feature generation unit 12 may use the voice information extracted from the split scenes to perform voice recognition and convert it into natural language to create a speech language file.

The features from the spoken language file may use a proprietary trained encoding model, or a trained model published as open source software. For example, an encoding model published as an API by OpenAI may be used, in which case 1536-dimensional features can be generated.

<Calculation of similarity>
In step S503, the similarity calculation unit 13 calculates the similarity between the split scenes. The similarity calculation unit 13 calculates the video similarity, sound similarity, and speech similarity between the split scenes based on the video feature amount, sound feature amount, and speech feature amount of the divided split scenes.

The similarity calculation unit 13 may calculate the similarity (image similarity, sound similarity, speech similarity) of each feature between all split scenes, and store it in the storage unit as shown in Figures 3(b) to (d). When there are N split scenes, the similarity calculation unit 13 performs N x N calculations to calculate the similarity of each feature. The similarity calculation unit 13 may calculate the similarity using cosine similarity, which calculates the similarity between vectors.

The similarity calculation unit 13 may or may not calculate the similarity between the same split scenes (for example, the similarity between split scene IDs "M1_C1" and "M1_C1"). If it is acceptable to extract scenes that the user has viewed once as similar scenes that are similar to the user's preferred scenes, the similarity calculation unit 13 may calculate the similarity between the same split scenes to be high. On the other hand, if it is not acceptable to extract scenes that the user has viewed once, the similarity calculation unit 13 may calculate the similarity between the same split scenes to be low, or may not calculate the similarity as in Figures 3(b) to (d).

<Extraction of favorite scenes>
In step S504, the preference scene extraction unit 14 extracts preference scenes that the user prefers from the multiple split scenes based on the user's viewing history. The preference scene extraction unit 14 stores viewing history information regarding the viewing history of the split scenes for each user in the storage unit. The viewing history information includes a user ID, a split scene ID, and a cumulative viewing time, as shown in FIG. 3(e).

The preferred scene extraction unit 14 counts the viewing time for each split scene for each user and calculates the cumulative viewing time. This allows the scene extraction device 1 to refer to, for example, the fact that user ID "U1" previously viewed the split scene with split scene ID "M2_C1" for a total of 280 seconds.

The preference scene extraction unit 14 extracts preference scenes that are preferred by the user based on the cumulative viewing time. For example, if the cumulative viewing time of a split scene exceeds a certain time (threshold), the preference scene extraction unit 14 may extract the split scene as a preference scene. When the preference scene extraction unit 14 extracts a preference scene, it may store the preferred scene in the memory unit as shown in FIG. 3(e) (associating the split scene ID with a preference flag "1").

The preferred scene extraction unit 14 may also extract preferred scenes using the proportion of the cumulative viewing time to the playback time of the split scene. If the cumulative viewing time of a split scene is equal to or greater than a predetermined proportion (e.g., half) of the playback time of the scene, the preferred scene extraction unit 14 extracts the split scene as a preferred scene. In this embodiment, this proportion is set to half, but is not limited to this and the administrator, etc., can set the proportion via the terminal.

For example, since the playback time of split scene ID "M1_C1" is 600 seconds, if the cumulative viewing time of user "U1" is 300 seconds, the preference scene extraction unit 14 extracts split scene ID "M1_C1" as a preference scene of user "U1". On the other hand, since the playback time of split scene ID "M2_C2" is 630 seconds, if the cumulative viewing time of user "U1" is 300 seconds, the preference scene extraction unit 14 does not extract split scene ID "M2_C2" as a preference scene of user "U1".

In addition, the number of preferred scenes may be limited. For example, the preferred scene extraction unit 14 may extract the top M (M = 1, 2, ...) split scenes in terms of cumulative viewing time as preferred scenes. The preferred scene extraction unit 14 may also extract the top M split scenes in terms of the proportion of the cumulative viewing time to the playback time of the split scene as preferred scenes.

The preference scene extraction unit 14 may further extract preference scenes using a genre associated with the split scene. By storing the video ID or split scene ID in association with a genre in the storage unit, the preference scene extraction unit 14 may extract preference scenes using this genre.

For example, the user's preferred genres are received in advance via the user terminal, and the memory unit stores the genres in association with the user ID, so that the preferred scene extraction unit 14 can extract preferred scenes based on the user's cumulative viewing time and the received genres.

<Calculation of preference score>
In step S505, the preference score calculation unit 15 calculates a preference score based on the preferred scene, the video similarity, the sound similarity, the speech similarity, and the importance associated with the user ID. The storage unit stores the video importance, the sound importance, and the speech importance for each user, as shown in FIG. 4(f). The sum of the video importance, the sound importance, and the speech importance for each user may be 1. The sum of the video importance, the sound importance, and the speech importance may be different for each user. The preference score calculation unit 15 may receive the importance for each user via the user terminal 2.

Which of the three - video, sound, or speech - is emphasized when extracting similar scenes varies from user to user. Video importance is information about how much importance the user places on the video element when extracting similar scenes. Sound importance is information about how much importance the user places on the sound element when extracting similar scenes. Speech importance is information about how much importance the user places on the speech (natural language) element when extracting similar scenes.

The preference score calculation unit 15 can calculate a preference score for extracting similar scenes suitable for each user by applying an importance (weight) to each similarity calculated by the similarity calculation unit 13.

First, the preference score calculation unit 15 calculates similarity score information as shown in FIG. 4(g) based on the preferred scene, video similarity, sound similarity, and speech similarity. The brackets next to each score in FIG. 4(g) indicate the calculation for calculating each score (written for convenience to show the correspondence with the similarities in FIGS. 3(b) to (d)). For example, in the case of the video score of split scene ID "M1_C2", the video similarity of 0.12 between "M1_C2" and the preferred scene "M1_C1" and the video similarity of 0.22 between "M1_C2" and the preferred scene "M2_C1" are calculated (summed up) to result in a video score of 0.34.

The preference score calculation unit 15 calculates a similarity score (video score, sound score, speech score) by adding up the similarity between each of the split scenes and the preferred scene for each split scene, based on the similarity between each of the preferred scenes (preference flag is 1) and each of the split scenes.

For example, in FIG. 3(e), the preferred scenes for user ID "U1" are split scene IDs "M1_C1" and "M2_C1", so the similarities for each are added together to calculate a similarity score. The video score for split scene ID "M1_C2" in FIG. 4(g) is 0.34, which is the sum of the video similarity of split scene IDs "M1_C1" and "M1_C2" (0.12) and the video similarity of split scene IDs "M2_C1" and "M1_C2" (0.22). In this way, a similarity score that takes multiple preferred scenes into consideration can be calculated, and similar scenes can be extracted.

In addition, the preference score calculation unit 15 may set the video similarity between split scene IDs "M1_C1" and "M1_C2" of 0.12 and the video similarity between split scene IDs "M2_C1" and "M1_C2" of 0.22 as the video score for split scene ID "M1_C2".

The preference score calculation unit 15 calculates a preference score based on the similarity score and the importance level. For example, the preference score calculation unit 15 calculates a preference score for each split scene using formula (1), and stores the preference score in the storage unit in association with the user ID and split scene ID as shown in FIG. 4(h). The brackets in the preference score in FIG. 4(h) indicate the calculation using formula (1) (written for convenience to show the correspondence with the score in FIG. 4(g)). For example, the preference score for split scene ID "M2_C2" is 1.022, and the brackets indicate the calculation for deriving that score.

<Extraction of Similar Scenes>
In step S506, the similar scene extraction unit 16 extracts scenes similar to the user's preferred scenes from the split scenes based on the preferred scenes, video similarity, audio similarity, speech similarity, and importance associated with the user ID. The similar scene extraction unit 16 extracts scenes similar to the user's preferred scenes from the split scenes based on the preference scores calculated by the preference score calculation unit 15.

For example, the similar scene extraction unit 16 may extract split scenes with preference scores equal to or greater than a certain numerical value (threshold value) as similar scenes. The similar scene extraction unit 16 may also extract the top L scenes (L=1, 2, ...) with the highest preference scores from among the split scenes as similar scenes.

<Recommendation of similar scenes>
In step S507, the presentation unit 17 presents (recommends) similar scenes based on the similar scenes extracted by the similar scene extraction unit 16. The presentation unit 17 may further present similar scenes for each genre based on the genre associated with the video ID or the similar scene ID.

<Creating the degree of importance of change>
6 is a flowchart of the importance update process in this embodiment. Processes similar to those described above are denoted by the same reference numerals and their description will be omitted. In step S601, the change importance creation unit 18 creates a change importance by changing all or a part of the importance based on the importance and a predetermined condition.

The change importance level is an importance level obtained by changing the importance level associated with the user ID based on a predetermined condition. In order to extract a scene (a change importance level similar scene) that is different from the similar scene extracted based on the original importance level (importance level associated with the user ID), the change importance level creation unit 18 creates an importance level (change importance level) obtained by changing the importance level associated with the user ID.

The change importance creation unit 18 creates each change importance (changed image importance, changed sound importance, changed speech importance) using, for example, formulas (2) to (4). The change importance creation unit 18 creates a changed image importance that is X times the image importance (for example, X = 1.1) using formulas (2) to (4), and further creates a changed sound importance and a changed speech importance such that the sum of the changed image importance, changed sound importance, and changed speech importance is 1.

The change importance level creation unit 18 can create a changed sound importance level by multiplying the sound importance level by X, and a changed speech importance level by multiplying the speech importance level by X, in the same way as with equations (2) to (4). X may be an integer or a non-integer fraction. Furthermore, the change importance level creation unit 18 may determine X randomly, or may change X periodically.

<Extraction of scenes with similar change importance based on change importance>
In step S602, the preference score calculation unit 15 may calculate a preference score based on the preferred scene, the video similarity, the sound similarity, the speech similarity, and the change importance degree. The similar scene extraction unit 16 extracts change importance degree similar scenes from the split scenes based on the preferred scene, the video similarity, the sound similarity, the speech similarity, and the change importance degree.

<Recommendation of scenes with similar change importance>
In step S603, the presentation unit 17 presents a certain proportion of scenes with similar change importance levels. The presentation unit 17 presents scenes with similar change importance levels at a lower proportion than the similar scenes.

When presenting K (e.g., 10) scenes, for example, the presentation unit 17 presents 70% of similar scenes with the highest preference scores, 10% of similar change-importance scenes with X times the video importance and the highest preference scores, 10% of similar change-importance scenes with X times the sound importance and the highest preference scores, and 10% of similar change-importance scenes with X times the speech importance and the highest preference scores.

In this way, by the presentation unit 17 presenting scenes including a small number of scenes with similar change importance among the scenes to be presented, there is a possibility that the user will prefer to watch the scenes with similar change importance. If the user prefers to watch the scenes with similar change importance that are presented in smaller numbers than the similar scenes, it can be determined that the change importance is more preferred by the user than the importance associated with the current user ID.

<Update on priority>
In step S604, the importance updating unit 19 updates the change importance related to the change importance similar scene as the user's importance by linking it to the user ID based on the user's viewing history for the change importance similar scene. If the importance updating unit 19 determines that the user prefers to view a change importance similar scene with a video importance multiplied by X, it updates the user's importance to the change importance (a changed video importance multiplied by X, and the sound importance and speech importance changed accordingly).

For example, if the split scene that the user watched for the longest time in one day is a scene with a similar change importance with the video importance multiplied by X, the importance update unit 19 determines that the user prefers to watch scenes with a similar change importance with the video importance multiplied by X, and updates the change importance as the user's importance by linking it to the user ID.

In this way, by periodically changing the change importance level and further updating the importance level based on the user's viewing history, it is possible to bring the importance level closer to the user's preferences.

<Extracting frames from similar scenes>
7 is a flowchart of the digest movie creation process in this embodiment. Processes similar to those described above are denoted by the same reference numerals, and their description will be omitted. In step S701, the frame extraction unit 1a extracts the first frame (still image) and the last frame (still image) of the multiple similar scenes extracted by the similar scene extraction unit 16.

<Calculation of Frame Similarity>
In step S702, the frame similarity calculation unit 1b calculates the frame similarity between the first frame and the last frame extracted by the frame extraction unit 1a. The frame similarity calculation unit 1b calculates the frame similarity between the first frame of the similar scene extracted by the frame extraction unit 1a and the last frame of the similar scene extracted by the frame extraction unit 1a excluding the first similar scene.

The frame similarity calculation unit 1b generates features of the first frame and the last frame of all similar scenes extracted by the similar scene extraction unit 16, and stores similar scene information such as that shown in FIG. 4(i) in the storage unit. It is considered that the frame similarity calculation unit 1b generates features in the same way that the feature generation unit 12 generates features.

The frame similarity calculation unit 1b further calculates similarities based on the feature amounts of the first frames of all similar scenes and the feature amounts of the last frames of all similar scenes, and stores similar scene similarity information such as that shown in FIG. 4(j) in the storage unit. It is considered that the frame similarity calculation unit 1b calculates the similarity of the feature amounts using cosine similarity, which calculates the similarity between vectors. In this embodiment, as shown in FIG. 4(j), the frame similarity calculation unit 1b does not calculate the similarity between the first frame and the last frame of the same similar scene.

<Creating digest videos>
In step S703, the digest movie creation unit 1c creates a digest movie using the plurality of similar scenes extracted by the similar scene extraction unit 16. The digest movie creation unit 1c may create a digest movie by connecting similar scenes from among the extracted similar scenes in descending order of preference score, for example.

The digest video creation unit 1c may create a digest video by setting the extracted similar scene with the highest preference score calculated by the preference score calculation unit 15 as the beginning of the digest video, and setting the similar scene with the highest frame similarity to the last frame of the beginning similar scene, among the first frames of similar scenes other than the beginning similar scene, as the next scene after the beginning similar scene.

The digest video creation unit 1c may further create a digest video by selecting the similar scene that has the highest frame similarity with the last frame of the Jth (J=1, 2, ...) similar scene among the first frames of similar scenes excluding similar scenes used in creating the digest videos up to that point, as the J+1th similar scene.

For example, suppose that the similar scene extraction unit 16 extracts similar scene IDs "M4_C1," "M6_C2," "M10_C1," and "M11_C1" as similar scenes. Furthermore, if the preference score of the similar scene ID "M10_C1" is the highest, the digest movie creation unit 1c sets the similar scene ID "M10_C1" as the first scene.

As shown in FIG. 4(j), the similarity between the last frame of similar scene ID "M10_C1" and the first frame of similar scene ID "M4_C1" is the highest, so the digest movie creation unit 1c selects similar scene ID "M4_C1" as the next scene.

Furthermore, according to FIG. 4(j), the similarity between the last frame of similar scene ID "M4_C1" and the first frame of similar scene ID "M10_C1" is the highest, but similar scene ID "M10_C1" has already been used to create the digest movie. Therefore, the digest movie creation unit 1c excludes this similar scene and sets the next scene to similar scene ID "M11_C1", which has the highest frame similarity with the last frame of similar scene ID "M4_C1". The digest movie creation unit 1c repeats this process to create the digest movie until all extracted similar scenes have been used.

As described above, the configuration of the present invention can provide a new technology for extracting scenes that match the user's preferences.

0 Scene extraction system 1 Scene extraction device 11 Dividing unit 12 Feature quantity generating unit 13 Similarity calculation unit 14 Preference scene extraction unit 15 Preference score calculation unit 16 Similar scene extraction unit 17 Presentation unit 18 Change importance level creation unit 19 Importance level update unit 1a Frame extraction unit 1b Frame similarity calculation unit 1c Digest movie creation unit 2 User terminal NW Network

Claims (9)

A scene extraction system for extracting scenes from a video, comprising:
the scene extraction system includes a storage unit, a division unit, a similarity calculation unit, a preference scene extraction unit, and a similar scene extraction unit;
The division unit divides the video into scenes,
the storage unit stores, for each user, a video importance level for a video-related feature, a sound importance level for a sound-related feature, and an utterance importance level for an utterance-related feature in the divided split scenes, in association with a user ID;
the similarity calculation unit calculates a video similarity, a sound similarity, and a speech similarity between the split scenes based on a video feature, a sound feature, and a speech feature of the split scenes;
the preference scene extraction unit extracts preference scenes preferred by the user from a plurality of split scenes based on a viewing history of the user;
the similar scene extraction unit extracts similar scenes from the split scenes based on the preferred scene, the video similarity, the sound similarity, the speech similarity, and an importance level associated with the user ID;
Scene extraction system. The scene extraction system includes a change importance level creation unit,
the change importance level creation unit creates a change importance level by changing all or a part of the importance level based on the importance level and a predetermined condition;
the similar scene extraction unit extracts scenes having a similar change importance level from the split scenes based on the preference scene, the video similarity level, the sound similarity level, the speech similarity level, and the change importance level;
The scene extraction system according to claim 1 . The scene extraction system includes an importance updating unit,
the importance updating unit updates the change importance level related to the change importance level similar scene as the importance level of the user by linking the change importance level related to the change importance level similar scene with a user ID based on a viewing history of the user regarding the change importance level similar scene;
The scene extraction system according to claim 2 . The scene extraction system includes a presentation unit,
the presentation unit presents scenes with a similar change importance level that have a lower ratio than the similar scenes;
The scene extraction system according to claim 3 . the scene extraction system includes a digest movie creation unit,
the digest movie creation unit creates a digest movie using a plurality of the similar scenes;
The scene extraction system according to claim 1 . The scene extraction system includes a frame extraction unit and a frame similarity calculation unit,
The frame extraction unit extracts the first and last frames of the plurality of similar scenes;
the frame similarity calculation unit calculates frame similarity between the extracted first frame and last frame;
the digest movie creator creates the digest movie based on the frame similarity.
The scene extraction system according to claim 5 . The scene extraction system includes a preference score calculation unit,
the preference score calculation unit calculates a preference score based on the preference scene, the video similarity, the sound similarity, the speech similarity, and the importance level;
the digest movie creation unit creates the digest movie by setting the similar scene with the highest preference score as the beginning of the digest movie, and by setting, among the first frames of similar scenes other than the beginning similar scene, a similar scene having the highest frame similarity with a last frame of the beginning similar scene as a scene next to the beginning similar scene.
The scene extraction system according to claim 6. A scene extraction method executed by a scene extraction system for extracting scenes from a video, comprising:
the scene extraction system includes a storage unit, a division unit, a similarity calculation unit, a preference scene extraction unit, and a similar scene extraction unit;
A step of dividing the video into scenes by the dividing unit;
storing, by the storage unit, a video importance level for a video feature, a sound importance level for a sound feature, and an utterance importance level for an utterance feature, for each user, in the divided split scenes, in association with a user ID;
a step of calculating a video similarity, a sound similarity, and a speech similarity between the split scenes based on a video feature, a sound feature, and a speech feature of the split scenes by the similarity calculation unit;
a step of the preference scene extraction unit extracting preference scenes preferred by the user from a plurality of split scenes based on a viewing history of the user;
The similar scene extraction unit extracts similar scenes from the split scenes based on the preferred scene, the video similarity, the audio similarity, the speech similarity, and an importance level associated with the user ID.
Scene extraction method. A scene extraction program for extracting scenes from a video, comprising:
A computer is caused to function as a storage unit, a division unit, a similarity calculation unit, a preferred scene extraction unit, and a similar scene extraction unit;
The division unit divides the video into scenes,
the storage unit stores, for each user, a video importance level for a video-related feature, a sound importance level for a sound-related feature, and an utterance importance level for an utterance-related feature in the divided split scenes, in association with a user ID;
the similarity calculation unit calculates a video similarity, a sound similarity, and a speech similarity between the split scenes based on a video feature, a sound feature, and a speech feature of the split scenes;
the preference scene extraction unit extracts preference scenes preferred by the user from a plurality of split scenes based on a viewing history of the user;
the similar scene extraction unit extracts similar scenes from the split scenes based on the preferred scene, the video similarity, the sound similarity, the speech similarity, and an importance level associated with the user ID;
Scene extraction program.