JP2020129357A - Moving image editing server and program - Google Patents

Abstract

To provide a server and a program that enable convenient creation of moving image content.SOLUTION: Provided is a moving image editing server for creating a moving image to be distributed to user terminals, comprising: a recommended article extraction unit for extracting recommended articles including a material moving image on the basis of preference profile information of a user; a template creation unit for creating a template in which multiple scenes provided with scene labels are specified; and a composite moving image creation unit for creating a composite moving image on the basis of the material moving image and the template. The composite moving image creation unit comprises: a moving image splitting unit for splitting the material moving image into multiple split material moving images; a first classifier for adding scene labels to the split material moving images; and a split moving image combining unit for creating a composite moving image by extracting and combining the split material moving images that match the respective scenes in the template on the basis of the scene labels. Also provided is a program therefor.SELECTED DRAWING: Figure 2

Description

The present invention relates to a server and a program that automatically generate a composite moving image to be distributed to a user terminal.

Conventionally, a moving image is divided into a plurality of chapters and metadata is added.
For example, Patent Document 1 discloses a moving image processing apparatus that efficiently retrieves a scene image at a desired moment from a moving image having a plurality of chapters, and divides the moving image into a plurality of large blocks at predetermined unit time intervals. A block division unit, a complexity analysis unit that digitizes the complexity of image change of each large block, and a small block division that divides the playback time of each large block into a plurality of small blocks according to the numerical value of the complexity. There has been proposed a moving image processing apparatus including a section and a chapter creating section that creates a chapter by sequentially dividing a plurality of small blocks into a predetermined number in a time series.

JP, 2011-130007, A

Since it takes a lot of time and effort to create moving image content, it has been required to provide a system capable of automatically creating moving image content. There is also a problem that even a moving image content created with a great deal of effort will not be played back to the end unless the content matches the user's taste.
There is also a need to animate existing materials such as in-house manuals and product catalogs, but there is a high threshold for building an in-house video editing environment for video servers and editors with specialized technology. However, there is a problem in that the outsourcing cost increases when the outsourcing is performed in.

Therefore, it is an object of the present invention to provide a server and a program that automatically generate moving image content in order to solve the above problems.

The video editing server of the present invention is a server that creates a video to be distributed to a user terminal, and is provided with a scene label and a recommended article extraction unit that extracts a recommended article including a material video based on user preference profile information. A template creating unit that creates a template in which a plurality of scenes are defined, and a composite moving image creating unit that creates a composite moving image based on the material moving image and the template. Of the divided material moving images, a first classifier that gives a scene label to the divided material moving images, and a divided material moving image that matches each scene of the template based on the scene label is extracted and combined. Accordingly, a divided moving image combining unit that creates a combined moving image is provided.

In the moving image editing server, the combined moving image creating unit may include a second classifier that attaches an object name label indicating an object name displayed in the divided material moving image.
In the moving image editing server, the divided moving image combining unit may extract a divided material moving image that matches each scene of the template based on the scene label and the object name label.
In the moving image editing server, the recommended article extracting unit extracts the recommended articles with a recommendation value based on similarity information, and the divided moving image combining unit divides the recommended articles obtained from the recommended articles having high recommendation values. The feature may be that the material moving image is preferentially adopted.
In the moving image editing server, the synthetic moving image creating unit may include a telop inserting unit that inserts a telop into the synthetic moving image.
In the moving image editing server, the telop inserting unit may insert a telop created by summarizing the character information described in the recommended article into the synthetic moving image.
In the moving image editing server, the template may define an allowable time range of each scene, and the divided moving image combining unit may extract a divided material moving image that matches the allowable time range.
In the moving image editing server, the moving image dividing unit has a learning moving image for creating learning data of the first classifier in addition to a material moving image dividing function for dividing the material moving image into a plurality of divided material moving images. It may be characterized by having a dividing function.

A video editing server program of the present invention is a video editing program for a server that distributes video content to a user terminal accessed via the Internet. As a recommended article extraction unit that extracts a template, a template creation unit that creates a template in which a plurality of scenes with scene labels are defined, and a composite video creation unit that creates a composite video based on the material video and the template Functioning, the composite moving image creating unit divides a material moving image into a plurality of divided material moving images, a first classifier that gives a scene label to the divided material moving images, and the template based on the scene label. And a divided moving image combining unit that creates a combined moving image by extracting and combining divided material moving images that match each scene.

According to the present invention, it is possible to provide a server and a program that automatically generate moving image content that matches a user's taste. In addition, it is possible to easily animate existing materials such as in-house manuals and product catalogs without constructing an animation editing environment in-house.

It is a block diagram of the video editing system which concerns on the example of embodiment. It is a block diagram of the video editing server which concerns on the example of embodiment. It is a figure explaining an example of a template. It is a block diagram of a moving image creation unit. It is a figure explaining extraction of the division material animation which matches the conditions of each scene. It is a flow of a moving image editing process. It is explanatory drawing of a moving image division process. It is explanatory drawing of a division moving image combining process. It is a flow of a summary sentence creation process.

<Structure>
As shown in FIG. 1, the moving image editing system of the present invention according to the embodiment includes a moving image editing server 1, an administrator terminal 2, and a plurality of user terminals 3. In addition, in the example of FIG. 1, the example in which the video editing server 1 is configured by one unit has been described, but it is also possible to realize the video editing server 1 by a plurality of servers, and a high-spec device equipped with a GPU. From the viewpoint of work efficiency, it is preferable to separately provide and perform work such as creating a later-described classifier.

The moving image editing server 1 is constructed by installing moving image editing software and database software in a server device including a processing unit having a CPU, a storage unit having a storage device such as an HDD, and a communication unit having a LAN port. .. The moving image editing software includes a browsing history storage unit 11, a recommended article extracting unit 12, a template creating unit 13, a classifier creating unit 14, and a composite moving image creating unit 15. The database software manages a browsing record DB 21, a recommended article DB 22, a template DB 23, a learning data DB 24, a synthetic moving image DB 25, and a label DB 26.

The browsing history storage unit 11 records, in the browsing record DB, browsing records of articles on the Web that the user has browsed while logged in. A unique article ID is given to the article on the Web that the user browsed, and the article is recorded in association with the user ID and the browsing time. Alternatively, a mechanism for recording the browsing record as an anonymous user may be adopted.

The recommended article extracting unit 12 regularly (for example, weekly) performs a work of collecting recommended articles from the Web based on the user's preference profile. In the example of the embodiment, the browsing history information of the user is acquired from the browsing record DB 21, the article with the moving image similar to the article browsed by the user is extracted by the content-based filtering, and the recommended value is added with the recommendation value based on the similarity information. It is recorded in DB22. Alternatively, the degree of similarity may be determined by collaborative filtering, or hybrid filtering that uses both content-based filtering and collaborative filtering may be employed. The recommended article DB 22 constantly stores dozens or hundreds of dozens of recommended articles with high recommendation values.

The template creating unit 13 creates a template that defines conditions (label, order, and allowable time range of each scene) attached to the scenes forming the moving image, and stores the template in the template DB 23. The template is composed of a plurality of scenes, and in each scene, target object information to be displayed as a divided moving image and characteristics of its movement are defined. In other words, for example, the flow of the video content is defined as "move the building corridor" -> "zoom out of the building room" -> "appearance", and there is no discomfort when the scenes are connected. It is important to define the characteristics of the object information and its movement with this in mind. The template creation by the template creating unit 13 is executed regularly (for example, several times a year) or when necessary. FIG. 3 is an example of a template consisting of four scenes. Scenes 1 and 2 are labeled with a “cooking” scene label, scene 3 is labeled with a “after cooking” scene label, and scene 4 is labeled with a scene label. Has a scene label of “inside view/appearance”. Although each scene of the template can be labeled with an object name label, which will be described later, it is preferable that many objects can be applied by defining object information that is a superordinate concept of the object name label. Further, the allowable number of characters range of the telop to be inserted in each scene may be set as a condition.

The classifier creating unit 14 creates the first classifier 153 and the second classifier 154 that are the learned models by acquiring the learning data from the learning data DB 24 and performing machine learning. The first classifier is created by using a moving image data set and learning the dependency relationship of the time-series features of each scene. The second classifier learns the features of the object based on the image data set. The classifier creation unit 14 creates a classifier that is a learned model, for example, about several times a year. As the learning data, data collected from the Internet or data labeled by the company may be used, or a labeled data set may be procured and used.

As shown in FIG. 4, the composite moving image creating unit 15 includes a recommended article reading unit 151, a moving image dividing unit 152, a first classifier 153, a second classifying unit 154, a divided moving image combining unit 155, and music insertion. A portion 156 and a telop insertion portion 157 are provided.
The recommended article reading unit 151 acquires, from the recommended article DB 22, a recommended article including a material moving image that is a material for creating a composite moving image.

The moving image dividing unit 152 divides the material moving image into frames, determines a dividing position by setting a section where the color greatly changes from the immediately preceding frame as a section, and creates a divided material moving image (see FIG. 7 ). More specifically, for example, the colors of all the pixels in the image of the frame are digitized, the average color of the colors of all the pixels is calculated, and the average color of the pixels of the previous frame is divided, and the average color is divided. To create a split material movie. Such a division method is useful when collecting scenes that have good visual connection as one divided moving image, and a divided material moving image that is connected for a relatively long time is generated.
The moving image dividing unit 152 is also used to create a finely divided moving image used as learning data. More specifically, for example, the divided video for learning data is created by digitizing the color of all pixels of the frame image, judging the change in the color saturation of black from the brightness of the pixel, and dividing when it becomes dark to some extent. To do.
As described above, the moving image dividing unit 152 according to the present embodiment selectively uses the moving image dividing technique according to the purpose.

The first classifier 153 is a classifier that uses a recurrent neural network. When a moving image is input, the first classifier 153 outputs a label as a classification result in which motion is also determined and stores it in the label DB 26. More specifically, the first classifier 153 is used to attach a word (annotation word) representing a situation of a scene such as during cooking and inside the store as a scene label to the divided material moving image.
The second classifier 154 is a classifier that uses a convolutional neural network. When a moving image or an image is input, the second classifier 154 stores the object name in the moving image or the image in the label DB 26 and outputs it to the label DB 26. More specifically, the second classifier 154 is used to attach a word (annotation word) representing an object name such as seafood, grilled meat, a person, furniture, etc. to the divided material moving image as an object name label. The divided moving image to be extracted is selected based on the metadata obtained by the second classifier 154. For example, when it is determined that the divided moving image extracted from the first moving image contains seafood, the divided moving images in which the seafood is reflected are preferentially extracted from the second and third moving images.

The divided moving image combining unit 155 selects the template having the highest affinity for the material moving images, extracts the divided material moving images that match the conditions of each scene of the selected template, and combines the extracted divided material moving images. FIG. 5 is a diagram for explaining extraction of a divided material moving image that matches the conditions of each scene. It is assumed that among the divided material moving images divided into 10, the sixth to ninth divided material moving images from the start match the condition. It has been extracted.
The divided moving image combining unit 155 extracts divided material moving images from the plurality of extracted recommended articles in descending order of the recommendation value. If no split material videos that match all the scenes in the template were extracted by the first recommended article, then match the split material videos that match the conditions of the missing scene for the recommended article with the next highest recommendation value. By doing so, a segmented material moving image for satisfying all the scenes required by the template is extracted. At this time, the metadata obtained from the second classifier 154 is used so that the one in which a similar object is reflected in each scene is selected from the divided videos extracted from the videos of each article.

The music inserting unit 156 inserts music into the combined moving image (without music) combined by the divided moving image combining unit 155.
The telop insertion unit 157 inserts a telop into the composite moving image (without telop) combined by the divided moving image combining unit 155. When the allowable number of characters range of the telop to be inserted in each scene is set as a condition, the telop having the number of characters satisfying the condition is inserted.

The administrator terminal 2 and the user terminal 3 are computers provided with an input unit, a display unit, a processing unit, a storage unit, and a communication unit. A computer with a browser.
The administrator uses the administrator terminal 2 to change the settings of the video editing server 1 and manage the operation of the database.
The user can access the moving image editing server 1 using the user terminal 3 and browse the automatically generated moving image content.

<Operation>
The flow of the moving image editing process will be described with reference to FIG.
The moving image generation agent periodically (for example, once a week) executes the moving image automatic generation process by the synthetic moving image generation unit 15 (STEP 1). The recommended article reading unit 151 of the composite moving image creating unit 15 reads the recommended articles stored in the recommended article DB 22 in order from the highest recommended value (STEP 2).
The moving image dividing unit 152 of the combined moving image creating unit 15 divides the moving image (material moving image) included in the recommended article to be processed into a plurality of moving images (divided material moving images) (STEP 3). The composite moving image creating unit 15 labels each divided material moving image by the first classifier 153 and the second classifier 154 (STEP 4). The division position of the material moving image may be stored in the recommended article DB 22 and/or the learning data DB 24. The synthetic|combination moving image production part 15 extracts the division|segmentation material moving image which corresponds to the label and permissible time of the selected template (STEP5). For example, in FIG. 8, the third scene condition of the template is “after cooking”, “2 to 4 seconds”, and the fourth scene condition is “inside view/appearance” and “2 to 4 seconds”. Two divided material moving images satisfying these conditions are extracted.

When the divided material moving images are not acquired for all the scenes, the work of STEPs 2 to 5 is repeated for the next recommended article (STEP 6, STEP 7). For example, in FIG. 8, since the divided material moving image that satisfies the first and second scene conditions is not extracted, it is necessary to continuously search for the divided material moving image that satisfies the scene condition for the next recommended article.

When the divided material moving images are acquired for all the scenes, the divided moving image combining unit 155 of the combined moving image creating unit 15 combines the extracted divided material moving images to create a combined moving image (STEP 8). The music inserting unit 156 of the synthetic moving image creating unit 15 inserts BGM prepared in advance into the synthetic moving image (STEP 9). The telop insertion unit 157 of the composite moving image creating unit 15 inserts a telop into the composite moving image (STEP 10). The moving image in which the music and the telop are inserted is stored in the synthesized moving image DB 25 as a completed moving image, and the user can browse the completed moving image within the authority of his/her user ID (STEP 11).

According to the video editing system of the exemplary embodiment described above, it becomes possible to automatically generate video content that matches the user's taste based on the user's taste profile, and periodically deliver it as a recommended video. .. It is also possible to create video advertisements, video press releases, and animate manuals and product catalogs without having to prepare video editing software, video servers, editors with specialized skills, etc. ..

<Modification>
The moving image editing server 1 in a preferable mode has a summary document creating function for inserting as a telop. The flow of creating a summary document will be described with reference to FIG.
STEP 91: Paragraph division/document division The telop insertion unit 157 divides the input document into paragraphs, and divides the documents in each paragraph into documents. Further, a sentence that is too long to be displayed as a telop of a moving image and deteriorates readability is further divided into a plurality of sentences at a portion satisfying a condition such as a specific part of speech or notation.
STEP 92: Morphological Analysis of Document The telop insertion unit 136 subjects each sentence to morphological analysis and extracts a token that is the minimum unit of syntactic analysis.
STEP 93: Deletion of Unnecessary Words/Unnecessary Paragraphs The telop insertion unit 157 deletes sentences and paragraphs defined as invalid according to the predefined invalid sentence determination rule. For example, a line starting from a specific symbol such as “■” or “▼”, a line surrounded by the specific symbol, a URL, a mail address, an address/telephone number, etc. are deleted.

STEP 94: Deletion of Stop Words, etc. The telop insertion unit 157 removes words such as “ni”, “kara”, “kore”, and “san” that are not so significant words (stop words) and specific parts of speech such as particles. delete.
STEP95: Creation of token bigram A token bigram is obtained by connecting a plurality of tokens satisfying a specific condition (for example, a predefined part-of-speech condition). For example, “2014” (noun, proper noun, general) and “June” (noun, proper noun, general) are connected to be “June 2014”, or “Verdi” (proper noun) and “sponsorship” "(Common noun) is connected and it is called "Verdi sponsorship".

STEP96: Extraction of Important Sentences TF-IDF score based on tokens and token bigrams Tokens and token viagrams that are characteristic words are extracted from an index that evaluates the importance of words, and sentence segmentation is performed based on the aforementioned word similarity determination. The summary is done by extracting important sentences from each segment.
STEP97: Fitting to template The summary (important sentences) is subjected to syntactic analysis and divided into clauses and syntactic trees. In the above-mentioned template, the number of characters required for each scene is defined. Based on the modification relation between clauses, the sentence is cut so that a natural section as a sentence fits in each template and the template is applied.
The above-described summary sentence creating function can support not only Japanese but also multiple languages such as English.

Although the preferred embodiments of the present invention have been described above, the technical scope of the present invention is not limited to the description of the above embodiments. Various modifications and improvements can be added to the above-described embodiment, and such modifications and improvements are also included in the technical scope of the present invention.

1 Video Editing Server 2 Administrator Terminal 3 User Terminal 11 Browsing History Saving Section 12 Recommended Article Extracting Section 13 Template Creating Section 14 Classifier Creating Section 15 Synthetic Video Creating Section 21 Viewing Record DB
22 Recommended Article DB
23 Template DB
24 Learning data DB
25 Synthetic video DB
26 Label DB
151 Recommended Article Reading Unit 152 Video Dividing Unit 153 First Classifier 154 Second Classifier 155 Divided Video Combining Unit 156 Music Inserting Unit 157 Telop Inserting Unit

Claims (9)

On the server that creates the video for distribution to the user terminal,
A recommended article extracting unit that extracts recommended articles including material videos based on user preference profile information;
A template creation unit that creates a template in which multiple scenes with scene labels are specified,
A composite moving image creating unit that creates a composite moving image based on the material moving image and the template,
A moving picture dividing section for dividing the material moving picture into a plurality of divided material moving pictures;
A first classifier for attaching a scene label to the divided material moving image;
A moving image editing server, comprising: a divided moving image combining unit that extracts combined material moving images that match each scene of the template based on the scene label and combines the divided material moving images to create a combined moving image. The moving image editing server according to claim 1, wherein the synthetic moving image creating unit includes a second classifier that attaches an object name label indicating an object name displayed in the divided material moving image. The moving image editing server according to claim 2, wherein the divided moving image combining unit extracts a divided material moving image that matches each scene of the template based on the scene label and the object name label. The recommended article extraction unit extracts the recommended article with a recommendation value based on similarity information,
4. The moving image editing server according to claim 1, wherein the divided moving image combining unit preferentially adopts a divided material moving image obtained from a recommended article having a high recommendation value. The moving image editing server according to claim 1, wherein the synthetic moving image creating unit includes a telop inserting unit that inserts a telop into the synthetic moving image. The moving image editing server according to claim 5, wherein the telop insertion unit inserts a telop created by summarizing the character information described in the recommended article into the composite moving image. In the template, the allowable time range of each scene is defined,
7. The moving image editing server according to claim 1, wherein the divided moving image combining unit extracts a divided material moving image that matches the allowable time range. In addition to the material moving image dividing function for dividing the material moving image into a plurality of divided material moving images, the moving image dividing unit has a learning moving image dividing function for creating learning data of the first classifier. The moving image editing server according to claim 1, wherein the moving image editing server is a moving image editing server. In the video editing program for the server that delivers the video content to the user terminal accessed via the Internet,
The server
A recommended article extraction unit that extracts recommended articles including material videos based on user preference profile information,
A template creation unit that creates a template in which a plurality of scenes with scene labels are defined, and
Functioning as a composite moving image creating unit that creates a composite moving image based on the material moving image and the template,
A moving picture dividing section for dividing the material moving picture into a plurality of divided material moving pictures;
A first classifier for attaching a scene label to the divided material moving image;
A divided moving image combining unit that creates a combined moving image by extracting divided material moving images that match each scene of the template based on the scene label and combining them;
A program for a video editing server, comprising: