CN112565875B - Method, device, equipment and computer readable storage medium for automatically generating video - Google Patents

Abstract

According to example embodiments of the present disclosure, there are provided a method, apparatus, device, computer-readable storage medium and computer program product for automatically generating a video. Involved in the fields of knowledge graph, deep learning and video creation. A method for automatically generating a video, comprising receiving user input, the user input including first multimedia content and key phrases for describing the video, the first multimedia content having at least one data format in a plurality of predetermined data formats ; based on key phrases, determine at least one node from a pre-built knowledge map; based on the first multimedia content, obtain second multimedia content associated with at least one node; and based on the first multimedia content and the second Multimedia content, generate video. As a result, videos can be automatically and efficiently generated.

Description

Method, device, device and computer-readable storage medium for automatically generating video

technical field

Embodiments of the present disclosure mainly relate to the field of information processing, and more specifically, to methods, devices, devices, computer-readable storage media and computer program products for automatically generating videos.

Background technique

With the development of mobile data networks, the proportion of video data on the Internet is gradually surpassing that of text. In the technical field related to video production, innovative applications based on artificial intelligence are still in a blank state, and there is no automatic video production scheme. However, the traditional video production process has the following defects: high requirements for users, qualified video requires producers to be able to apply a lot of complex software, and the materials used to make videos are not easy to collect. Therefore, a solution to automatically generate high-quality video is needed.

Contents of the invention

According to an example embodiment of the present disclosure, a solution for automatically generating a video is provided.

In a first aspect of the present disclosure, there is provided a method for automatically generating a video, comprising: receiving user input, the user input includes a first multimedia content and a keyphrase for describing the video, the first multimedia content has At least one data format in a plurality of predetermined data formats; based on key phrases, determine at least one node from a pre-built knowledge map; based on the first multimedia content, obtain second multimedia content associated with at least one node ; and generating a video based on the first multimedia content and the second multimedia content.

In a second aspect of the present disclosure, there is provided an apparatus for automatically generating a video, including: an input receiving module configured to receive user input, the user input including first multimedia content and key phrases used to describe the video, The first multimedia content has at least one data format in a plurality of predetermined data formats; the first node determination module is configured to determine at least one node from a pre-built knowledge map based on key phrases; the first multimedia content An acquisition module configured to acquire second multimedia content associated with at least one node based on the first multimedia content; and a first video generation module configured to acquire the second multimedia content based on the first multimedia content and the second multimedia content Media content, generate video.

In a third aspect of the present disclosure, an electronic device is provided, including one or more processors; and a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors Execution causes the one or more processors to implement the method according to the first aspect of the present disclosure.

In a fourth aspect of the present disclosure, there is provided a computer-readable medium on which is stored a computer program that implements the method according to the first aspect of the present disclosure when executed by a processor.

In a fifth aspect of the present disclosure, there is provided a computer program product comprising computer program instructions, the computer program instructions being implemented by a processor as the method of the first aspect of the present disclosure.

It should be understood that what is described in the Summary of the Invention is not intended to limit the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will be readily understood through the following description.

Description of drawings

The above and other features, advantages and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description when taken in conjunction with the accompanying drawings. In the drawings, identical or similar reference numerals denote identical or similar elements, wherein:

Figure 1 shows a schematic diagram of an example environment in which various embodiments of the present disclosure can be implemented;

FIG. 2 shows a flowchart of an example of a process of automatically generating a video according to some embodiments of the present disclosure;

FIG. 3 shows a flowchart of another example of a process of automatically generating a video according to some embodiments of the present disclosure;

Fig. 4 shows a schematic block diagram of an apparatus for automatically generating video according to an embodiment of the present disclosure; and

Figure 5 shows a block diagram of a computing device capable of implementing various embodiments of the present disclosure.

Detailed ways

Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the drawings, it should be understood that the disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein; A more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for exemplary purposes only, and are not intended to limit the protection scope of the present disclosure.

In the description of the embodiments of the present disclosure, the term "comprising" and its similar expressions should be interpreted as an open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be read as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same object. Other definitions, both express and implied, may also be included below.

As mentioned above, in the traditional solution, there are the following defects when creating a video: (1) In the video production process, in addition to the content shot by the user, a variety of video materials are often required to complete the content required by the user. However, these video materials have insurmountable professional thresholds for users, and mainly have problems such as difficulty in obtaining, few types, and high prices; (2) users need to spend a lot of time to synthesize materials, such as determining different formats. Transition effects between clips or determining the position of text in the video, etc.

Exemplary embodiments of the present disclosure propose a solution for automatically generating videos. In this solution, firstly, the first multimedia content input by the user and the description of the video to be generated are received. Then according to the above multimedia content and description, acquire the second multimedia content. Finally, the first multimedia content input by the user and the second acquired multimedia content are combined to generate a video. Thereby, high-quality multimedia content for generating videos can be automatically acquired to efficiently generate videos.

FIG. 1 shows a schematic diagram of an example environment 100 in which various embodiments of the present disclosure can be implemented. As shown, example environment 100 includes computing device 110 , database 120 , and video 130 and users 140 . Computing device 110 may be connected to database 120 . Computing device 110 may also receive user input from user 140 . The database 120 may be any suitable centralized or distributed database, including but not limited to databases based on knowledge graph technology and databases based on retrieval.

In one embodiment, in order to generate the video 130 , the computing device 110 may acquire multimedia content for the video 130 , that is, material, from a knowledge graph stored in a database. The knowledge graph essentially aims to describe the knowledge that exists objectively in the real world and the semantic network of the relationship between knowledge. Based on the application fields of knowledge graphs, knowledge graphs are usually divided into general knowledge graphs and vertical knowledge graphs (also known as industry knowledge graphs). The general knowledge graph is not oriented to a specific field, and can be compared to structured encyclopedic knowledge. This type of knowledge graph contains a large amount of common sense knowledge and emphasizes the breadth of knowledge. The vertical knowledge graph is oriented to specific fields, built on the basis of industry knowledge, and emphasizes the depth of knowledge. The knowledge graph in this paper may be a knowledge graph dedicated to video, where each node in the knowledge graph stores multimodal data associated with the node. But it can also be a general knowledge graph, which is not limited in this disclosure.

In an alternative embodiment, some data in the knowledge graph in the database 130 is incomplete. For example, taking the engine as an example, when the knowledge map is initially constructed, the concept of the engine may include attributes such as fuel consumption, color, displacement, brand, and model. These attributes are common sense knowledge and are well known to the public, so in When the knowledge graph is initially constructed, these concepts about engine attributes can be added to the knowledge graph, and a concept is located at a node. The overall framework of a knowledge map is preliminarily constructed. However, such knowledge as the specific fuel consumption, what colors are included, how many liters the displacement is, and what brands and models are included are ever-changing, and they do not belong to common sense knowledge, so they cannot be given in detail. These data need to be collected again. In addition, the existing data in the knowledge graph may only be unimodal, such as only in text format, which cannot be used for video creation.

In the case of the above-mentioned imperfect knowledge graph, the computing device 110 can obtain the whole network data from outside the database via the network to reconstruct the knowledge graph. The network may be any suitable network, including but not limited to the Internet, Local Area Networks (LAN), Metropolitan Area Networks (MAN), Wide Area Networks (WAN), wired networks such as fiber optic networks and coaxial cables, and networks such as WIFI, cellular telecommunications networks Wireless network such as Bluetooth and so on.

Computing device 110 may be any suitable computing device, centralized or distributed, including but not limited to personal computers, servers, clients, handheld or laptop devices, multiprocessors, microprocessors, set-top boxes, programmable consumer electronics products, network PCs, minicomputers, mainframe computer systems and distributed clouds, and combinations thereof.

The computing device 110 can also use the acquired multimedia content to synthesize a video 130, and the detailed process of generating the video will be described below.

FIG. 2 shows a flowchart of an example of a process 200 of automatically compositing videos according to some embodiments of the present disclosure. Process 200 may be implemented by computing device 110 .

At 210, computing device 110 receives user input, the user input including first multimedia content and a keyphrase for describing a video, the first multimedia content having at least one data format among a plurality of predetermined data formats.

The computing device 110 may receive user input from the user 140, the user input includes first multimedia content provided by the user for video creation, and the computing device 110 uses the first multimedia content as basic data to generate a video. The first multimedia content may be data in a format of text data, picture data, video data or sound data or multimodal data. The first multimedia content may be a common structured type such as excel, json, csv, etc.

The user input also includes a key phrase used to describe the video, that is, a short description of what the user 140 is to create the video to reflect, which may be the subject of the video. For example, the first multimedia content input by user A is the excel text data of the weather temperature in February, the key phrase input by user A is "temperature chart in February", or the first multimedia content input by user B is The picture of its own image, the key phrase "user B takes you to visit world famous places" input by user B.

In one embodiment, the computing device 110 may clean the text data in the first multimedia content, and perform a word segmentation operation on the text data to extract entity attribute information of the text data. The cleaned text data will be used to generate video subtitles, charts, titles and other information.

In one embodiment, the computing device 110 may perform summary processing on the video data in the first multimedia content to obtain important material therein for use in final composite video. The computing device 110 can also perform scene segmentation on the video data, and then use a specific algorithm to understand the video for each scene, obtain information such as the theme, classification, and people involved in each scene, and then select according to the key phrases input by the user. The most suitable scene is used as the material.

In one embodiment, the computing device 110 can identify the picture data in the first multimedia content, and filter pictures with a size smaller than 300*300, containing advertisements, pornography and politics, etc., if the filtered pictures If too few, query the multimodal map for intelligent supplementation.

In one embodiment, the computing device 110 can convert the sound data in the first multimedia content into text data and/or convert the text data into sound data, so as to form subtitles or audio information in the final video generation .

The above processing of different types of data will be further elaborated below.

At 220, the computing device 110 determines at least one node from the pre-built knowledge graph based on the key phrase. There may be multiple nodes in the pre-built multimodal knowledge graph, and each node may correspond to multiple similar key phrases and store associated multimodal data. The pre-built multi-modal knowledge graph can be stored in the database 120, and can also be constructed by the computing device 110, which is not limited by the present disclosure.

In one embodiment, the computing device 110 determines the matching degree between the key phrase and the target key phrase corresponding to the target node in the knowledge graph, and if it is determined that the matching degree is greater than a first predetermined threshold, the target node is determined as at least one node . For example, continue to use the above-mentioned user A and user B as examples for description. The computing device 110 determines that the matching degree between the target key phrases "trend", "trend", "situation", etc. in the knowledge graph and the key phrase "trend graph" input by user A is greater than the first predetermined threshold of 0.8 (the maximum is 1) , then determine the node A corresponding to the above target key phrases "trend", "trend", and "situation" as the target node, or determine the node A corresponding to the target key phrases "weather" and "temperature" through "temperature" ' is the target node. The matching degree within the key phrase bracket can be obtained by calculating the Euclidean distance of the vector of the phrase, which will not be repeated in this disclosure. Similarly, the computing device 110 may determine, as the target node, a node that corresponds more to the target keyphrase "monument site", wherein the matching degree between the target keyphrase "monument site" and the key phrase "famous place" input by user B is greater than the first predetermined Threshold 0.8. The numerical value of the first predetermined threshold 0.8 is illustrative, and the present disclosure does not limit it here.

At 230, computing device 110 obtains second multimedia content associated with at least one node based on the first multimedia content. For example, after the computing device 140 determines at least one node in the above 220, it may determine the associated second multimedia content in the knowledge graph through the node as the supplementary material of the first multimedia content.

In one embodiment, the computing device 110 determines at least one data format not included in the first multimedia content, and obtains data in at least one data format associated with at least one node as the second multimedia content content. For example, continue to use the above-mentioned user A and user B as examples for description. The computing device 110 determines that the weather in the first multimedia content input by user A is excel text data, then the computing device 110 determines that the first multimedia content does not include picture data, video data and sound data. Then the computing device 110 can determine from the knowledge graph in the database 120 the data associated with the nodes A and A' in the form of picture data, video data and sound data, for example including but not limited to weather icons, the sound of thunder, Background animation of snow, animation template of graph, etc.

For user B, computing device 110 determines that the first multimedia content is a picture of user B's own image, and then it can determine from the knowledge map in database 120 that associated with node B, in the form of text data, video data and sound data Format data, such as including but not limited to videos, text introductions and animations of world attractions.

In another embodiment, the computing device 110 determines the data volume of the multimedia content in at least one data format in the first multimedia content, and if it is determined that the data volume is less than a second predetermined threshold, the computing device 110 obtains the data volume associated with at least one node Associated data in at least one data format serves as second multimedia content. For example, continue to use the above-mentioned user A and user B as examples for description. Computing device 110 determines that the data volume of the excel text data in the first multimedia content input by user A is less than the second predetermined threshold 8KB, for example, it only records the weather information of 10 days in February, then computing device 110 obtains and node The data associated with A' in at least one data format is used as the second multimedia content, for example, the text information of the weather in February is used as the second multimedia content.

For user B, computing device 110 determines that the data volume of the first multimedia content input by user 140 is, for example, less than the second predetermined threshold of 20MB, then computing device 110 acquires data associated with Node B in at least one data format as Secondary multimedia content, such as videos, text introductions and animations of world attractions.

Please note that the above-mentioned values of the first predetermined threshold and the second predetermined threshold are only exemplary, and different thresholds may be adjusted according to user input and the computing device 110 .

By automatically analyzing the basic material provided by the user and the requirements for the video, and matching the analysis results in the multi-modal knowledge map to automatically obtain data with insufficient data in the basic material or high-quality data with insufficient data format, by This solves the defect that it is difficult to obtain materials in the traditional solution.

In an alternative embodiment, the computing device 110 can use the preset video retrieval module, similar image retrieval module, and topic retrieval module to search for relevant fields in the multimodal graph to supplement insufficient information in the given data.

At 240, computing device 110 generates a video based on the first multimedia content and the second multimedia content. For example, the computing device 110 may further process the acquired multimedia content in multiple formats to generate a video.

In one embodiment, the computing device 110 performs semantic analysis on text content in the first multimedia content and the second multimedia content to generate text elements. Then, the computing device 110 determines at least one of the position of the text element in the video, the size of the text in the text element, the display effect of the text element, and the display time of the text element to generate the video.

For example, the computing device 110 performs semantic analysis on the text content in the multimedia content, associates it with the picture information, and further determines its limited position in the associated picture, continuous display time, and dynamic changes in the size and position of the text , Dynamic effects of text, etc. Follow the steps above. Text content can be associated with image frames so that the text information clearly describes each image frame.

In another embodiment, the computing device 110 acquires video content in the first multimedia content and the second multimedia content. Computing device 110 then determines a number of image frames in the video content that are associated with the key phrase. The computing device 110 then determines the sequence of the multiple image frames in the video and the transition effect between the multiple image frames, and finally, the computing device 110 generates the video by using the transition effect according to the sequence. For example, the computing device 110 may determine to determine the key image frame in the video content in the multimedia content according to the key phrase input by the user 140 . The computing device 110 may use the image frames whose matching degree with the key phrase is greater than a threshold among the image frames as a plurality of image frames associated with the key phrase. The image frame can often best reflect the final presentation effect of the video expected by the user. The computing device 110 may then determine image frame-image frame, image frame-video, and video-image frame transition effects based on the determined associated plurality of image frames. For example, taking the above-mentioned user B as an example, the computing device 110 can determine the appearance order of the pictures and videos in the final synthesized video according to the different styles of world famous attractions, and add transition effects between the pictures and video data to make the transition more accurate. nature.

By analyzing the properties of text, pictures and videos in the video material, it can automatically optimize the order in which they appear in the video according to their properties, and automatically set the conversion effect to make the video more smooth and natural as a whole.

Thus, it is possible to automatically acquire high-quality multimedia content for generating a video, and efficiently generate a high-quality video according to the relationship between the multimedia content. It solves the key problem that the production of video requires high technology for users and the material is difficult to obtain.

FIG. 3 shows a flowchart of another example of the process of automatically generating a video according to some embodiments of the present disclosure. Process 300 may be implemented by computing device 110 . The computing device 110 can realize the steps of the present disclosure through the underlying framework based on FFMPEG, that is, a series of operations of text, picture, video, sound, etc. are collectively referred to as a video process. FFMPEG is a relatively low-level video processing tool in linux-like systems. Its main functions include encoding and decoding video. Of course other frameworks may also be applied and this disclosure is not intended to be limiting.

At 310 , computing device 110 receives user input from user 140 of first multimedia content and a keyphrase describing the video.

At 320, the computing device 110 processes the first multimedia content input by the user, and acquires second multimedia content from the knowledge graph 360 according to the user input to supplement the first multimedia content.

At 330, the computing device 110 performs basic processing on the first multimedia content and the second multimedia content. The basic processing includes but is not limited to: sound-related operations, including multi-source sound synthesis, sound transformation, sound clipping, etc.; custom mask operation, this function is mainly for the subsequent application processing 340 service, and can realize various animations The production of effects; video-related functions, specifically realize the dynamic effect operation of video size, position transformation, video color, frame number and duration.

At 440 , computing device 110 performs further application processing based on base processing 330 . Compared with the basic processing 330, the application processing 340 implements a higher-level encapsulation of the functions of FFMPEG.

For example, computing device 110 may apply underlying processing 340 to generate a chart video. Computing device 110 may analyze the data entered by user 140 and then select different charts to add the data to video 130 as required by user 140 .

In one embodiment, the computing device 110 can be based on a custom mask method, which is mainly applied to the situation where the chart can be displayed at one time. When implementing it, it is necessary to select a dynamic template first, and then use the dynamic template as the current filled chart mask to form an animation effect to generate a video.

In another embodiment, the computing device 110 may be based on the method of normalizing the coordinate axes, which is mainly applied to the situation where the graph cannot be displayed at one time. Only 7 days are displayed each time, that is, 28 7-day trend information need to be displayed. These trend lines are displayed continuously in the video, which causes the content in the chart to change constantly. In order to reflect this change, the video size Under the condition of not changing, the computing device 110 can display the change of the trend in real time by calculating the current data information and continuously moving the coordinate axis, so as to generate the required chart video.

In the application processing 340, the computing device 110 may use the AR character interface provided by the database 120 to automatically generate an AR character broadcast according to the input data. Computing device 110 can convert sound data into video data, convert the sound into text first, and then calculate the relationship between the text and the time point of the sound to determine the size and location of the text at each time point to form a series of time texts . The computing device 110 can also count information such as the length, number of entity words, and redundant numbers of a given text string, and then break up and mix according to the above data, and then perform the position, moving speed, color, and so on of the barrage according to the video time. font size, etc. to generate automatic subtitles.

Finally, at 350 , computing device 110 composites video 130 . The computing device can firstly store the video 130 in the cloud, and then send the stored address to the user 140 for the user to view and share.

According to the present disclosure, automatic synthesis of high-quality video can be realized by analyzing user input data, supplementing user data, and then performing the above-mentioned processing on the data.

The specific implementation of each step in step 410 to step 450 refers to the description in FIG. 2 , and details are not repeated here.

Fig. 4 shows a schematic block diagram of an apparatus 400 for automatically generating video according to an embodiment of the present disclosure. As shown in FIG. 4 , the device 400 includes: an input receiving module 410 configured to receive user input, the user input includes first multimedia content and key phrases used to describe the video, the first multimedia content has various predetermined At least one data format in the data format; the first node determination module 420 is configured to determine at least one node from a pre-built knowledge map based on the key phrase; the first multimedia content acquisition module 430 is configured to be based on the first A multimedia content, acquiring second multimedia content associated with at least one node; and a first video generation module 440 configured to generate a video based on the first multimedia content and the second multimedia content.

In some embodiments, the first node determination module 420 may include: a matching module configured to determine the matching degree between the key phrase and the target key phrase corresponding to the target node in the knowledge graph; and the second node determination module , configured to determine the target node as at least one node if it is determined that the matching degree is greater than a first predetermined threshold.

In some embodiments, the first multimedia content acquisition module 430 includes: a data format determination module configured to determine at least one data format not included in the first multimedia content; and the second multimedia content An acquisition module configured to acquire data associated with at least one node and in at least one data format as the second multimedia content.

In some embodiments, the first multimedia content acquisition module 430 includes: a data volume determination module configured to determine the data volume of the multimedia content in at least one data format in the first multimedia content; and the third The multimedia content obtaining module is configured to obtain data associated with at least one node and in at least one data format as second multimedia content if it is determined that the amount of data is less than a second predetermined threshold.

In some embodiments, the first video generation module 440 includes: a text element generation module configured to perform semantic analysis on the text content in the first multimedia content and the second multimedia content to generate text elements; and a second video generation module configured to generate a video based on the text element.

In some embodiments, the second video generation module includes: a third video generation module configured to determine the position of the text element in the video, the size of the text in the text element, the display effect of the text element, and the display time of the text element At least one of the to generate a video.

In some embodiments, the first video generation module 440 includes: a video content acquisition module configured to acquire video content in the first multimedia content and the second multimedia content; an image frame determination module configured to Determining a plurality of image frames associated with the key phrase in the video content; and a fourth video generating module configured to generate a video based on the plurality of image frames.

In some embodiments, the fourth video generation module includes: a conversion effect determination module configured to determine the sequence of multiple image frames in the video and the conversion effect between multiple image frames; and the fifth video generation module is configured Configured to generate video in sequence using transition effects.

In some embodiments, the plurality of predetermined data formats include at least one of text data format, picture data format, video data format and sound data format.

Fig. 5 shows a schematic block diagram of an example device 500 that may be used to implement embodiments of the present disclosure. Device 500 may be used to implement computing device 110 of FIG. 1 . As shown, the device 400 includes a central processing unit (CPU) 510 which can be programmed according to computer program instructions stored in a read only memory (ROM) 520 or loaded from a storage unit 580 into a random access memory (RAM) 530 program instructions to perform various appropriate actions and processes. In the RAM 530, various programs and data necessary for the operation of the device 500 can also be stored. The CPU 510 , ROM 520 and RAM 530 are connected to each other through a bus 540 . An input/output (I/O) interface 550 is also connected to bus 540 .

Multiple components in the device 500 are connected to the I/O interface 550, including: an input unit 560, such as a keyboard, a mouse, etc.; an output unit 570, such as various types of displays, speakers, etc.; a storage unit 580, such as a magnetic disk, an optical disk, etc. ; and a communication unit 590, such as a network card, a modem, a wireless communication transceiver, and the like. The communication unit 590 allows the device 500 to exchange information/data with other devices through a computer network such as the Internet and/or various telecommunication networks.

The processing unit 510 executes various methods and processes described above, such as the process 200 and/or the process 300 . For example, in some embodiments, process 200 and/or process 300 may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as storage unit 580 . In some embodiments, part or all of the computer program may be loaded and/or installed on the device 500 via the ROM 520 and/or the communication unit 590 . When a computer program is loaded into RAM 530 and executed by CPU 610, one or more steps of process 200 and/or process 300 described above may be performed. Alternatively, in other embodiments, CPU 510 may be configured to perform process 200 and/or process 300 in any other suitable manner (eg, by means of firmware).

The functions described herein above may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate array (FPGA), application specific integrated circuit (ASIC), application specific standard product (ASSP), system on a chip (SOC), load programmable logic device (CPLD), etc.

Program codes for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special purpose computer, or other programmable data processing devices, so that the program codes, when executed by the processor or controller, make the functions/functions specified in the flow diagrams and/or block diagrams Action is implemented. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media would include one or more wire-based electrical connections, portable computer discs, hard drives, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, compact disk read only memory (CD-ROM), optical storage, magnetic storage, or any suitable combination of the foregoing.

In addition, while operations are depicted in a particular order, this should be understood to require that such operations be performed in the particular order shown, or in sequential order, or that all illustrated operations should be performed to achieve desirable results. Under certain circumstances, multitasking and parallel processing may be advantageous. Likewise, while the above discussion contains several specific implementation details, these should not be construed as limitations on the scope of the disclosure. Certain features that are described in the context of separate embodiments can also be implemented in combination in a single implementation. Conversely, various features that are described in the context of a single implementation can also be implemented in multiple implementations separately or in any suitable subcombination.

Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims (20)

1. A method of automatically generating video, comprising:

receiving user input comprising first multimedia content and a key phrase describing the video, the first multimedia content having at least one of a plurality of predetermined data formats, and the first multimedia content being multimedia content for the video obtained from a knowledge-graph stored in a database;

determining at least one node from a pre-constructed knowledge graph based on the key phrase;

obtaining second multimedia content associated with the at least one node based on the first multimedia content; and

generating the video based on the first multimedia content and the second multimedia content.

2. The method of claim 1, wherein determining at least one node from a pre-constructed knowledge-graph based on the key phrase comprises:

determining the matching degree between the key phrases and target key phrases corresponding to target nodes in the knowledge graph; and

and if the matching degree is determined to be larger than a first preset threshold value, determining the target node as the at least one node.

3. The method of claim 1, wherein obtaining second multimedia content associated with the at least one node based on the first multimedia content comprises:

determining at least one data format not included in the first multimedia content; and

data associated with the at least one node in the at least one data format is obtained as second multimedia content.

4. The method of claim 1, wherein obtaining second multimedia content associated with the at least one node based on the first multimedia content comprises:

determining a data amount of multimedia content of at least one data format in the first multimedia content; and

if it is determined that the amount of data is less than a second predetermined threshold, data associated with the at least one node in the at least one data format is retrieved as second multimedia content.

5. The method of claim 1, wherein generating the video based on the first multimedia content and the second multimedia content comprises:

performing semantic analysis on text content in the first multimedia content and the second multimedia content to generate text elements; and

generating the video based on the text element.

6. The method of claim 5, wherein generating the video based on the text element comprises:

determining at least one of a position of the text element in a video, a word size in the text element, a display effect of the text element, and a display time of the text element to generate the video.

7. The method of claim 1, wherein generating the video based on the first multimedia content and the second multimedia content comprises:

acquiring video contents in the first multimedia content and the second multimedia content;

determining a plurality of image frames in the video content associated with the key phrase; and

generating the video based on the plurality of image frames.

8. The method of claim 7, wherein generating the video based on the image frames comprises:

determining an order of the plurality of image frames in the video and a transition effect between the plurality of image frames; and

and generating the video by using the conversion effect according to the sequence.

9. The method of claim 1, wherein the plurality of predetermined data formats includes at least one of a text data format, a picture data format, a video data format, and a sound data format.

10. An apparatus for automatically generating video, comprising:

an input receiving module configured to receive a user input comprising first multimedia content and a key phrase for describing the video, the first multimedia content having at least one of a plurality of predetermined data formats, and the first multimedia content being multimedia content for the video obtained from a knowledge-graph stored in a database;

a first node determination module configured to determine at least one node from a pre-constructed knowledge-graph based on the key phrase;

a first multimedia content acquisition module configured to acquire second multimedia content associated with the at least one node based on the first multimedia content; and

a first video generation module configured to generate the video based on the first multimedia content and the second multimedia content.

11. The apparatus of claim 10, wherein the first node determination module comprises:

a matching module configured to determine a matching degree between the key phrase and a target key phrase corresponding to a target node in the knowledge-graph; and

a second node determination module configured to determine the target node as the at least one node if it is determined that the degree of match is greater than a first predetermined threshold.

12. The device of claim 10, wherein the first multimedia content acquisition module comprises:

a data format determination module configured to determine at least one data format not included in the first multimedia content; and

a second multimedia content acquisition module configured to acquire data associated with the at least one node in the at least one data format as second multimedia content.

13. The device of claim 10, wherein the first multimedia content acquisition module comprises:

a data amount determination module configured to determine a data amount of multimedia content of at least one data format in the first multimedia content; and

a third multimedia content obtaining module configured to obtain data associated with the at least one node in the at least one data format as second multimedia content if it is determined that the amount of data is less than a second predetermined threshold.

14. The apparatus of claim 10, wherein the first video generation module comprises:

a text element generation module configured to perform semantic analysis on text content in the first multimedia content and the second multimedia content to generate text elements; and

a second video generation module configured to generate the video based on the text element.

15. The apparatus of claim 14, wherein the second video generation module comprises:

a third video generating module configured to determine at least one of a position of the text element in a video, a size of a word in the text element, a display effect of the text element, and a display time of the text element, and generate the video.

16. The apparatus of claim 10, wherein the first video generation module comprises:

a video content obtaining module configured to obtain video content of the first multimedia content and the second multimedia content;

an image frame determination module configured to determine a plurality of image frames in the video content associated with the key phrase; and

a fourth video generation module configured to generate the video based on the plurality of image frames.

17. The apparatus of claim 16, wherein the fourth video generation module comprises:

a transition effect determination module configured to determine an order of the plurality of image frames in the video and a transition effect between the plurality of image frames; and

a fifth video generation module configured to generate the video using the conversion effect in the order.

18. The apparatus of claim 10, wherein the plurality of predetermined data formats includes at least one of a text data format, a picture data format, a video data format, and a sound data format.

19. An electronic device, the electronic device comprising:

one or more processors; and

storage means for storing one or more programs which, when executed by the one or more processors, cause the one or more processors to carry out the method of any one of claims 1 to 8.

20. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1 to 9.

Images