CN117648445A - Rich media training courseware knowledge graph construction method and device and computer equipment - Google Patents

Abstract

Embodiments of the present invention disclose rich media training courseware knowledge graph construction methods, devices and computer equipment. The method includes: obtaining rich media training courseware; preprocessing the rich media training courseware to obtain a text script; and constructing a knowledge graph based on the text script. By implementing the method of the embodiment of the present invention, it is possible to establish a knowledge graph for the rich media courseware library to meet the needs of scenarios such as search and intelligent recommendation.

Description

Rich media training courseware knowledge graph construction method, device and computer equipment

Technical field

The present invention relates to a knowledge graph construction method, and more specifically to a knowledge graph construction method, device and computer equipment for rich media training courseware.

Background technique

Rich media training courseware uses text, pictures, audio, video, animation and other media elements. Training courseware produced using rich media is vivid, intuitive and easy to understand, so it is widely used in power safety education and training. Electric power training departments have generally accumulated a large amount of rich media courseware and established a rich media courseware library mainly composed of video and audio courseware. Rich media content is presented in the form of multimedia such as video and audio, rather than traditional text, and therefore cannot be directly indexed and searched. Search engines often rely on text and markup to understand and organize content, and content in rich media is not easy to interpret and analyze directly. The existing method of increasing the searchability of rich media courseware is to add metadata to videos, including adding metainformation such as titles, descriptions, keywords, and tags to videos, so that search engines can better understand the content and content of rich media courseware. theme. This solution has many shortcomings. For example, because it only contains a small part of the information in the courseware, it cannot achieve a complete search of the knowledge in the courseware; it cannot directly locate the search content to the specific location of the video, and it cannot implement semantic search.

Therefore, it is necessary to design a new method to establish a knowledge graph for rich media courseware libraries to meet the needs of search, intelligent recommendation and other scenarios.

Contents of the invention

The purpose of the present invention is to overcome the shortcomings of the existing technology and provide a knowledge graph construction method, device and computer equipment for rich media training courseware.

In order to achieve the above objectives, the present invention adopts the following technical solution: a rich media training courseware knowledge graph construction method, including:

Get rich media training courseware;

Preprocess the rich media training courseware to obtain text scripts;

Build a knowledge graph based on the text script.

The further technical solution is: preprocessing the rich media training courseware to obtain a text script, including:

Preprocess the audio courseware in the rich media training courseware to obtain the courseware timing script;

Preprocess the video courseware in the rich media training courseware to obtain voice and courseware timing scripts and image and courseware timing scripts;

The courseware timing script, the voice and courseware timing script, and the image and courseware timing script are combined to obtain a text script.

The further technical solution is: preprocessing the audio courseware in the rich media training courseware to obtain the courseware timing script, including:

For the audio courseware in the rich media training courseware, speech recognition and natural language processing technology are used to identify the human speech content, convert the human speech content into text, and generate a courseware timing script.

The further technical solution is: preprocessing the video courseware in the rich media training courseware to obtain voice and courseware timing scripts and images and courseware timing scripts, including:

Extract the video courseware audio track from the video courseware in the rich media training courseware, and use speech recognition and natural language processing technology to identify the human speech content to generate a speech and courseware timing script containing timing information;

Extract the video courseware in the rich media training courseware into pictures frame by frame;

Perform OCR recognition on designated areas of the picture to generate images and courseware timing scripts containing timing information.

The further technical solution is to perform OCR recognition on the designated area of the picture to generate images and courseware timing scripts containing timing information, including:

Perform OCR recognition on designated areas of the picture to extract subtitle information;

Repeated subtitles are removed from the subtitle information to generate images and courseware timing scripts containing timing information.

The further technical solution is: constructing a knowledge graph based on the text script, including:

Extract named entities from the text script;

Extract the relationship of the named entity from the text script;

Using the named entity and the relationship of the named entity to construct an entity relationship triplet with context;

The knowledge graph is constructed using the entity relationship triplet.

The present invention also provides a device for constructing a knowledge graph for rich media training courseware, including:

Courseware acquisition unit, used to obtain rich media training courseware;

A preprocessing unit, used to preprocess the rich media training courseware to obtain text scripts;

A building unit used to build a knowledge graph based on the text script.

Its further technical solution is: the preprocessing unit includes:

The first preprocessing subunit is used to preprocess the audio courseware in the rich media training courseware to obtain the courseware timing script;

The second preprocessing subunit is used to preprocess the video courseware in the rich media training courseware to obtain voice and courseware timing scripts and image and courseware timing scripts;

The combination subunit is used to combine the courseware timing script, the voice and courseware timing script, and the image and courseware timing script to obtain a text script.

The present invention also provides a computer device. The computer device includes a memory and a processor. A computer program is stored on the memory. When the processor executes the computer program, the above method is implemented.

The present invention also provides a storage medium that stores a computer program. When the computer program is executed by a processor, the above method is implemented.

Compared with the existing technology, the beneficial effects of the present invention are: the present invention extracts text scripts in the rich media training courseware that can reflect its content through speech recognition, image recognition, natural language processing and other methods, and based on this, constructs a time series The knowledge graph of information solves the problem of rich media courseware being unable to construct a knowledge graph because it cannot directly extract entities and entity relationships. It supports semantic search of the rich media training library, and the search results can directly locate the specific content of the rich media courseware. Location, the time series information is saved into the knowledge graph, and the search results can directly locate the specific location of the rich media courseware, realizing the establishment of a knowledge graph for the rich media courseware library to meet the needs of scenarios such as search and intelligent recommendation.

The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

Description of drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are some embodiments of the present invention, which are of great significance to this field. Ordinary technicians can also obtain other drawings based on these drawings without exerting creative work.

Figure 1 is a schematic diagram of the application scenario of the rich media training courseware knowledge graph construction method provided by the embodiment of the present invention;

Figure 2 is a schematic flow chart of a method for constructing a knowledge graph for rich media training courseware provided by an embodiment of the present invention;

Figure 3 is a schematic sub-flow diagram of a method for constructing a knowledge graph for rich media training courseware provided by an embodiment of the present invention;

Figure 4 is a schematic sub-flow diagram of a method for constructing a knowledge graph for rich media training courseware provided by an embodiment of the present invention;

Figure 5 is a schematic sub-flow diagram of a method for constructing a knowledge graph for rich media training courseware provided by an embodiment of the present invention;

Figure 6 is a schematic sub-flow diagram of a method for constructing a knowledge graph for rich media training courseware provided by an embodiment of the present invention;

Figure 7 is a schematic block diagram of a device for constructing a knowledge graph for rich media training courseware provided by an embodiment of the present invention;

Figure 8 is a schematic block diagram of the preprocessing unit of the device for building a knowledge graph of rich media training courseware provided by an embodiment of the present invention;

Figure 9 is a schematic block diagram of the second preprocessing subunit of the rich media training courseware knowledge graph construction device provided by an embodiment of the present invention;

Figure 10 is a schematic block diagram of the identification module of the device for building a knowledge graph of rich media training courseware provided by an embodiment of the present invention;

Figure 11 is a schematic block diagram of the construction unit of the rich media training courseware knowledge graph construction device provided by an embodiment of the present invention;

Figure 12 is a schematic block diagram of a computer device provided by an embodiment of the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present invention.

It should be understood that, when used in this specification and the appended claims, the terms "comprises" and "comprises" indicate the presence of described features, integers, steps, operations, elements and/or components but do not exclude the presence of one or The presence or addition of multiple other features, integers, steps, operations, elements, components and/or collections thereof.

It should also be understood that the terminology used in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in this specification and the appended claims, the singular forms "a", "an" and "the" are intended to include the plural forms unless the context clearly dictates otherwise.

It will be further understood that the term "and/or" as used in the specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items. .

Please refer to Figures 1 and 2. Figure 1 is a schematic diagram of an application scenario of the method for constructing a knowledge graph for rich media training courseware provided by an embodiment of the present invention. Figure 2 is a schematic flow chart of a method for constructing a knowledge graph of rich media training courseware provided by an embodiment of the present invention. The rich media training courseware knowledge graph construction method is applied to the server. The server interacts with the terminal in data and uses each courseware in the rich media training courseware library to build a knowledge graph. The knowledge graph contains a large number of contents contained in the rich media courseware library. Part of the knowledge can provide effective support for application scenarios such as semantic search and intelligent recommendation of rich media courseware libraries. In particular, the knowledge graph constructed using this method contains the timing information of the knowledge entity relationship appearing in the corresponding rich media courseware (that is, the specific appearance position, such as which seconds of the video courseware contains the knowledge entity). When applied to search scenarios In time, this timing information can be used to accurately locate the specific location containing the search results in the courseware. Users do not need to watch the entire courseware to manually locate the location of the search results, which effectively improves the user experience and improves search efficiency.

Figure 2 is a schematic flowchart of a method for constructing a knowledge graph of rich media training courseware provided by an embodiment of the present invention. As shown in Figure 2, the method includes the following steps S110 to S130.

S110. Obtain rich media training courseware.

In this embodiment, rich media training courseware refers to each courseware from the rich media training courseware library,

S120. Preprocess the rich media training courseware to obtain a text script.

In this embodiment, text scripts refer to text scripts extracted from audio courseware and video courseware.

In an embodiment, please refer to FIG. 3 , the above-mentioned step S120 may include steps S121 to S123.

S121. Preprocess the audio courseware in the rich media training courseware to obtain a courseware timing script.

In this embodiment, it refers to the courseware timing script. When rich media courseware generates text scripts, while saving each sentence, the corresponding timing information needs to be saved, that is, the appearance time of this sentence in the rich media courseware. The final generated The script will be a collection of <timestamp, sentence> pairs, and the specific physical storage method of the script is not limited. One implementation is to save it to a txt file in the format of "<timestamp 1, sentence 1> <timestamp 2, sentence 2>...".

Convert rich media courseware content into text scripts and create a knowledge graph based on this. In order to allow users to use the established knowledge graph to view search results, they can directly jump to the video/audio position corresponding to the search results in the rich media courseware, and build a courseware timing script.

Specifically, speech recognition and natural language processing technology are used to identify the content of human speech for the audio courseware in the rich media training courseware, convert the human speech content into text, and generate a courseware timing script.

Among them, human speech content includes narration, dialogue and other human languages.

For mp3, WAV and other audio-type training courseware, such as important leadership speeches, songs, etc., the content of this type of courseware is basically contained in the human voice part, so speech recognition technology is used to convert the human voice speech part into text and save it as The courseware timing script of the courseware. The script obtained in this way can retain most of the information of the courseware. The specific speech recognition method is an existing technology and can be implemented using, for example, the SAN-M model in Alibaba Cloud's "Intelligent Voice Interaction" service, which will not be described again here.

S122. Preprocess the video courseware in the rich media training courseware to obtain the voice and courseware timing script and the image and courseware timing script.

In this embodiment, the speech and courseware timing script refers to the courseware timing script composed of text converted from speech.

Images and courseware timing scripts refer to courseware timing scripts composed of text converted from video images.

Specifically, for video-type training courseware such as MP4, since the information is distributed in the sound and picture parts of the video, it is necessary to extract information from the two dimensions of sound and picture, and generate voice and courseware timing scripts and image and courseware timing scripts respectively. , these two types of scripts combined can retain the main information in the courseware.

In an embodiment, please refer to FIG. 4 , the above-mentioned step S122 may include steps S1221 to S1223.

S1221. Extract the video courseware audio track from the video courseware in the rich media training courseware, and use speech recognition and natural language processing technology to identify the human speech content to generate a speech and courseware timing script containing timing information.

In this embodiment, the rich media training courseware for course explanations is generally equipped with human voice narration, so the voice basically contains all the courseware information; for the video type, the human voice dialogue contains part of the courseware information. Therefore, the audio track of the courseware can be extracted, and then speech recognition and natural language processing technology can be used to identify the human speech content for processing, and generate the speech-courseware timing script of the courseware. There are many mature solutions for the specific implementation of audio track extraction, such as using the extract_audio function of the open source tool FFmpeg, which will not be described here.

S1222. Extract the video courseware in the rich media training courseware into pictures frame by frame.

In this embodiment, the information of video-based rich media training courseware is mainly contained in the screen. However, the existing artificial intelligence technology cannot accurately identify and understand the content of the screen. Considering that rich media courseware is generally used for training purposes, It is equipped with subtitles, including explanatory subtitles at the bottom and cutscene subtitles when switching key paragraphs. The function of these subtitles is to explain the image content of the courseware. Therefore, the script obtained through subtitle recognition can reflect most of the image information and can be used to build knowledge. Map.

Specifically, the video courseware is first split into pictures frame by frame, each frame corresponds to a picture, and then OCR technology is used to identify each frame of picture and extract the text information in it. There are many mature solutions for frame splitting and OCR. One implementation is to use FFmpeg to split the video into frame-by-frame pictures, and use Tesseract OCR to identify the subtitle text in the picture. I will not go into details here.

S1223. Perform OCR recognition on the designated area of the picture to generate an image and courseware timing script containing timing information.

In an embodiment, please refer to FIG. 5 , the above-mentioned step S1223 may include steps S12231 to S12232.

S12231. Perform OCR recognition on the designated area of the picture to extract subtitle information.

In this embodiment, when OCR is used to identify text in a picture to extract subtitles, in addition to subtitles, the picture will also include a large number of other irrelevant text, such as text on billboards and road signs in the video, etc. The text has little correlation with the content of the courseware, and if included in the identification, it will contaminate the script. One possible approach is to specify identification areas. Because in a rich media training courseware, subtitles generally have a fixed display area, such as a narrow rectangular area at the bottom of the screen, only the text in this area can be recognized and included in the script as subtitles. There are many mature solutions for specific implementations. One implementation method is to use the image_to_string function in the Tesseract OCR SDK package. The return value of this function after image recognition includes the area information of the text in the image, and only needs to display the given subtitles. The coordinates of the upper left corner and lower right corner of the rectangular area, that is, the subtitle text in this area is extracted from the return value.

S12232. Remove duplicate subtitles from the subtitle information to generate images and courseware timing scripts containing timing information.

In this embodiment, since the same subtitles will appear in multiple consecutive frames of pictures, the text sequence extracted in each frame of the picture needs to be compared and deduplicated with the text sequence extracted in the previous frame to prevent a large number of duplications. The text causes the script to be too large.

S123. Combine the courseware timing script, the voice and courseware timing script, and the image and courseware timing script to obtain a text script.

In this embodiment, text scripts include three types of scripts, courseware timing scripts, voice and courseware timing scripts, and image and courseware timing scripts.

S130. Construct a knowledge graph according to the text script.

Specifically, text scripts corresponding to all courseware in the rich media training library are obtained, and the text of the scripts can basically reflect the specific content of the rich media courseware. At the same time, the three types of scripts also contain specific courseware timing index information. All scripts are analyzed below to build a knowledge graph of the rich media courseware library.

Use NLP technology to extract named entities and relationships in three types of scripts; disambiguate named entities; construct entity relationship triplet SPO with contextual information; construct a knowledge graph.

In an embodiment, please refer to FIG. 6 , the above-mentioned step S130 may include steps S131 to S134.

S131. Extract named entities from the text script;

S132. Extract the relationship of the named entity from the text script;

S133. Use the named entity and the relationship of the named entity to construct an entity relationship triplet with context;

S134. Construct a knowledge graph using the entity relationship triplet.

Specifically, by performing the first stage of processing on all courseware in the rich media training courseware library, a courseware timing script library will be obtained. Each audio courseware in the rich media training courseware library corresponds to a script in the courseware timing script library. One video courseware corresponds to two (voice and courseware timing script, image and courseware timing script). These scripts are natural language texts that reflect the knowledge contained in the corresponding rich media courseware, so a knowledge graph can be constructed directly based on them.

Just use the existing mature solution of using text to build a knowledge graph. One implementation method is briefly described as follows: use the named entity recognition function provided by the natural language processing toolkit HanLP to extract candidate named entities from text scripts, and use the BERT neural network model to perform entity disambiguation to obtain named entities; use HanLP's The deep learning semantic analysis function extracts entity relationships; uses the named entities and entity relationships obtained in the above steps to construct triples, and at the same time saves the timing information into the corresponding triples, that is, the courseware number and occurrence time containing the relationship. Import the generated entity relationship group into the graph database Neo4J, save the entities as "nodes" of the graph in Neo4J, save the relationships as "relationships" in the graph, and save the timing information in the entity-relationship triplet as "relationships" in the graph " attribute. If a relationship appears multiple times in the rich media courseware library, all timing information will be saved in a set in the "Relationship" attribute. Through the above steps, the construction of the knowledge graph corresponding to the rich media courseware library can be completed.

The above-mentioned rich media training courseware knowledge graph construction method uses speech recognition, image recognition, natural language processing and other methods to extract the text scripts in the rich media training courseware that can reflect its content, and use this as a basis to build a knowledge graph with temporal information. , which solves the problem that rich media courseware cannot directly extract entities and entity relationships and cannot construct a knowledge graph. It also supports semantic search of rich media training libraries, and the search results can directly locate the specific location of rich media courseware, and integrate the time series into the rich media courseware. The information is saved in the knowledge graph, and the search results can directly locate the specific location of the rich media courseware, enabling the establishment of a knowledge graph for the rich media courseware library to meet the needs of search, intelligent recommendation and other scenarios.

FIG. 7 is a schematic block diagram of a device 300 for building a rich media training courseware knowledge graph provided by an embodiment of the present invention. As shown in Figure 7, corresponding to the above rich media training courseware knowledge graph construction method, the present invention also provides a rich media training courseware knowledge graph construction device 300. The rich media training courseware knowledge graph construction device 300 includes a unit for executing the above rich media training courseware knowledge graph construction method, and the device can be configured in a server. Specifically, please refer to FIG. 7 . The rich media training courseware knowledge graph construction device 300 includes a courseware acquisition unit 301 , a preprocessing unit 302 and a construction unit 303 .

The courseware acquisition unit 301 is used to obtain rich media training courseware; the preprocessing unit 302 is used to preprocess the rich media training courseware to obtain text scripts; the construction unit 303 is used to build a knowledge graph according to the text scripts .

In an embodiment, as shown in FIG. 8 , the preprocessing unit 302 includes a first preprocessing subunit 3021, a second preprocessing subunit 3022, and a combining subunit 3023.

The first preprocessing subunit 3021 is used to preprocess the audio courseware in the rich media training courseware to obtain the courseware timing script; the second preprocessing subunit 3022 is used to preprocess the audio courseware in the rich media training courseware. The video courseware is preprocessed to obtain the voice and courseware timing script and the image and courseware timing script; the combination subunit 3023 is used to combine the courseware timing script, the voice and courseware timing script, and the image and courseware timing script to obtain the text script. .

In one embodiment, the first preprocessing subunit 3021 is used to use speech recognition and natural language processing technology to identify the content of human speech in the audio courseware in the rich media training courseware, and convert the content of the human speech into For text, generate courseware timing scripts.

In one embodiment, as shown in Figure 9, the second preprocessing subunit 3022 includes an audio track processing module 30221, a picture extraction module 30222, and an identification module 30223.

The audio track processing module 30221 is used to extract the video courseware audio track from the video courseware in the rich media training courseware, and use speech recognition and natural language processing technology to identify the human speech content to generate speech and courseware timing scripts containing timing information. ; Picture extraction module 30222, used to extract the video courseware in the rich media training courseware into pictures frame by frame; Recognition module 30223, used to perform OCR recognition on the designated area of the picture to generate images containing timing information and Courseware timing script.

In one embodiment, as shown in Figure 10, the identification module 30223 includes a subtitle extraction sub-module 302231 and a deduplication sub-module 302232.

The subtitle extraction sub-module 302231 is used to perform OCR recognition on the designated area of the picture to extract subtitle information; the deduplication sub-module 302232 is used to remove duplicate subtitles from the subtitle information to generate an image containing timing information and Courseware timing script.

In one embodiment, as shown in Figure 11, the construction unit 303 includes an entity extraction subunit 3031, a relationship extraction subunit 3032, a triplet construction subunit 3033, and a graph construction subunit 3034.

The entity extraction subunit 3031 is used to extract named entities from the text script; the relationship extraction subunit 3032 is used to extract the relationship of the named entities from the text script; the triple construction subunit 3033 is used to use the The named entity and the relationship of the named entity construct an entity relationship triplet with context; the graph construction subunit 3034 is used to construct a knowledge graph using the entity relationship triplet.

It should be noted that those skilled in the art can clearly understand that for the specific implementation process of the above-mentioned rich media training courseware knowledge graph construction device 300 and each unit, reference can be made to the corresponding descriptions in the foregoing method embodiments. For the convenience of description and It’s concise and I won’t go into details here.

The above-mentioned rich media training courseware knowledge graph construction device 300 can be implemented in the form of a computer program, and the computer program can run on the computer device as shown in Figure 12.

Please refer to Figure 12, which is a schematic block diagram of a computer device provided by an embodiment of the present application. The computer device 500 may be a server, where the server may be an independent server or a server cluster composed of multiple servers.

Referring to Figure 12, the computer device 500 includes a processor 502, a memory and a network interface 505 connected through a system bus 501, where the memory may include a non-volatile storage medium 503 and an internal memory 504.

The non-volatile storage medium 503 can store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions. When executed, the program instructions can cause the processor 502 to execute a rich media training courseware knowledge graph construction method.

The processor 502 is used to provide computing and control capabilities to support the operation of the entire computer device 500 .

The internal memory 504 provides an environment for the execution of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, it can cause the processor 502 to execute a rich media training courseware knowledge map construction method.

The network interface 505 is used for network communication with other devices. Those skilled in the art can understand that the structure shown in Figure 12 is only a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device 500 to which the solution of the present application is applied. The specific computer device 500 may include more or fewer components than shown, some combinations of components, or a different arrangement of components.

Wherein, the processor 502 is used to run the computer program 5032 stored in the memory to implement the following steps:

Obtain rich media training courseware; preprocess the rich media training courseware to obtain a text script; and construct a knowledge graph based on the text script.

In one embodiment, when the processor 502 implements the step of preprocessing the rich media training courseware to obtain a text script, the processor 502 specifically implements the following steps:

Preprocess the audio courseware in the rich media training courseware to obtain the courseware timing script; preprocess the video courseware in the rich media training courseware to obtain the voice and courseware timing script and the image and courseware timing script; The courseware timing script, the voice and courseware timing script, and the image and courseware timing script are combined to obtain a text script.

In one embodiment, when the processor 502 implements the step of preprocessing the audio courseware in the rich media training courseware to obtain the courseware timing script, the processor 502 specifically implements the following steps:

For the audio courseware in the rich media training courseware, speech recognition and natural language processing technology are used to identify the human speech content, convert the human speech content into text, and generate a courseware timing script.

In one embodiment, when the processor 502 implements the step of preprocessing the video courseware in the rich media training courseware to obtain the voice and courseware timing scripts and the image and courseware timing scripts, the processor 502 specifically implements the following steps:

Extract the video courseware audio track from the video courseware in the rich media training courseware, and use speech recognition and natural language processing technology to identify the human speech content to generate a speech and courseware timing script containing timing information; The video courseware in the system is extracted into pictures frame by frame; OCR recognition is performed on the designated area of the picture to generate images and courseware timing scripts containing timing information.

In one embodiment, when the processor 502 implements the step of performing OCR recognition on a designated area of the picture to generate an image and courseware timing script containing timing information, the processor 502 specifically implements the following steps:

Perform OCR recognition on designated areas of the picture to extract subtitle information; remove duplicate subtitles from the subtitle information to generate images and courseware timing scripts containing timing information.

In one embodiment, when the processor 502 implements the step of constructing a knowledge graph based on the text script, the processor 502 specifically implements the following steps:

Extract named entities from the text script; extract the relationship of the named entities from the text script; use the named entities and the relationship of the named entities to construct an entity relationship triplet with context; use the entity relationship Triplets build knowledge graphs.

It should be understood that in the embodiment of the present application, the processor 502 may be a central processing unit (Central Processing Unit, CPU). The processor 502 may also be other general-purpose processors, digital signal processors (Digital Signal Processor, DSP), special integrated processors, etc. Circuit (Application Specific Integrated Circuit, ASIC), off-the-shelf programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general processor may be a microprocessor or the processor may be any conventional processor.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of implementing the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program includes program instructions, and the computer program can be stored in a storage medium, and the storage medium is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiments of the above method.

Therefore, the present invention also provides a storage medium. The storage medium may be a computer-readable storage medium. The storage medium stores a computer program, wherein when the computer program is executed by the processor, it causes the processor to perform the following steps:

Obtain rich media training courseware; preprocess the rich media training courseware to obtain a text script; and construct a knowledge graph based on the text script.

In one embodiment, when the processor executes the computer program to implement the step of preprocessing the rich media training courseware to obtain a text script, the processor specifically implements the following steps:

Preprocess the audio courseware in the rich media training courseware to obtain the courseware timing script; preprocess the video courseware in the rich media training courseware to obtain the voice and courseware timing script and the image and courseware timing script; The courseware timing script, the voice and courseware timing script, and the image and courseware timing script are combined to obtain a text script.

In one embodiment, when the processor executes the computer program to implement the step of preprocessing the audio courseware in the rich media training courseware to obtain the courseware timing script, the processor specifically implements the following steps:

For the audio courseware in the rich media training courseware, speech recognition and natural language processing technology are used to identify the human speech content, convert the human speech content into text, and generate a courseware timing script.

In one embodiment, when the processor executes the computer program to implement the step of preprocessing the video courseware in the rich media training courseware to obtain the voice and courseware timing script and the image and courseware timing script, , specifically implement the following steps:

Extract the video courseware audio track from the video courseware in the rich media training courseware, and use speech recognition and natural language processing technology to identify the human speech content to generate a speech and courseware timing script containing timing information; The video courseware in the system is extracted into pictures frame by frame; OCR recognition is performed on the designated area of the picture to generate images and courseware timing scripts containing timing information.

In one embodiment, when the processor executes the computer program to implement the **** step, the processor specifically implements the following steps:

In one embodiment, when the processor executes the computer program to implement the OCR recognition of a designated area of the picture to generate an image and courseware timing script containing timing information, the processor specifically implements the following steps:

Perform OCR recognition on designated areas of the picture to extract subtitle information; remove duplicate subtitles from the subtitle information to generate images and courseware timing scripts containing timing information.

In one embodiment, when the processor executes the computer program to implement the step of constructing a knowledge graph based on the text script, the processor specifically implements the following steps:

Extract named entities from the text script; extract the relationship of the named entities from the text script; use the named entities and the relationship of the named entities to construct an entity relationship triplet with context; use the entity relationship Triplets build knowledge graphs.

The storage medium may be a U disk, a mobile hard disk, a read-only memory (Read-Only Memory, ROM), a magnetic disk or an optical disk, and other computer-readable storage media that can store program codes.

Those of ordinary skill in the art can appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, computer software, or a combination of both. In order to clearly illustrate the relationship between hardware and software Interchangeability, in the above description, the composition and steps of each example have been generally described according to functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each specific application, but such implementations should not be considered to be beyond the scope of the present invention.

In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of each unit is only a logical function division, and there may be other division methods during actual implementation. For example multiple units or components may be combined or integrated into another system, or some features may be omitted, or not implemented.

The steps in the methods of the embodiments of the present invention can be sequence adjusted, combined, and deleted according to actual needs. The units in the device of the embodiment of the present invention can be merged, divided and deleted according to actual needs. In addition, each functional unit in various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit.

If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention is essentially or contributes to the existing technology, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium , including several instructions to cause a computer device (which can be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the method described in various embodiments of the present invention.

The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person familiar with the technical field can easily think of various equivalent methods within the technical scope disclosed in the present invention. Modifications or substitutions shall be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims (10)

1. Rich media training courseware knowledge graph construction method, which is characterized by including: Get rich media training courseware; Preprocess the rich media training courseware to obtain text scripts; Build a knowledge graph based on the text script. 2. The method for constructing a knowledge graph of rich media training courseware according to claim 1, characterized in that the preprocessing of the rich media training courseware to obtain a text script includes: Preprocess the audio courseware in the rich media training courseware to obtain the courseware timing script; Preprocess the video courseware in the rich media training courseware to obtain voice and courseware timing scripts and image and courseware timing scripts; The courseware timing script, the voice and courseware timing script, and the image and courseware timing script are combined to obtain a text script. 3. The rich media training courseware knowledge graph construction method according to claim 2, characterized in that the preprocessing of the audio courseware in the rich media training courseware to obtain the courseware timing script includes: For the audio courseware in the rich media training courseware, speech recognition and natural language processing technology are used to identify the human speech content, convert the human speech content into text, and generate a courseware timing script. 4. The knowledge graph construction method of rich media training courseware according to claim 2, characterized in that the video courseware in the rich media training courseware is preprocessed to obtain voice and courseware timing scripts and images and courseware Timing scripts, including: Extract the video courseware audio track from the video courseware in the rich media training courseware, and use speech recognition and natural language processing technology to identify the human speech content to generate a speech and courseware timing script containing timing information; Extract the video courseware in the rich media training courseware into pictures frame by frame; Perform OCR recognition on designated areas of the picture to generate images and courseware timing scripts containing timing information. 5. The rich media training courseware knowledge map construction method according to claim 4, characterized in that the OCR recognition is performed on the designated area of the picture to generate an image and courseware timing script containing timing information, including: Perform OCR recognition on designated areas of the picture to extract subtitle information; Repeated subtitles are removed from the subtitle information to generate images and courseware timing scripts containing timing information. 6. The method for constructing a knowledge graph of rich media training courseware according to claim 1, characterized in that said constructing a knowledge graph according to the text script includes: Extract named entities from the text script; Extract the relationship of the named entity from the text script; Using the named entity and the relationship of the named entity to construct an entity relationship triplet with context; The knowledge graph is constructed using the entity relationship triplet. 7. Rich media training courseware knowledge graph construction device, which is characterized by including: Courseware acquisition unit, used to obtain rich media training courseware; A preprocessing unit, used to preprocess the rich media training courseware to obtain text scripts; A building unit used to build a knowledge graph based on the text script. 8. The rich media training courseware knowledge graph construction device according to claim 7, characterized in that the preprocessing unit includes: The first preprocessing subunit is used to preprocess the audio courseware in the rich media training courseware to obtain the courseware timing script; The second preprocessing subunit is used to preprocess the video courseware in the rich media training courseware to obtain voice and courseware timing scripts and image and courseware timing scripts; The combination subunit is used to combine the courseware timing script, the voice and courseware timing script, and the image and courseware timing script to obtain a text script. 9. A computer device, characterized in that the computer device includes a memory and a processor, a computer program is stored on the memory, and when the processor executes the computer program, it implements any one of claims 1 to 6 method described in the item. 10. A storage medium, characterized in that the storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.