TWI790630B - Method and device for automatically generating notes - Google Patents

Abstract

A method and a device for automatically generating notes are provided. The method for automatically generating notes includes the following steps. When a lecturer gives an explanation through a slide, a video and a voice are recorded. A graphic-to-text process is performed on the video to obtain a slide text data. A graphic capture process is performed on the video to obtain a slide graphic data. A voice-to-text process is performed on the voice to obtain a voice text data. The voice text data is classified into several paragraphs. Several summary texts are analyzed from the paragraphs. Several keywords are analyzed based on the slide text data and the voice text data. The slide text data, the slide graphic data, the voice text data, the summary texts and the keywords are integrated.

Description

Method and device for automatically generating notes

The present disclosure relates to an information processing method and device, and in particular to a method and device for automatically generating notes.

A considerable amount of information can be obtained when attending an educational training or listening to a presentation. However, ignorant copying of notes not only affects thinking, but also is not easy to organize after class.

We can find that some lecturers' slides can complement the lecture content, but some lecturers only put some essentials in the slides. Simple video or audio recordings are difficult to sort out, and it is also impossible to clearly obtain the key points of the lectures.

In order to make learning more efficient, researchers are working on developing a technology that automatically generates notes, so that listeners can pay more attention to lectures during class, and can quickly get the key points of sorting out after class.

This disclosure is about a method and device for automatically generating notes, which converts and analyzes video and audio data to obtain summary text and keywords. In this way, the listeners can pay more attention to the lectures during class, and can quickly obtain the key points after class, greatly improving learning efficiency.

According to an aspect of the present disclosure, a method for automatically generating notes is provided. The method for automatically generating notes includes the following steps. When a lecturer is giving an explanation through a slideshow, video and audio recording are performed to obtain a video data and an audio data. Performing a graphics-to-text program on the video data to obtain a slideshow text data. A graphic capture procedure is performed on the video data to obtain a slide graphic data. A voice-to-text program is performed on the recorded data to obtain a voice-to-text data. Classify audio-text data into several paragraphs. Several summary texts are analyzed for these paragraphs respectively. Several keywords are analyzed for the powerpoint text data and the audio text data. Collect the powerpoint text data, powerpoint graphic data, audio text data, these summary texts and these keywords.

According to another aspect of the present disclosure, a device for automatically generating notes is provided. The device for automatically generating notes includes a video recording device, a recording device, a graphics-to-text unit, a graphics extraction unit, a voice-to-text unit, a voice-to-text classification unit, a summary unit, a keyword analysis unit and a compilation unit. The video recording device is used for video recording when a lecturer is explaining through a slide film, so as to obtain a video data. The recording device is used for recording when the lecturer gives an explanation through the slide, so as to obtain a recording material. The graphics-to-text unit performs a graphics-to-text program on the video data to obtain a slideshow text data. The graphic capture unit is used for performing a graphic capture process on the video data to obtain a slide graphic data. The speech-to-text unit is used for performing a speech-to-text program on the recording data to obtain a speech-to-text data. The audio-text classification unit is used to classify the audio-text data into several paragraphs. The summary unit is used to analyze several summary texts respectively for these paragraphs. The keyword analysis unit is used to analyze several keywords for the slide text data and the voice text data. The integration unit is used for integrating the powerpoint text data, the powerpoint graphic data, the audio text data, the abstract text and the keywords.

In order to have a better understanding of the above and other aspects of the present disclosure, the following specific embodiments are described in detail in conjunction with the attached drawings as follows:

100: A device for automatically generating notes

110:Video installation

120: Recording device

130: Graphics to text unit

140: graphics capture unit

150:Speech-to-text unit

160:Phonetic text taxonomy

170:Summary unit

180: Keyword Analysis Unit

190: Integration unit

DW: slide graphics data

KW: Keyword

PG: page

S101, S102, S103.S104, S105, S106, S107, S108, S109, S110: steps

SG: Paragraph

SM: Summary text

SP: Speech text data

TI: Time Information

TX: slide text data

TX1: printed text

TX2: handwritten text

VC: recording material

VD: video data

Figure 1 shows a schematic diagram of a slideshow and a lecturer's explanation according to an embodiment

FIG. 2 shows a block diagram of a device for automatically generating notes according to an embodiment.

FIG. 3 shows a flowchart of a method for automatically generating notes according to an embodiment.

Figures 4~6 illustrate the steps in Figure 3.

Please refer to FIG. 1 , which shows a schematic diagram of a slideshow and a lecturer's explanation according to an embodiment. When the lecturer is explaining, explain to the page PG of the slideshow. There will be slide text data TX and slide graphics data DW on the page PG. The slide text data TX includes a printed text TX1 and a handwritten text TX2. Following the lecturer's explanation, the speech and text data SP will be generated. These materials need to be recorded in the notes Content. In this embodiment, these data will be automatically analyzed and integrated to further analyze the abstract text SM and keywords KW.

Please refer to FIG. 2 , which shows a block diagram of a device 100 for automatically generating notes according to an embodiment. The device 100 for automatically generating notes includes a video recording device 110, a recording device 120, a graphics-to-text unit 130, a graphics capture unit 140, a voice-to-text unit 150, a voice-to-text classification unit 160, a summary unit 170, A keyword analysis unit 180 and a collection unit 190 . The video recording device 110 is used for performing a video recording procedure. The recording device 120 is used for recording procedure. The video recording device 110 and the recording device 120 are, for example, integrated in the same electronic device and operate synchronously. The video recording device 110 and the recording device 120 are, for example, a smart phone, a video camera, a tablet computer or a notebook computer. The graphics-to-text unit 130, the graphics extraction unit 140, the voice-to-text unit 150, the voice-to-text classification unit 160, the summary unit 170, the keyword analysis unit 180, and the collection unit 190 are used to perform various specific processing procedures, such as a chip, A circuit, a circuit board, a program code or a storage device for storing the program code. In this embodiment, the video data VD obtained by the video recording device 110 and the recording data VC obtained by the recording device 120 pass through the graphics-to-text unit 130, the graphics extraction unit 140, the voice-to-text unit 150, the voice-to-text classification unit 160, After conversion and analysis by the summary unit 170 and the keyword analysis unit 180, the summary text SM and keywords KW can be obtained. The operation of each component is described in detail below with a flow chart.

Please refer to Figures 3-6. Figure 3 shows a flow chart of a method for automatically generating notes according to an embodiment, and Figures 4-6 illustrate the steps in Figure 3. in step In step S101 , the video recording device 110 and the recording device 120 perform video and audio recording when a lecturer gives an explanation through a slide, so as to obtain video data VD and audio data VC. When the video data VD and the audio data VC are recorded, the video data VD and the audio data VC are simultaneously marked with a time information TI. The time information TI is, for example, an absolute time or a relative time. Absolute time is eg Greenwich Mean Time. The relative time is, for example, the time relative to the reference time point after the video recording device 110 and the recording device 120 are started at the same reference time point.

Next, in step S102 , the graphics-to-text unit 130 performs a graphics-to-text program on the video data VD to obtain the slideshow text data TX. For example, the graphics-to-text unit 130 analyzes that several consecutive frames display the same page PG, and some pages PG may be partially covered by the speaker. The graphics-to-text unit 130 selects or overlaps a complete page PG from these pages PG. Then analyze the slideshow text data TX from this complete page PG.

In this step, the image-to-text unit 130 further marks a time information TI on the slide text data TX. The graphics-to-text unit 130 marks, for example, the appearance time interval of the page PG in each sentence.

The image-to-text unit 130 can classify the slideshow text data TX according to the pages PG according to the time information TI. For example, as shown in FIG. 4, the powerpoint text data TX is classified into the pages PG where it appears according to these pages PG.

In this step, the image-to-text unit 130 classifies the slide text data TX into printed text TX1 (shown in FIG. 1 ) and handwritten text TX2 (shown in FIG. 1 ). The graphics-to-text unit 130 can distinguish the printed text TX1 and the handwritten text TX2 according to the regularity of the text and the variation of the straight line. or, graphics The text-to-text unit 130 can first analyze the font, and then distinguish the printed text TX1 and the handwritten text TX2 according to the font. Alternatively, the image-to-text unit 130 may use a machine learning algorithm to identify the printed text TX1 and the handwritten text TX2.

Then, in step S103 , the graphics capture unit 140 performs a graphics capture procedure on the video data VD to obtain the slide graphics data DW. In this step, for example, the graphic capture unit 140 analyzes that several consecutive frames display the same page PG, and some pages PG may be partially covered by the speaker. The graphics-to-text unit 130 selects or overlaps a complete page PG from these pages PG. Then analyze the graphic data DW of the slideshow from the complete page PG.

The graphic capture unit 140 can perform edge detection on the page PG to analyze several objects. From these objects, the graphics data DW of the slideshow is analyzed.

In this step, the image capture unit 140 further marks the time information TI on the slide image data DW. The graphic capture unit 140 marks the appearance time interval of the page PG for each object, for example.

The graphics retrieval unit 140 can classify the slide graphics data DW according to the pages PG according to the time information TI. For example, as shown in FIG. 4, the slide graphics data DW are classified into the pages PG where they appear according to these pages PG.

Next, in step S104 , the speech-to-text unit 150 performs a speech-to-text program on the recorded data VC to obtain the speech-to-text data SP. For example, the speech-to-text unit 150 analyzes the analysis recording data VC word by word according to a preset language type to obtain several words. Speech to text list Yuan 150 converts these single characters into words again. Finally, segment these continuous words into several sentences to obtain the speech and text data SP.

In this step, the speech-to-text unit 150 further marks the time information TI on the speech-to-text data SP. The speech-to-text unit 150, for example, marks the occurrence time interval of the page PG in each sentence.

The speech-to-text unit 150 can classify the speech-to-text data SP according to the pages PG according to the time information TI. For example, as shown in FIG. 4, the speech and text data SP are classified into the pages PG where they appear according to these pages PG.

Then, in step S105 , as shown in FIG. 4 , the phonetic text classification unit 160 classifies the phonetic text data SP into a plurality of paragraphs SG. In this step, the phonetic text classification unit 160 classifies the phonetic text data SP into several segments SG through a K-nearest neighbors (KNN) algorithm.

Next, in step S106 , as shown in FIG. 5 , the phonetic text classification unit 160 adjusts two adjacent sentences corresponding to the phonetic text data SP of two adjacent pages PG according to the slide text data SP. For example, the phonetic text classification unit 160 can determine whether two adjacent sentences of the phonetic text data SP of two adjacent pages PG belong to the same paragraph SG through the K-nearest neighbor algorithm. If they belong to the same paragraph SG, the two sentences are classified as the same paragraph SG of the same page PG.

Next, in step S107 , the summary unit 170 analyzes several summary texts SM for these paragraphs SG respectively. In this step, the summary unit 170 can filter out some superfluous words to obtain the summary text SM.

Then, in step S108 , as shown in FIG. 6 , the keyword analysis unit 180 analyzes several keywords KW for the slide text data TX and the phonetic text data SP. In this step, the keyword analysis unit 180 compares the slide text The data TX and the phonetic text data SP are used to analyze the words that appear in the slideshow text data TX and the phonetic text data SP at the same time, and use these words as keywords KW.

Next, in step S109 , as shown in FIG. 6 , the keyword analysis unit 180 sorts the keywords KW according to their frequency of occurrence. For example, keywords KW with higher occurrence frequency will be sorted first. The keyword analysis unit 180 can sort each page PG once, so that users can quickly know the key content of the page PG.

Then, in step S110 , as shown in FIG. 6 , the integration unit 190 aggregates the powerpoint text data TX, the powerpoint graphic data DW, the phonetic text data SP, the abstract text SM and the keywords KW. Fig. 6 presents a collection method, but the present invention is not limited to the example in Fig. 6 .

According to the above-mentioned embodiment, in the technique of automatically generating notes, after the video data VD and the audio data VC are converted and analyzed, the summary text SM and the keywords KW can be obtained. In this way, the listeners can pay more attention to the lectures during class, and can quickly obtain the key points after class, greatly improving learning efficiency.

To sum up, although the present disclosure has been disclosed above with embodiments, it is not intended to limit the present disclosure. Those with ordinary knowledge in the technical field to which this disclosure belongs may make various changes and modifications without departing from the spirit and scope of this disclosure. Therefore, the scope of protection of this disclosure should be defined by the scope of the appended patent application.

100: A device for automatically generating notes

110:Video installation

120: Recording device

130: Graphics to text unit

140: graphics capture unit

150:Speech-to-text unit

160:Phonetic text taxonomy

170:Summary unit

180: Keyword Analysis Unit

190: Integration unit

DW: slide graphics data

KW: Keyword

SG: Paragraph

SM: Summary Text

SP: Speech text data

TI: Time Information

TX: slide text data

VC: recording material

VD: video data

Claims (10)

A method for automatically generating notes, the method for automatically generating notes is executed by a computer, the method for automatically generating notes includes: when a lecturer is explaining through a slide, video and audio recording are performed, so as to obtain a video data and an audio recording data; perform a graphics-to-text program on the video data to obtain a powerpoint text data; perform a graphics extraction process on the video data to obtain a powerpoint graphics data; perform a voice-to-text program on the audio data , to obtain a phonetic text data; classify the phonetic text data into a plurality of paragraphs, and classify the phonetic text data into these paragraphs through a K-nearest neighbor algorithm (k-nearest neighbors, KNN); filter out the paragraphs superfluous words, so as to separately analyze a plurality of summary texts; analyze a plurality of keywords for the slide text data and the audio text data, and sort these keywords according to the frequency of occurrence of these keywords; and compile the projection A piece of text data, the slide graphic data, the audio text data, the summary text and the keywords. The method for automatically generating notes as described in Claim 1, wherein the slideshow includes a plurality of pages, In the step of obtaining the text data of the slide, the text data of the slide is classified according to the pages; in the step of obtaining the graphic data of the slide, the graphic data of the slide is classified according to the pages; In the step of text data, the audio text data is classified according to the pages. The method for automatically generating notes as described in claim 2 further includes: adjusting two adjacent sentences of the phonetic text data corresponding to two adjacent pages according to the slide text data. The method for automatically generating notes as described in Claim 1, wherein in the step of obtaining the slideshow text data, the slideshow text data is classified into a printed text and a handwritten text. The method for automatically generating notes as described in claim item 1, wherein in the step of obtaining the slide text data, a time information is further marked on the slide text data; in the step of obtaining the slide graphics data, further in Marking the time information on the slide graphic data; and marking the time information on the audio text data in the step of obtaining the audio text data. A device for automatically generating notes, comprising: a video recording device for recording when a lecturer is giving an explanation through a slide to obtain a video data; a recording device for giving an explanation by the lecturer through the slide recording, to obtain a recording data; a graphics-to-text unit, which is used to perform a graphics-to-text program on the video data, to obtain a powerpoint text data; a graphics capture unit, to perform a graphics conversion on the video data A graphics extraction program, to obtain a slide graphics data; a voice-to-text unit, to perform a voice-to-text program on the recording data, to obtain a voice-text data; a voice-to-text classification unit, to the The phonetic text data is classified into a plurality of paragraphs, and the phonetic text data is classified into these paragraphs through a K-nearest neighbors (KNN) algorithm; a summary unit is used to filter redundant words for these paragraphs, to Analyzing a plurality of summary texts respectively; a keyword analysis unit, which is used to analyze a plurality of keywords for the slide text data and the audio text data, and sort the keywords according to the frequency of occurrence of the keywords; and an integration unit for integrating the slide text data, the slide graphics data, the voice text data, the summary text and the keywords. The device for automatically generating notes as described in claim 6, wherein the slideshow includes a plurality of pages, The graphic-to-text unit classifies the slideshow text data according to the pages; the graphic capture unit classifies the slideshow graphic data according to the pages; the speech-to-text unit classifies the phonetic text data according to the pages. The device for automatically generating notes as described in Claim 7, wherein the phonetic text classification unit adjusts two adjacent sentences of the phonetic text data corresponding to two adjacent pages according to the slideshow text data. The device for automatically generating notes as described in Claim 6, wherein the image-to-text unit classifies the slide text data into a printed text and a handwritten text. The device for automatically generating notes as described in claim item 6, wherein the graphics-to-text unit continuously marks a time information on the slide text data; the graphics capture unit continuously marks the time information on the slide graphics data; The speech-to-text unit continuously marks the time information on the speech-to-text data.

Images