KR20230102475A - Method for summarizing minutes for an online meeting - Google Patents

Abstract

A method of automatically summarizing minutes of a non-face-to-face meeting based on the type of meeting is provided. In the method for summarizing meeting minutes of a non-face-to-face meeting according to an embodiment of the present invention, for a non-face-to-face meeting of a predetermined type, a user's intention can be understood by a machine by natural language processing of user utterances in a meeting text obtained by TTS conversion of audio during a meeting. Speech understanding step of converting into a semantic structure; a conversation management step of extracting an optimal keyword according to the conference type based on the semantic structure of the user's speech converted in the speech understanding step and managing the conversation; and a summary generating step of generating a summary according to a summary template according to the conference type based on the information generated in the conversation management step.

Description

Method for summarizing minutes for an online meeting}

The present invention relates to a method for summarizing minutes of a non-face-to-face meeting, and more particularly, to a method for automatically summarizing minutes of a non-face-to-face meeting based on a meeting type.

Since the pandemic, non-face-to-face video conferencing is rapidly increasing, and various video conferencing solutions for this are emerging. Some solutions are now capable of editing transcripts from video conferences to create multimedia meeting minutes that include video, audio and text.

Looking at the process of generating meeting minutes through a conventional video conferencing system, video and audio from the start to the end of a meeting are recorded and recorded, respectively, to collect data necessary for generating meeting minutes, and the collected data is stored by type. The stored data is collected based on time, and the user creates meeting minutes by editing remarks and keywords from the collected data.

Meanwhile, in addition to these basic functions, attempts have been made to add various functions related to meeting minutes.

Registered Patent No. 10-2061291 proposes a configuration in which conference participants are identified by analyzing the volume data of the participant's standard volume and the volume of the acoustic signal, and the content of the speech is recorded.

Patent Registration No. 10-1290439 extracts important sentences from meeting minutes and identifies the relationship between these sentences to create a summary of meeting minutes in a tree structure. Specifically, a set of subject sentences is generated by separating sentences by subject of the meeting based on the speeches of the meeting moderator in the meeting minutes, and the subject sentence and other sentences are based on the words used in the sentence in the generated subject sentence set. The similarity between the sentences is calculated and the important sentences are extracted accordingly. Thereafter, the important sentences are expressed in a tree structure according to the subordinate relationship of the extracted important sentences. However, it is difficult to apply in the case of a meeting where there is no separate meeting host or it is difficult to distinguish. In addition, even in the case where there is a meeting moderator, it is difficult to apply when remarks on the subject come from participants other than the meeting moderator.

An object of the present invention is to provide a method for automatically summarizing meeting minutes of a non-face-to-face meeting. Another object of the present invention is to provide a method for summarizing meeting minutes even when there is no meeting moderator or it is difficult to distinguish them. Another object of the present invention is to provide a method for summarizing meeting minutes of a non-face-to-face meeting that can be applied to various types of meetings.

In the method for summarizing meeting minutes of a non-face-to-face meeting according to an embodiment of the present invention, for a non-face-to-face meeting of a predetermined type, a user's intention can be understood by a machine by natural language processing of user utterances in a meeting text obtained by TTS conversion of audio during a meeting. Speech understanding step of converting into a semantic structure; a conversation management step of extracting an optimal keyword according to the conference type based on the semantic structure of the user's speech converted in the speech understanding step and managing the conversation; and a summary generating step of generating a summary according to a summary template according to the conference type based on the information generated in the conversation management step.

In one embodiment, the method for summarizing meeting minutes of a non-face-to-face meeting of the present invention may be performed in a non-face-to-face conference server. The non-face-to-face conference server receives conference information including the conference type, conference start time, and duration of the conference from the conference originator and generates an invitation code to participate in the conference; converting the conference voice into text using a Text to Speech (TTS) function when the conference starts; an utterance understanding step of converting the user's intention into a semantic structure that can be understood by a machine by natural language processing of the user's utterance in the text; a conversation management step of extracting an optimal keyword according to the conference type based on the semantic structure of the user's speech converted in the speech understanding step and managing the conversation; And based on the information generated in the conversation management step, a summary generating step of generating a summary according to a summary template according to the conference type is performed to summarize the meeting minutes of the non-face-to-face meeting.

The speech understanding step includes a morpheme analysis step of identifying morphemes of sentences spoken by a user, an entity name extraction step of identifying and extracting entity names of user-spoken sentences by searching entity names corresponding to each morpheme, and identifying morphemes and entity names. A dialogue act and intention analysis step of identifying an intention of a sentence spoken by a user by analyzing action information through dialogue act analysis using the information may be included.

In the dialogue management step, a summary object extraction step of recombining semantic components defined according to the conference type and a plurality of dialogue scenarios prepared according to the conference type are performed according to the flow of multilateral speech. A context-based dialogue structuring step may be included. The semantic component may be composed of a combination of the extracted entity name, dialogue act, and intention, and the entity name, dialogue act, and intention are defined according to the conference type.

The summary generation step applies an automatic summary model that adjusts the elements and units constituting the meeting minutes according to the meeting type, aggregates the topics and contents analyzed in each context, and outputs them in accordance with the template format corresponding to the meeting type. can be performed by The automatic summary model aggregates topics and contents in units of contexts of utterances and outputs them according to the template format.

In the dialogue act and intention analysis step, an intention/dialogue act knowledge base may be used. The intention/dialogue act knowledge base may be constructed and strengthened by conducting error correction learning through intention/dialogue act learning data and automatic summary failure data that has not been extracted or incorrectly extracted after the summarization step.

In one embodiment, the speech understanding step, the dialogue management step, and the summary generation step may be sequentially performed during the conference. In another embodiment, at least a part of the speech understanding step, the dialog management step, and the summary generating step may be performed after the conference ends.

According to the present invention, since optimal keywords are extracted according to conference types, conversations are managed, and summaries are generated according to summary templates according to conference types, meeting summary summaries can be efficiently generated for various types of conferences. In addition, according to the present invention, a summary of meeting minutes can be generated more accurately by selecting keywords in consideration of intention, dialogue act, and context transition relationship according to the type of meeting and extracting summary text.

1 is a network diagram for explaining a method of summarizing meeting minutes of a non-face-to-face meeting of the present invention.
2 is a flowchart illustrating one example in which the method of the present invention is applied in the course of a non-face-to-face meeting.
3 is a flow chart showing a meeting minutes summary process.
4 is a flowchart showing operations in the speech understanding step.
5 is a flowchart showing operations in the dialog management step.
6 is an example of a screen when accessing a non-face-to-face conference.
7 is an example of a screen in the middle of summarizing meeting minutes.
8 is an example of a completed meeting minutes summary.
9 is an example of a scenario.
10 shows an example of a dialogue act analysis result of a psychological counseling type meeting.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings.

1 is a network diagram for explaining a method of summarizing meeting minutes of a non-face-to-face meeting of the present invention.

One of the conference participants connects to the non-face-to-face conference server 20 and establishes a conference. In the present invention, a conference type can be set when opening a conference. When the meeting creator sends a link to access the opened meeting to the meeting participants, the meeting participants use terminals 30a, .., 30n such as personal computers, tablet PCs, and smartphones to connect non-face-to-face through the link. Connect to the conference server (20).

When a meeting starts, the meeting record maker 10 records the meeting by video recording or audio recording, and converts the meeting voice into text using a Text to Speech (TTS) function. The converted text undergoes morpheme analysis, object name extraction, dialogue act and intention analysis, etc., and the meeting minutes writing device 10 generates a meeting minutes summary according to the set meeting type. The process of generating a summary of meeting minutes may be performed sequentially during the meeting or may be performed collectively after the meeting ends. On the other hand, although the meeting minutes preparation device 10 is shown as a separate device from the non-face-to-face conference server 20 in FIG. 1, it is also possible to implement it as a program that operates within the non-face-to-face conference server 20.

2 is a flowchart illustrating one example in which the method of the present invention is applied in the course of a non-face-to-face meeting.

The conference establishment establishes a conference by accessing the non-face-to-face conference server 20 . An example of the screen at this time is shown in FIG. 6 . In the present invention, a conference type can be set when opening a conference. The conference organizer sets the conference type by selecting the conference type selection button 61 on the screen (a) of FIG. 6 (step S10). When the conference type selection button 61 is pressed, a selectable conference type list 63 is displayed as shown in FIG. 6(b). On this screen, if the meeting organizer selects one type, a meeting minutes summary is created for this meeting according to the selected type.

The meeting organizer inputs the meeting name into the meeting name input field 62 . Meanwhile, although not shown in FIG. 6 , information such as the start time of the meeting and the duration of the meeting is also input in addition to the meeting name when establishing the meeting (step S20). After inputting conference information, the conference organizer presses the invitation code copy button 64 to copy the invitation code and deliver it to conference participants. When the meeting initiator presses the meeting start button 65 near the meeting start time, the meeting starts (step S30).

When the conference starts, the non-face-to-face conference server 20 records a video or voice conversation of the ongoing conference, and converts the voice into text using a Text to Speech (TTS) function (step S40). It is also possible to display the conversation contents of the conference as subtitles on the main screen of the conference using the converted text according to the selection of the conference organizer or the conference participant.

When the meeting ends, the recorded or recorded file and the TTS-converted text are saved as files (step S50). Then, a summary of the meeting minutes is generated using the stored file, and the summary of the meeting minutes as shown in FIG. 8 is presented to the meeting attendees so that they can be modified (step S60). Depending on the embodiment, the process of generating a summary of meeting minutes may be sequentially performed during a meeting. In this case, as the meeting progresses, as shown in the example shown in FIG. 7, the gist of the meeting participant's remarks is added to the meeting minutes summary template. When reviewing the generated summary, it may be configured to reproduce the recorded contents by clicking the icon 71 in the shape of a recording tape. In addition, it is also possible to configure the recorded contents of the corresponding part to be reproduced by clicking on the summarized part in the summary of FIG. 8 .

Next, with reference to FIGS. 3 to 5 , a process of generating a meeting minutes summary will be described in detail. 3 is a flow chart showing the process of summarizing meeting minutes, FIG. 4 is a flow chart showing operations in the speech understanding stage, and FIG. 5 is a flowchart showing operations in the dialogue management stage.

The process of generating a summary of meeting minutes goes through three steps: understanding speech (S100), managing dialogue (S200), and generating summary (S300), and performing natural language analysis, dialogue act and intention analysis, summary object extraction according to meeting type, etc. automatically generate a summary of meeting minutes.

The speech understanding step (S100) is a step of converting the user's intention into a semantic structure that can be understood by a machine by natural language processing of the user's speech in the conference text obtained by TTS conversion of the voice during the conference. In the speech understanding step (S100), natural language processing such as morpheme analysis, object name identification, speech act analysis, and intention analysis is performed on the conference content converted into text.

As shown in FIG. 4, the speech understanding step (S100) includes a morpheme analysis step (S110), an object name extraction step (S120), and a dialogue act and intention analysis step (S130).

In the morpheme analysis step (S110), the input sentence is separated into syllable units. A morpheme is the smallest unit that presupposes a meaning, whether lexical or grammatical, in a sentence composed of numerous grammatical units. In the morpheme analysis step (S110), the morpheme of the user-spoken sentence is identified by analyzing the user-spoken sentence until it no longer has any meaning. In the process of morpheme analysis, a natural language processing (NLP) knowledge base for natural language processing, such as a dictionary constructed as a dictionary for cases where there is no ambiguity in part-of-speech tagging and a compound noun dictionary that collects compound nouns, can be used. As the morpheme analysis method, for example, methods described in Registered Patent No. 10-1997783 and Registered Patent No. 10-1358614 may be used, and the present invention is not limited to a specific morpheme analysis method.

In the entity name extraction step (S120), the entity name of the user's spoken sentence is identified and extracted by searching the entity name corresponding to each morpheme analyzed in the entity name dictionary. Entity name extraction patterns can also be used for entity name extraction. It is possible to analyze the meaning of a sentence spoken by a user by applying a linguistic pattern (LSP: Lexico Syntactic Pattern) expressing sentence types having the same meaning to each pre-analyzed morpheme. These grammatical patterns may include at least one or more of information about entity names, semantic features, vocabularies, and parts of speech tags according to the concept of a sentence spoken by a user. Methods described in Korean Patent Registration No. 0829401 and Korean Patent Registration No. 10-2175539 may be used as a method for identifying and extracting entity names, and the present invention is not limited to a specific entity name extraction method.

In the dialogue act and intention analysis step (S130), speech act information such as questions, statements, and requests is analyzed through the dialogue act analysis learned based on the intention and dialogue act knowledge base using morphemes and entity identification information processed in real time. This becomes an important clue information in the meeting summary that determines the components of the meeting. Intent analysis clearly identifies the utterance intention in which the object and action are combined based on the entity name representing the target information and the dialogue act analysis result representing the action. As a dialogue act and intention analysis method, methods described in Patent Registration 10-2285232 and Patent Publication 10-2019-0061706 may be used, and the present invention is not limited to a specific dialogue act and intention analysis method.

The dialogue management step (S200) is a step of extracting an optimal keyword according to the selected conference type based on the previously analyzed semantic structure of the user's utterance and managing the conversation. Meeting types can be classified according to the purpose of the meeting, for example information transfer type, problem solving type, understanding coordination type, etc. Alternatively, it may be classified according to a more specific meeting format or purpose, such as decision-making meetings, progress monitoring meetings, regular meetings, consultations, religious gatherings, lectures, and the like. In the dialog management step (S200), the problems of existing summary algorithms that depend on simple keyword appearance frequency can be solved by maintaining and utilizing information obtained from the current user's utterance and conversations with the previous user.

5 further subdivides the dialog management step (S200). The dialog management step (S200) may include a summary entity extraction step (S210) and a context-based dialog structure step (S220).

Various semantic components constituting the meeting are composed of a combination of the extracted entity name, dialogue act, and intention, and in the summary entity extraction step (S210), these semantic components are recombined. Components can be defined separately according to the conference type.

For example, in the case of a consultation type meeting, the intention can be classified as follows based on the contents to be consulted.

Intent Classification (Consultation Classification) Health problem > person > medical history Health problems > family > medical history Relationships>Family>Parents>Discord Human relationship>Family>Couple>Discord Relationships > Others > Friends > Discord Work problems>Interpersonal relationships>Discord ...

Table 2 is an example of defining object names for a consultation type conference.

keyword entity name synonym caregiver person > identity dad Person>Status>Family father, father mom Person>Status>Family mother, mother sister Person>Status>Family company group Team Leader Person>Status>Company ...

Table 3 is an example of defining dialogue acts for a consultation type meeting.

keyword Synonym Emotion type worry worry negative emotions scary afraid negative emotions Shame negative emotions satisfied positive emotions Satisfaction is high Satisfaction goes up positive emotions hope hope positive emotion_hope ...

10 shows an example of combining an extracted object, dialogue act, and intention in the case of a consultation type conference. In FIG. 10, (a) shows the result of extracting intention, object name, and dialogue act from utterances during a meeting, and (b) shows an example of expressing the counselee's utterance in a triple form (subject, object, predicate).

In the context-based dialogue structuring step (S220), progress or processing is performed according to the flow of multilateral speech through a plurality of dialogue scenarios prepared according to the set conference type, and through this process, a semantic block in which a plurality of conversations are combined will create

9 shows an example of a dialog scenario regarding scheduling the next meeting. For example, if there is an utterance 90 saying, "Shall we hold the next meeting at 2:00 PM on Tuesday?" Accordingly, the flow of multilateral utterances is processed through the conversation scenario shown in FIG. 9 for these utterances. That is, if there is a positive utterance after the utterance to propose the next meeting schedule (90), the proposed meeting schedule is accepted (92), and if there is a negative utterance, the proposed meeting schedule is rejected (93).

In the summary generation step (S300), based on the information found according to the user's intention in the previous step, a summary is generated according to a quantified frame such as summary object arrangement and question and answer according to dialogue act according to the summary template for each conference type. The summary may include information such as topics or commitments to be addressed in the future. In the summary generating step (S300), an automatic summary model may be applied that adjusts the elements and units constituting the minutes according to the type of the meeting, aggregates the topics and contents analyzed in each context, and outputs them according to the template format. Depending on the embodiment, only the keywords representing the intent, object name, and dialog act derived in the previous step may be used for content output according to the template format.

7 shows an example of a weekly meeting type summary template. The weekly meeting summary template of FIG. 7 is an element constituting meeting minutes and includes agenda, main remarks, decisions, pending/new matters, and next meeting information. As another example, the summary template may include elements such as agenda, key remarks, decisions, inquiries, requests, opinions, and statements. As another example, the monthly report summary template may include elements such as monthly performance report, problem report, solution, and instructions. In this way, a more accurate and concise summary can be created by summarizing the meeting minutes using an appropriate summary template according to the type of meeting.

The automatic summary model aggregates topics and contents in units of context of utterances and outputs them according to the template format.

For example, in the case of a summary template containing elements such as agenda, key remarks, decisions, inquiries, requests, opinions, statements, etc., the utterance "What do you mean when you say engine coating?" Since it is an utterance that says "inquiry", it can be output as "engine coating" in the inquiry column.

As another example in the case of the same summary template, the utterance "I will receive log data and check it once and check if there is a problem next time" is judged to be related to a decision from "I will", and "Check log data, once Keywords of the sentence, such as ", problem, check", etc., can be output.

As another example, the utterance "I would like some data" may be judged to be related to the request from "please", and "data" may be output in the request field.

As another example, the barahwa "I think it will be difficult to complete within this year" is judged to be an utterance expressing an opinion from the word "I think", and output "this year, it will be completed, it will be difficult" in the comment column. can

As another example, the utterance "The business you just mentioned started on April 18, 2021, and the KCC is still reviewing this matter" is judged to state the current facts from "I'm in the middle" and entered the statement It can be output like "Business, starting from April 18, 2021, Korea Communications Commission, under review".

Meanwhile, for words that are not analyzed or misanalyzed in the speech understanding step (S100), a process of improving the dictionary of the NLP knowledge base may be performed according to the natural language processing learning method. In addition, based on the NLP knowledge base, machine learning and rule-based intention/dialogue act learning can be performed. Error correction learning can be performed through intent/dialogue act learning data and automatic summary failure data that has not been extracted or mis-extracted after the summary generation step, and the intention/dialogue act knowledge base can be built and strengthened.

In one embodiment, the user's correction process for the automatic summary data is determined as verification, misanalysis or failure data is collected through the typing process, and it is possible to identify and correct the quality problem at which stage of the natural language analysis. . By going through this process, it is possible to secure the learning data of the intention and dialog act analyzer that uses machine learning based on deep learning.

In the above, the present invention has been described with several examples, but even if all components constituting an embodiment of the present invention are combined or described as operating in combination, the present invention is not necessarily limited to these embodiments. Within the scope of the object of the present invention, all of the components may be selectively combined with one or more to operate. In addition, although all of the components may be implemented as a single independent piece of hardware, some or all of the components are selectively combined to perform some or all of the combined functions in one or a plurality of hardware. It may be implemented as a computer program having. Codes and code segments constituting the computer program may be easily inferred by a person skilled in the art. Such a computer program may implement an embodiment of the present invention by being stored in a computer or processor-readable storage medium, read and executed by a computer or processor.

Terms such as "comprise", "comprise" or "having" described above mean that the corresponding component may be present unless otherwise stated, and therefore do not exclude other components. It should be construed that it may further include other components.

The above description is merely an example of the technical idea of the present invention, and various modifications and variations can be made to those skilled in the art without departing from the essential characteristics of the present invention. Therefore, the embodiments disclosed in the present invention are not intended to limit the technical idea of the present invention, but to explain, and the scope of the technical idea of the present invention is not limited by these embodiments. The protection scope of the present invention should be construed according to the claims below, and all technical ideas within the equivalent range should be construed as being included in the scope of the present invention.

10 Meeting minutes writing device,
20 contactless conference servers;
30a~30n Conference participant terminal.

Claims (14)

For a non-face-to-face conference of a predetermined conference type, a speech understanding step of converting user intention into a semantic structure that can be understood by a machine by natural language processing of user speech in a conference text obtained by TTS conversion of voice during a conference;
a conversation management step of extracting an optimal keyword according to the conference type based on the semantic structure of the user's speech converted in the speech understanding step and managing the conversation; and
a summary generating step of generating a summary sentence according to a summary template according to the conference type based on the information generated in the conversation management step;
A method for summarizing the meeting minutes of a non-face-to-face meeting having a.
The method of claim 1, wherein the speech understanding step
A morpheme analysis step of identifying morphemes of a sentence spoken by a user;
An entity name extraction step of searching entity names corresponding to each morpheme to identify and extract entity names of user-spoken sentences;
Dialogue act and intention analysis step of identifying the intention of the user's spoken sentence by analyzing the behavior information through dialogue act analysis using morpheme and object name identification information
Including, how to summarize the meeting minutes of the non-face-to-face meeting.
The method of claim 2, wherein the dialogue management step
A summary entity extraction step of recombining semantic components defined according to the conference type;
A context-based dialogue structuring step that proceeds or processes according to the flow of multilateral speech through a plurality of dialogue scenarios prepared according to the type of conference.
Including, how to summarize the meeting minutes of the non-face-to-face meeting.
According to claim 3,
The semantic component is composed of a combination of the extracted entity name, dialogue act, and intention,
The object name, dialogue act, and intention are defined according to the conference type,
How to summarize the minutes of a face-to-face meeting.
According to claim 3,
The summary generation step applies an automatic summary model that adjusts the elements and units constituting the meeting minutes according to the meeting type, aggregates the topics and contents analyzed in each context, and outputs them in accordance with the template format corresponding to the meeting type. A method of summarizing the minutes of a non-face-to-face meeting, which is performed by
According to claim 5,
The automatic summary model summarizes meeting minutes of a non-face-to-face meeting by aggregating topics and contents in context units of utterances and outputting them according to the template format.
According to any one of claims 2 to 6,
In the dialogue act and intention analysis step, the intention/dialogue act knowledge base is used,
Constructing and strengthening the intention / dialogue act knowledge base by conducting error correction learning through intention / dialogue act learning data and automatic summary failure data that is not extracted or misextracted after the summary generation step,
How to summarize the minutes of a face-to-face meeting.
In the method of summarizing minutes of a non-face-to-face meeting in a non-face-to-face conference server,
receiving conference information including a conference type, a start time, and a duration of the conference from a conference organizer and generating an invitation code to participate in the conference;
converting the conference voice into text using a Text to Speech (TTS) function when the conference starts;
an utterance understanding step of converting the user's intention into a semantic structure that can be understood by a machine by natural language processing of the user's utterance in the text;
a conversation management step of extracting an optimal keyword according to the conference type based on the semantic structure of the user's speech converted in the speech understanding step and managing the conversation; and
a summary generating step of generating a summary sentence according to a summary template according to the conference type based on the information generated in the conversation management step;
A method for summarizing the minutes of a non-face-to-face meeting having a.
The method of claim 8, wherein the speech understanding step
A morpheme analysis step of identifying morphemes of a sentence spoken by a user;
An entity name extraction step of searching entity names corresponding to each morpheme to identify and extract entity names of user-spoken sentences;
Dialogue act and intention analysis step of identifying the intention of the user's spoken sentence by analyzing the behavior information through dialogue act analysis using morpheme and object name identification information
Including, how to summarize the meeting minutes of the non-face-to-face meeting.
10. The method of claim 9, wherein the dialogue management step
A summary entity extraction step of recombining semantic components defined according to the conference type;
A context-based dialogue structuring step that proceeds or processes according to the flow of multilateral speech through a plurality of dialogue scenarios prepared according to the type of conference.
Including, how to summarize the meeting minutes of the non-face-to-face meeting.
According to claim 10,
The semantic component is composed of a combination of the extracted entity name, dialogue act, and intention,
The object name, dialogue act, and intention are defined according to the conference type,
How to summarize the minutes of a face-to-face meeting.
According to claim 10,
The summary generation step applies an automatic summary model that adjusts the elements and units constituting the meeting minutes according to the meeting type, aggregates the topics and contents analyzed in each context, and outputs them in accordance with the template format corresponding to the meeting type. A method of summarizing the minutes of a non-face-to-face meeting, which is performed by
According to any one of claims 10 to 12,
In the dialogue act and intention analysis step, the intention/dialogue act knowledge base is used,
Constructing and strengthening the intention / dialogue act knowledge base by conducting error correction learning through intention / dialogue act learning data and automatic summary failure data that is not extracted or misextracted after the summary generation step,
How to summarize the minutes of a face-to-face meeting.
According to any one of claims 9 to 12,
The speech understanding step, the dialogue management step, and the summary generation step are sequentially performed during the conference.
How to summarize the minutes of a face-to-face meeting.

Images