KR20230047849A - Method and system for summarizing document using hyperscale language model - Google Patents

Abstract

The present disclosure relates to a method for summarizing a document, which is performed by at least one processor. The method for summarizing a document comprises the steps of: receiving a document including a first sentence set; inputting the first sentence set into a category analyzer to extract a second sentence set belonging to a first category; clustering the second sentence set belonging to the first category into a first cluster set; extracting a third sentence set from the first cluster set; and inputting the third sentence set into a language model to generate a first summary sentence associated with the first category of the document, wherein the second sentence set is a subset of the first sentence set and the third sentence set is a subset of the second sentence set. Therefore, a document including a plurality of sentences can be summarized into a one-line sentence to be provided to a user.

Description

Method and system for summarizing documents using super large language model

The present disclosure relates to a method and system for summarizing documents using a very large language model, and more specifically, to a method and system for summarizing documents by category for documents consisting of multiple sentences.

Recently, various services using mobile or Internet are provided to users. A user's review of a provided service often contains objective information about the service. Therefore, when a user decides to use a service or purchases a product through the service, it is often determined by referring to user reviews. However, in the case of a service/product used by a large number of users, the amount of user reviews is so large that many inappropriate or meaningless user reviews are included, and it is difficult to check all user reviews.

In order to solve this problem, a service is being provided that selects and provides some of the reviews written by a plurality of users. However, in this case, since some of the reviews are selected and provided, there is a problem in that even if there are expressions that do not match spelling or spacing, they are provided without being corrected. In addition, there is a problem in that some of these reviews are only opinions of one person and cannot reflect all opinions of users.

In order to solve the above problem, various embodiments of the present disclosure provide a document summary method, a computer program stored in a recording medium, and an apparatus (system).

The present disclosure may be implemented in a variety of ways, including a method, apparatus (system) or computer program stored on a readable storage medium.

According to an embodiment of the present disclosure, a method for summarizing a document, performed by at least one processor, is provided. The document summarization method includes the steps of receiving a document including a first sentence set, inputting the first sentence set to a category analyzer and extracting a second sentence set belonging to a first category, and a second sentence set belonging to the first category. clustering into a first cluster set, extracting a third sentence set from the first cluster set, and inputting the third sentence set into a language model to generate a first summary sentence associated with the first category of the document. and wherein the second sentence set is a subset of the first sentence set and the third sentence set is a subset of the second sentence set.

According to one embodiment of the present disclosure, a computer program stored in a computer readable recording medium is provided to execute a document summary method on a computer.

According to one embodiment of the present disclosure, a document summarization system is provided. A document summarization system includes a communication module, a memory and at least one processor coupled with the memory and configured to execute at least one computer readable program contained in the memory. The at least one program receives a document including a first set of sentences, inputs the first set of sentences to a category analyzer, extracts a second set of sentences belonging to the first category, and extracts a second set of sentences belonging to the first category. into a first cluster set, extracting a third sentence set from the first cluster set, and inputting the third sentence set into a language model to generate a first summary sentence associated with the first category of the document. do. Here, the second sentence set is a subset of the first sentence set, and the third sentence set is a subset of the second sentence set.

According to various embodiments of the present disclosure, a document including a plurality of sentences may be summarized in one line and provided to the user.

According to various embodiments of the present disclosure, a user may be provided with a natural summary sentence associated with a specific category in a document prepared for one subject.

According to various embodiments of the present disclosure, users can easily check users' evaluations of a corresponding product by checking only summary reviews for each category without reading all user reviews related to a specific product in a shopping mall.

According to various embodiments of the present disclosure, in order to prevent content not included in a document to be summarized from being included in the summary sentence, the authenticity of the summary sentence is checked and the summary sentence is regenerated, so that the user can obtain a highly reliable summary sentence. can be provided.

The effects of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned are clear to those skilled in the art (referred to as "ordinary technicians") from the description of the claims. will be understandable.

1 is a diagram illustrating an example of a user using a document summary service using a user terminal according to an embodiment of the present disclosure.
2 is a schematic diagram illustrating a configuration in which an information processing system is communicatively connected to a plurality of user terminals in order to provide a document summary service according to an embodiment of the present disclosure.
3 is a block diagram showing internal configurations of a user terminal and an information processing system according to an embodiment of the present disclosure.
4 is a functional block diagram showing an internal configuration of a processor of an information processing system according to an embodiment of the present disclosure.
5 is a diagram illustrating an example of generating a category-by-category summary for a document, according to an embodiment of the present disclosure.
6 is a diagram illustrating an example of generating a summary sentence based on a summary sample and a summary target by a super-large language model according to an embodiment of the present disclosure.
7 is a diagram illustrating an example of extracting a representative sentence from a plurality of clusters to generate a summary sentence according to an embodiment of the present disclosure.
8 is a diagram illustrating examples of summary sentences generated according to alignment of representative sentences included in a summary target according to an embodiment of the present disclosure.
9 is a diagram illustrating an example of providing category information as a hint to a very large language model when generating a summary sentence according to an embodiment of the present disclosure.
10 is a diagram illustrating an example of verifying authenticity of a summary sentence and regenerating the summary sentence according to an embodiment of the present disclosure.
11 is a flowchart illustrating a method for generating summary sentences, according to one embodiment of the present disclosure.
12 is a flowchart illustrating a method for generating a summary sentence including a subject to be summarized, according to an embodiment of the present disclosure.

Hereinafter, specific details for the implementation of the present disclosure will be described in detail with reference to the accompanying drawings. However, in the following description, if there is a risk of unnecessarily obscuring the gist of the present disclosure, detailed descriptions of well-known functions or configurations will be omitted.

In the accompanying drawings, identical or corresponding elements are given the same reference numerals. In addition, in the description of the following embodiments, overlapping descriptions of the same or corresponding components may be omitted. However, omission of a description of a component does not intend that such a component is not included in an embodiment.

Advantages and features of the disclosed embodiments, and methods of achieving them, will become apparent with reference to the following embodiments in conjunction with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below and may be implemented in various different forms, but only the present embodiments make the present disclosure complete, and the present disclosure does not extend the scope of the invention to those skilled in the art. It is provided only for complete information.

Terms used in this specification will be briefly described, and the disclosed embodiments will be described in detail. The terms used in this specification have been selected from general terms that are currently widely used as much as possible while considering the functions in the present disclosure, but they may vary according to the intention of a person skilled in the related field, a precedent, or the emergence of new technology. In addition, in a specific case, there is also a term arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the invention. Therefore, the terms used in the present disclosure should be defined based on the meaning of the terms and the general content of the present disclosure, not simply the names of the terms.

Expressions in the singular number in this specification include plural expressions unless the context clearly dictates that they are singular. Also, plural expressions include singular expressions unless the context clearly specifies that they are plural. When it is said that a certain part includes a certain component in the entire specification, this means that it may further include other components without excluding other components unless otherwise stated.

Also, the term 'module' or 'unit' used in the specification means a software or hardware component, and the 'module' or 'unit' performs certain roles. However, 'module' or 'unit' is not meant to be limited to software or hardware. A 'module' or 'unit' may be configured to reside in an addressable storage medium and may be configured to reproduce one or more processors. Thus, as an example, a 'module' or 'unit' includes components such as software components, object-oriented software components, class components, and task components, processes, functions, and attributes. , procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, or variables. Functions provided within components and 'modules' or 'units' may be combined into a smaller number of components and 'modules' or 'units', or further components and 'modules' or 'units'. can be further separated.

According to one embodiment of the present disclosure, a 'module' or 'unit' may be implemented with a processor and a memory. 'Processor' should be interpreted broadly to include general-purpose processors, central processing units (CPUs), microprocessors, digital signal processors (DSPs), controllers, microcontrollers, state machines, and the like. In some circumstances, 'processor' may refer to an application specific integrated circuit (ASIC), programmable logic device (PLD), field programmable gate array (FPGA), or the like. 'Processor' refers to a combination of processing devices, such as, for example, a combination of a DSP and a microprocessor, a combination of a plurality of microprocessors, a combination of one or more microprocessors in conjunction with a DSP core, or a combination of any other such configurations. You may. Also, 'memory' should be interpreted broadly to include any electronic component capable of storing electronic information. 'Memory' includes random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erasable-programmable read-only memory (EPROM), It may also refer to various types of processor-readable media, such as electrically erasable PROM (EEPROM), flash memory, magnetic or optical data storage, registers, and the like. A memory is said to be in electronic communication with the processor if the processor can read information from and/or write information to the memory. Memory integrated with the processor is in electronic communication with the processor.

In the present disclosure, a 'hyperscale language model' may refer to a language model capable of inference without fine-tuning using a method such as a few-shot learning, and It can have more than 10 times as many parameters as regular language models (e.g., more than 100 billion parameters, etc.). For example, the super large language model may be HyperCLOVA or Generative Pretrained Transformer 3 (GPT 3). In the present disclosure, a very large language model may be referred to as a language model.

In the present disclosure, a 'summary sample' may refer to a summary sample sentence input to a very large language model to generate a summary sentence. According to an embodiment, the summary sample may be prepared in advance by a sentence summary service provider and input into the super large language model. For example, the summary sample may include a product name, a plurality of user review sentences, and a summary sentence in which the plurality of user review sentences are summarized into one sentence.

1 is a diagram illustrating an example in which a user 110 uses a document summary service using a user terminal according to an embodiment of the present disclosure. As shown, the user 110 may be provided with results 122 , 124 , and 126 summarizing various reviews left by buyers in one line for each category using the user terminal. Previously, a service was provided to select and provide one sentence for each category among reviews written by a large number of users, but since one of the many reviews is selected and provided, even if there is an expression that does not match spelling or spacing, it is provided without modification. there is a problem. In addition, there is a problem in that the opinions of the entire review cannot be reflected because a sentence selected from among sentences written by a plurality of users is only an opinion of one person. In order to solve such a problem, it is necessary to exclude sentences that have no content or are not helpful among the reviews written by a large number of users, and it is necessary to provide a summary of reviews by category reflecting the opinions of the majority. In one embodiment, a document summary service (eg, review summary service for each category of a specific product) may be provided using a very large language model. In other words, a very large language model that can infer and create sentences on its own can analyze documents and create natural summaries.

As shown, the user 110 may be provided with review summaries 122 , 124 , and 126 for each category predefined in relation to a specific product through the user interface 120 of the user terminal. The category may be set differently for each product, and in the illustrated example, color, design, and performance may be used as categories related to the product. For example, user 110 may be provided with a review summary 122 related to 'color', a review summary 124 related to 'design', and a review summary 126 related to 'performance' of a particular product.

Although FIG. 1 illustrates providing a summary of reviews of users related to a specific product in a shopping mall, the present invention is not limited thereto. For example, sentence summaries can be used to provide category-by-sentence summaries for multi-sentence documents (eg, novels, news articles, papers, etc.). With this configuration, it is possible for users to easily check the user's evaluation of the corresponding product by checking only summary reviews by category without reading all user reviews related to a specific product in the shopping mall.

2 is a schematic diagram illustrating a configuration in which an information processing system 230 is communicatively connected to a plurality of user terminals 210_1, 210_2, and 210_3 in order to provide a document summary service according to an embodiment of the present disclosure. The information processing system 230 may include a system capable of providing a document summary service. In one embodiment, the information processing system 230 is one or more server devices and/or databases capable of storing, providing, and executing computer-executable programs (eg, downloadable applications) and data associated with the document summary service; or It may include one or more distributed computing devices and/or distributed databases based on cloud computing services. For example, the information processing system 230 may include separate systems (eg, servers) for document summary services. The document summary service provided by the information processing system 230 may be provided to users through a document summary application installed in each of the plurality of user terminals 210_1 , 210_2 , and 210_3 , a web browser, and the like. In one embodiment, the information processing system 230 may provide a document summary service to a user terminal using the super large language model 240 .

A plurality of user terminals 210_1 , 210_2 , and 210_3 may communicate with the information processing system 230 through the network 220 . The network 220 may be configured to enable communication between the plurality of user terminals 210_1, 210_2, and 210_3 and the information processing system 230. Depending on the installation environment, the network 220 may be, for example, a wired network such as Ethernet, a wired home network (Power Line Communication), a telephone line communication device and RS-serial communication, a mobile communication network, a wireless LAN (WLAN), It may consist of a wireless network such as Wi-Fi, Bluetooth, and ZigBee, or a combination thereof. The communication method is not limited, and the user terminals (210_1, 210_2, 210_3) as well as a communication method utilizing a communication network (eg, mobile communication network, wired Internet, wireless Internet, broadcasting network, satellite network, etc.) that the network 220 may include ), short-range wireless communication between them may also be included.

In FIG. 2, a mobile phone terminal 210_1, a tablet terminal 210_2, and a PC terminal 210_3 are illustrated as examples of user terminals, but are not limited thereto, and the user terminals 210_1, 210_2, and 210_3 may perform wired and/or wireless communication. It can be any computing device capable of this. For example, the user terminal includes a smartphone, a mobile phone, a navigation device, a computer, a laptop computer, a digital broadcasting terminal, a personal digital assistant (PDA), a portable multimedia player (PMP), a tablet PC, a game console, and a wearable device ( wearable device), internet of things (IoT) device, virtual reality (VR) device, augmented reality (AR) device, and the like. In addition, although three user terminals 210_1, 210_2, and 210_3 are shown in FIG. 2 to communicate with the information processing system 230 through the network 220, it is not limited thereto, and a different number of user terminals may be connected to the network ( 220) to communicate with the information processing system 230.

In one embodiment, the information processing system 230 may receive all or part of a document to be summarized, a document summary request, and the like from the user terminals 210_1, 210_2, and 210_3. The information processing system 230 may generate summary sentences for each category for the document to be summarized, and provide the summary result to the user terminals 210_1, 210_2, and 210_3. Although the very large language model 240 is shown to exist outside the information processing system 230 in FIG. 2, it is not limited thereto, and the very large language model 240 may be stored and used inside the information processing system 230. In addition, although it is shown in FIG. 2 that the information processing system 230 generates summary sentences for each category for a document to be summarized and provides them to the user terminal, the present invention is not limited thereto, and hardware/software for providing a document summary service is provided to the user. It may be provided in the terminal.

Figure 3 is a block diagram showing the internal configuration of the user terminal 210 and the information processing system 230 according to an embodiment of the present disclosure. The user terminal 210 may refer to any computing device capable of executing a document summary application, a web browser, etc. and capable of wired/wireless communication, for example, the mobile phone terminal 210_1 of FIG. 2 or the tablet terminal 210_2 of FIG. ), PC terminal 210_3, and the like. As shown, the user terminal 210 may include a memory 312 , a processor 314 , a communication module 316 and an input/output interface 318 . Similarly, the information processing system 230 may include a memory 332 , a processor 334 , a communication module 336 and an input/output interface 338 . As shown in FIG. 3, the user terminal 210 and the information processing system 230 are configured to communicate information and/or data through the network 220 using respective communication modules 316 and 336. It can be. In addition, the input/output device 320 may be configured to input information and/or data to the user terminal 210 through the input/output interface 318 or output information and/or data generated from the user terminal 210.

The memories 312 and 332 may include any non-transitory computer readable media. According to one embodiment, the memories 312 and 332 are non-perishable mass storage devices such as random access memory (RAM), read only memory (ROM), disk drives, solid state drives (SSDs), flash memory, and the like. (permanent mass storage device) may be included. As another example, a non-perishable mass storage device such as a ROM, SSD, flash memory, or disk drive may be included in the user terminal 210 or the information processing system 230 as a separate permanent storage device separate from memory. Also, an operating system and at least one program code may be stored in the memories 312 and 332 .

These software components may be loaded from a computer readable recording medium separate from the memories 312 and 332 . A recording medium readable by such a separate computer may include a recording medium directly connectable to the user terminal 210 and the information processing system 230, for example, a floppy drive, a disk, a tape, a DVD/CD- It may include a computer-readable recording medium such as a ROM drive and a memory card. As another example, software components may be loaded into the memories 312 and 332 through the communication modules 316 and 336 rather than computer-readable recording media. For example, at least one program is loaded into the memories 312 and 332 based on a computer program installed by files provided by developers or a file distribution system that distributes application installation files through the network 220 . It can be.

The processors 314 and 334 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input/output operations. Instructions may be provided to processors 314 and 334 by memories 312 and 332 or communication modules 316 and 336 . For example, processors 314 and 334 may be configured to execute instructions received according to program codes stored in a recording device such as memory 312 and 332 .

The communication modules 316 and 336 may provide configurations or functions for the user terminal 210 and the information processing system 230 to communicate with each other through the network 220, and the user terminal 210 and/or information processing. System 230 may provide configurations or functions for communicating with other user terminals or other systems (eg, separate cloud systems, etc.). For example, a request or data generated by the processor 314 of the user terminal 210 according to a program code stored in a recording device such as the memory 312 (eg, a document summary request, etc.) of the communication module 316 According to the control, it may be transferred to the information processing system 230 through the network 220 . Conversely, a control signal or command provided under the control of the processor 334 of the information processing system 230 passes through the communication module 336 and the network 220 and through the communication module 316 of the user terminal 210. It may be received by the user terminal 210 . For example, the user terminal 210 may receive summary sentences for each category of a document from the information processing system 230 .

The input/output interface 318 may be a means for interfacing with the input/output device 320 . As an example, the input device may include a device such as a camera, keyboard, microphone, mouse, etc. including an audio sensor and/or image sensor, and the output device may include a device such as a display, speaker, haptic feedback device, or the like. can As another example, the input/output interface 318 may be a means for interface with a device in which a configuration or function for performing input and output is integrated into one, such as a touch screen. Although the input/output device 320 is not included in the user terminal 210 in FIG. 3 , it is not limited thereto, and the user terminal 210 and the user terminal 210 may be configured as one device. In addition, the input/output interface 338 of the information processing system 230 is connected to the information processing system 230 or means for interface with a device (not shown) for input or output that the information processing system 230 may include. can be In FIG. 3, the input/output interfaces 318 and 338 are shown as separate elements from the processors 314 and 334, but are not limited thereto, and the input/output interfaces 318 and 338 may be included in the processors 314 and 334. there is.

The user terminal 210 and the information processing system 230 may include more components than those shown in FIG. 3 . However, there is no need to clearly show most of the prior art components. In one embodiment, the user terminal 210 may be implemented to include at least some of the aforementioned input/output devices 320 . In addition, the user terminal 210 may further include other components such as a transceiver, a global positioning system (GPS) module, a camera, various sensors, and a database. For example, when the user terminal 210 is a smartphone, it may include components that are generally included in a smartphone, for example, an acceleration sensor, a gyro sensor, a microphone module, a camera module, and various physical Various components such as a button, a button using a touch panel, an input/output port, and a vibrator for vibration may be implemented to be further included in the user terminal 210 .

4 is a functional block diagram showing an internal configuration of a processor 400 of an information processing system according to an embodiment of the present disclosure. As shown, the processor 400 includes a category analysis unit 410, a filtering unit 420, a clustering unit 430, a very large language model 440, a fact check unit 450, and a sentence correction unit 460. can include Although one processor is shown in FIG. 4, it is not limited thereto, and the information processing system may include a plurality of processors to perform the same function as the processor 400.

According to an embodiment, the category analyzer 410 may be configured to receive a document including a plurality of sentences, separate the documents into sentence units, and extract reviews associated with a specific category. For example, the category analyzer 410 may extract sentences belonging to the first category from a document including a plurality of sentences. Similarly, the category analyzer 410 may extract sentences belonging to the second category from a document including a plurality of sentences. The category analyzer 410 may be a deep neural network (DNN) and rule-based hybrid model. Additionally, the category analyzer 410 may add a positive or negative flag to the extracted sentence. For example, the category analyzer 410 may add a positive flag to a sentence such as "It was delicious" belonging to the 'taste' category.

According to an embodiment, the filtering unit 420 may filter short or meaningless sentences by determining scores of sentences extracted by the category analysis unit 410 . Specifically, the filtering unit 420 may determine a score of each sentence based on at least one of the length of the sentence, the length of a part associated with a category in the sentence, whether a duplicate morpheme exists, and whether a part of speech or symbol is included. The score can be used to determine how much information is included in the sentence, and the filtering unit 420 can filter and sort the sentences using the score. For example, the filtering unit 420 may assign a low score to the sentence in the case of a short sentence, a sentence with few parts related to a category, a sentence in which meaningless expressions are repeated, and the like. Thereafter, the filtering unit 420 may provide the clustering unit 430 with a sentence having a score equal to or higher than a predetermined threshold (eg, 0.55 or higher).

According to an embodiment, the clustering unit 430 may cluster sentences belonging to one category into a plurality of clusters. Here, clustering means grouping related sentences (for example, sentences having similar expressions and meanings) together. For example, the clustering unit 430 may convert each sentence into a vector and perform clustering using a complete linkage clustering technique. In the case of a shopping review summary, the more sentences are grouped by clustering, the more it can be determined that the corresponding content is the main content of the current product.

Thereafter, the clustering unit 430 may sort the clusters based on size and extract a total of n sentences by extracting one sentence for each cluster. n is a natural number greater than or equal to 2, and may be n=4 considering speed and performance. Alternatively, the clustering unit 430 may extract one sentence from each cluster. The clustering unit 430 may repeat the above-described operation for all categories.

According to an embodiment, the super large language model 440 receives representative sentences for each category (eg, four sentences for each category) extracted by the clustering unit 430, and summarizes sentences for each category of the document. can create For example, the super large language model 440 may generate a summary sentence associated with the first category by receiving four representative sentences associated with the first category and summarizing them into one sentence. Similarly, the super large language model 440 may generate a summary sentence associated with the second category by receiving four representative sentences associated with the second category and summarizing them into one sentence.

According to an embodiment, the fact checker 450 may be configured to verify the authenticity of the summary sentence generated by the super large language model 440 . In the case of the extra-large language model 440, direct inference is possible and natural sentences can be generated because it is learned based on a large amount of data, but in some cases, content not included in the input sentence is newly created. That is, it is known that the super large language model 440 can generate false sentences that are not included in the input sentences, and according to related research, the super large language model 440 can generate false sentences at a rate of about 30%. . Therefore, a process of matching the sentences (summary sentences) generated by the extra large language model 440 with the contents of the input sentences (representative sentences associated with categories) is required.

In one embodiment, the fact checker 450 uses a natural language inference (NLI) model to input a sentence (a representative sentence associated with a category) and a sentence generated by the super large language model 440 (summary sentence). relationship between them can be determined. Here, the NLI model may be a model for determining whether one sentence is logically related to another sentence, and the relationship between an input sentence and a generated sentence is classified into a neutral relationship, an entailment relationship, and a contradictory relationship. (Contradiction) can be classified and determined. Specifically, when it is determined that the input sentence and the generated sentence do not have a connotational relationship, the fact checker 450 may determine the generated summary sentence to be false. For example, the fact checker 450 determines the relationship between any two sentences among the four representative sentences and the generated summary sentence. It can be determined that the generated summary sentence is true. Alternatively, the fact checker 450 may verify the authenticity of the generated summary sentence by determining a ROUGE score representing an overlap between words in the two sentences.

When it is determined that the summary sentence generated by the super large language model 440 is true, the fact checker 450 may provide the generated summary sentence to the sentence correction unit 460 . On the other hand, if the summary sentence generated by the extra large language model 440 is determined to be false, the extra large language model 440 may regenerate the summary sentence using the greedy option. Here, the greedy option may indicate an option for the extra large language model 440 to make the output sentence more similar to the input sentence. Thereafter, the fact checker 450 verifies whether the regenerated summary sentence is authentic, and if determined to be true, may provide the regenerated summary sentence to the sentence correction unit 460 .

According to an embodiment, the sentence corrector 460 may receive the summary sentence determined to be true by the fact checker 450 and determine whether it is a normal sentence. For example, if the summary sentence is out of spelling or includes profanity, it may be determined as an abnormal sentence. When the summary sentence is determined to be an abnormal sentence, the sentence correction unit 460 may input the summary sentence to the super large language model 440 and correct it into a normal sentence.

The internal configuration of the processor 400 shown in FIG. 4 is just an example, and in some embodiments, other configurations other than the illustrated internal configuration may be additionally included, or some configurations may be omitted, and some processes may be performed in other configurations or external systems. can be performed by

5 is a diagram illustrating an example of generating a category-by-category summary for a document, according to an embodiment of the present disclosure. As shown, the category analyzer 410 may receive a document 510 including a plurality of sentences. For example, document 510 may represent a document consisting of multiple sentences (eg, user reviews, novels, news articles, papers, etc.). Thereafter, the category analyzer 410 may separate the received document 510 into sentence units and extract sentences 520_1 and 520_2 associated with predefined categories, respectively. For example, the category analyzer 410 may receive the document 510 and extract a first sentence set 520_1 associated with the first category and a second sentence set 520_2 associated with the second category. Sentences 520_1 and 520_2 associated with each extracted category may be separately input to the clustering unit 430 .

The clustering unit 430 may cluster the sentences 520_1 and 520_2 associated with the category into a plurality of clusters, and extract a total of n sentences by extracting one sentence for each cluster. As for the cluster, the top n clusters (n is a natural number of 2 or more) may be selected based on the size. n may be n=4 considering speed and performance. For example, as shown, the clustering unit 430 clusters the first sentence set 520_1 associated with the first category into a plurality of clusters, and extracts one sentence from each of the four clusters with a higher size to generate the first sentence set 520_1. Sentences 530_1, 530_2, 530_3, and 530_4 may be extracted. Similarly, the clustering unit 430 clusters the second sentence set 520_2 associated with the second category into a plurality of clusters, and the fifth to eighth sentences 530_5, 530_6, 530_7, and 530_8 are selected from four clusters having a higher size. ) can be extracted.

Representative sentences (eg, 530_1 to 530_8) for each category are input to the super large language model 440 together with summary samples (eg, 540 and 550), and summary sentences may be generated. For example, as shown, the very large language model 440 receives the first to fourth sentences 530_1 to 530_4 associated with the first category and the first summary sample 540, and the first summary sentence ( 560) can be created. Similarly, the very large language model 440 may generate a second summary sentence 570 by receiving the fifth to eighth sentences 530_5 to 530_8 associated with the second category and the second summary sample 550 . there is. The first summary sentence 560 and the second summary sentence 570 may be provided as a first category summary sentence and a second category summary sentence for the document 510 , respectively. 5 shows that one summary sentence is output for each of two categories, but is not limited thereto, and one summary sentence for each of three or more categories may be output.

6 is a diagram illustrating an example of generating a summary sentence 630 based on a summary sample 610 and a summary object 620 by a super-large language model according to an embodiment of the present disclosure. The summary sentence 630 may be generated by inputting the summary object 620 and the summary sample 610 together into a very large language model. The summary sample 610 may be prepared in advance by a sentence summary service provider and input into a very large language model. The summary sample 610 may include a plurality of samples (eg, 612 and 614), and one summary sample may include a product name, a plurality of user review sentences, and a summary sentence. For example, as shown, the first summary sample 612 and the second summary sample 614 may consist of a product name, four user review sentences, and a sentence summarizing the four user review sentences into one sentence. . In one embodiment, the summary sample may be pre-built for each category by the service provider.

The summary object 620 may include a product name 622 and representative sentences 624 extracted from each of the four upper clusters. Also, the summary target 620 may further include an unwritten summary item 626 . A summary sentence 630 may be generated by inputting the summary sample 610 and the summary subject 620 together to the super-large language model. For example, as shown in FIG. 6, the super-large language model is the summary sentence 630 of the representative sentence 624 included in the summary target 620, 'This is a taste and aroma that anyone who likes coffee will like. .' can be generated and output.

7 illustrates an example of extracting representative sentences 712, 722, 732, and 742 from a plurality of clusters 710, 720, 730, and 740 to generate a summary sentence 750, according to an embodiment of the present disclosure. It is a drawing that represents The processor may cluster sentences classified into a specific category among sentences included in the document into a plurality of clusters. For example, the processor extracts sentences related to the 'satisfaction level' category among user reviews for a specific product, and converts the sentences into 'satisfaction related to size' (710), 'satisfaction with solid quality' (720), and 'transparent material'. It can be clustered into clusters such as 'satisfaction with '(730) and 'satisfaction related to price' (740).

After that, the processor may extract one sentence from the top 4 clusters based on the cluster size. At this time, the processor may randomly extract one sentence from each cluster or extract the sentence having the highest score. For example, a first sentence 712 may be extracted from the 'satisfaction related to size' cluster 710, and a second sentence 722 may be extracted from the 'satisfaction with solid quality' cluster 720. Similarly, a third sentence 732 may be extracted from the 'satisfaction with transparent materials' cluster 730 and a fourth sentence 742 may be extracted from the 'price-related satisfaction' cluster 740 . The sentences extracted in this way can be regarded as sentences representing user reviews of the corresponding product. Finally, the processor may generate a summary sentence 750 by inputting the first to fourth sentences 712 , 722 , 732 , and 742 into the super large language model.

8 is a diagram illustrating examples of summary sentences 812 and 822 generated according to alignment of representative sentences included in summary objects 810 and 820 according to an embodiment of the present disclosure. As shown, different summary sentences 812 and 822 may be generated according to an alignment criterion of representative sentences included in the summary objects 810 and 820 .

The first summary object 810 is an example in which representative sentences extracted from a plurality of clusters are arranged in descending order based on sentence length. A first summary subject 810 may be input to the super-large language model, and a first summary sentence 812 may be generated. The second summary object 820 is an example in which representative sentences extracted from a plurality of clusters are arranged in ascending order based on sentence length. A second summary sentence 822 may be generated by inputting the second summary subject 820 to the super-large language model. As shown, when the second summary object 820 in which representative sentences are arranged in ascending order is compared with the first summary object 810 in which representative sentences are arranged in descending order, a summary sentence containing a lot of information is generated. can confirm that This is because the super-large language model assigns a high weight to the last sentence among a plurality of input sentences.

9 is a diagram illustrating an example of providing category information as a hint to a very large language model when generating a summary sentence according to an embodiment of the present disclosure. As shown, the summary subject 910 may include information 912 about a product name, representative sentence, and category. In this case, a very large language model can generate summaries that are more relevant to the category being summarized. For example, as shown in the figure, when 'material' is entered into a super-large language model along with a product name and a representative sentence as category information for an apparel product, the product of apparel products such as "It is a soft and elastic material is good." A summary sentence associated with 'material' may be generated.

According to an embodiment, the category information 912 may include not only information about categories to be included in the summary text, but also information about categories not to be included in the summary text. For example, if 'material, minus cost/performance ratio' is entered into a super-large language model together with a product name and representative sentence as category information 912, a summary related to 'material' but not related to 'cost/performance ratio' sentences can be created. In another embodiment, when a summary sentence is generated by inputting 'stars' as the category information 912 to a super-large language model, asterisks such as '★★★★☆' are used as summary sentences for representative sentences. can be created

10 is a diagram illustrating an example of verifying authenticity of a summary sentence and regenerating the summary sentence according to an embodiment of the present disclosure. As shown, when the first summary sentence 1020 generated based on the summary object 1010 is determined to be false including content not described in the summary object 1010, the processor summarizes the summary using the greedy option. Based on the object 1030, the second summary sentence 1034 may be regenerated. At this time, the processor may determine whether the summary sentence 1020 is true or not by using the NLI model.

According to one embodiment, the processor determines a relationship between any two sentences among the four representative sentences 1012, 1014, 1016, and 1018 and the first summary sentence 1020, and if it is determined that the relationship is connotative even once, , it can be determined that the first summary sentence 1020 is true. Specifically, the processor determines (i) the relationship between the first sentence 1012 and the second sentence 1014 and the first summary sentence 1020, (ii) the second sentence 1014 and the third sentence 1016 , and the relationship between the first summary sentence 1020, (iii) the relationship between the third sentence 1016 and the fourth sentence 1018 and the first summary sentence 1020, (iv) the fourth sentence 1018 ) and the relationship between the first sentence 1012 and the first summary sentence 1020, (v) the relationship between the first sentence 1012 and the third sentence 1016 and the first summary sentence 1020, (vi) a relationship between the second sentence 1014 and the fourth sentence 1018 and the first summary sentence 1020 may be determined. If any one of the above six cases is determined to be a connotative relationship, the processor may determine that the first summary sentence 1020 is true. Conversely, when it is determined that all of the six cases are not connotative, the processor may determine that the first summary sentence 1020 is false.

As shown, the second summary sentence 1034 regenerated using the greedy option is more similar to at least one of the four representative sentences 1012, 1014, 1016, and 1018 compared to the first summary sentence 1020. can be created Specifically, the second summary sentence 1034 can be generated similarly to the third sentence 1016, and the processor can determine that the second summary sentence 1034 is true. In this case, the processor may output the second summary sentence 1034 as a summary of the summary object 1030 . On the other hand, if the second summary sentence 1034 is determined to be false, the processor may not output the second summary sentence 1034 .

Table 1 below shows the reliability when the relationship between one representative sentence included in the summary target and the summary sentence is determined using the NLI model.

condition Truth Coverage 1) Extract single cluster sentences 0.624 2) Extract multiple cluster sentences 0.609 3) 2)+Add the top of the property 0.605 4) Sort 2)+Sentence Length in Descending Order 0.597 5) Sort 3)+sentence length in descending order 0.691 6) Sort 3)+sentence length in ascending order 0.698 7) 1)+Greedy 0.858 8) 3)+Greedy 0.863 9) 5)+Greedy 0.906 10) 6)+Greedy 0.888

Table 2 below shows the reliability when the relationship between any two of the representative sentences included in the summary target and the summary sentence is determined using the NLI model.

condition Truth Coverage 1) Extract single cluster sentences 0.624 2) Extract multiple cluster sentences 0.609 3) 2)+Add the top of the property 0.717 4) Sort 2)+Sentence Length in Descending Order 0.597 5) Sort 3)+sentence length in descending order 0.792 6) Sort 3)+sentence length in ascending order 0.789 7) 1)+Greedy 0.858 8) 3)+Greedy 0.927 9) 5)+Greedy 0.959 10) 6)+Greedy 0.958

As can be seen from the table above, higher reliability can be obtained when the representative sentence included in the summary subject is grouped into two and compared with the summary sentence to determine authenticity.

11 is a flow diagram illustrating a method 1100 for generating a summary sentence, according to one embodiment of the present disclosure. Method 1100 may be performed by at least one processor of an information processing system (or at least one processor of a user terminal). According to an embodiment, the method 1100 may be initiated when a processor receives a document including a first sentence set (S1110). In one embodiment, a document may include multiple user reviews associated with a particular product. In another embodiment, a document may include a multi-sentence document (eg, a novel, news article, article, etc.).

According to an embodiment, the processor may extract a second sentence set belonging to the first category by inputting the first sentence set to the category analyzer (S1120). Here, the second sentence set may be a subset of the first sentence set.

Then, the processor may cluster the second sentence set belonging to the first category into the first cluster set (S1130). In one embodiment, the processor determines a score for each of the second set of sentences based on at least one of a length of the sentence, a length of a portion associated with the first category in the sentence, whether a redundant morpheme is present, or whether a symbol is included, and the second sentence Among the sets, sentences having scores higher than a predetermined threshold may be clustered into the first cluster set. With this configuration, it is possible to filter short sentences, sentences with few parts related to categories, and sentences in which meaningless expressions are repeated.

The processor may extract a third sentence set from the first cluster set (S1140). Here, the third sentence set may be a subset of the second sentence set. For example, the processor may extract one sentence from each of the first cluster sets. Alternatively, the processor may select top n clusters (n is 2 or more) from the first cluster set based on the cluster size, and extract one sentence from each of the selected top n clusters. Considering speed and performance, n may be 4.

The processor may generate a first summary sentence associated with the first category of the document by inputting the third sentence set to the language model (S1150). Here, the third sentence set may be input to the language model after being sorted in ascending order based on sentence length. For example, the processor may generate a summary sentence associated with the first category by inputting the third sentence set and the summary sample associated with the first category together to the language model. In another example, the processor may generate a summary sentence related to the first category by further inputting information about the first category to the language model.

The flowchart shown in FIG. 11 and the above description are only examples, and in some embodiments, one or more steps may be performed simultaneously or one or more steps may be repeatedly performed multiple times. For example, the processor may input the first sentence set to the category analyzer and extract sentences belonging to each of a plurality of categories. Specifically, the processor may input the first sentence set to the category analyzer and extract a fourth sentence set belonging to the second category. Here, the fourth sentence set may be a subset of the first sentence set. Then, the processor may cluster the fourth sentence set belonging to the second category into the second cluster set, and extract the fifth sentence set from the second cluster set. Here, the fifth sentence set may be a subset of the fourth sentence set. And, the processor may generate a second summary sentence associated with the second category of the document by inputting the fifth set of sentences to the language model.

12 is a flow diagram illustrating a method 1200 for regenerating a summary sentence including a summary subject, according to an embodiment of the present disclosure. Method 1200 may be performed by at least one processor of an information processing system (or at least one processor of a user terminal). According to one embodiment, the method 1200 may begin with a processor generating a summary sentence (S1210). For example, the summary sentence may be a first summary sentence generated according to method 1100 .

The processor may verify the authenticity of the first summary sentence by inputting the first summary sentence and the third sentence set to the fact check unit (S1220). In this case, the fact checker may determine a relationship between the third sentence set and the first summary sentence by using the NLI model. Specifically, the fact checker may determine a relationship between two sentences of the third sentence set and the first summary sentence. Here, in response to determining that the third set of sentences and the first summary sentence do not have a connotational relationship, the processor may determine the first summary sentence to be false.

If the summary sentence associated with the first category is determined to be false, the processor may regenerate the first summary sentence by inputting the third sentence set to the language model (S1260). In this case, the regenerated first summary sentence may be generated more similar to at least one of the third sentence set compared to the first summary sentence. For example, a processor can use the language model's greedy option to regenerate summary sentences.

After that, the processor may input the regenerated first summary sentence and the third sentence set to the fact check unit to verify the authenticity of the regenerated first summary sentence (S1270). If the regenerated first summary sentence is determined to be false, the processor may not output the regenerated first summary sentence (S1280).

If the first summary sentence first generated in S1210 or the first summary sentence regenerated in S1260 is determined to be true, the processor may determine whether the first summary sentence (or the regenerated first summary sentence) is a normal sentence ( S1230). When the first summary sentence is determined to be an abnormal sentence, the processor may correct the first summary sentence by inputting it to the language model (S1240), and output the corrected first summary sentence (S1250). On the other hand, if the first summary sentence is determined to be a normal sentence, the processor may output the initially generated first summary sentence (or regenerated first summary sentence) (S1250).

The flowchart shown in FIG. 12 and the above description are only examples, and in some embodiments, one or more steps may be performed simultaneously or one or more steps may be repeatedly performed multiple times.

The above method may be provided as a computer program stored in a computer readable recording medium to be executed on a computer. The medium may continuously store programs executable by a computer or temporarily store them for execution or download. In addition, the medium may be various recording means or storage means in the form of a single or combined hardware, but is not limited to a medium directly connected to a certain computer system, and may be distributed on a network. Examples of the medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROM and DVD, magneto optical media such as floptical disks, and Anything configured to store program instructions may include a ROM, RAM, flash memory, or the like. In addition, examples of other media include recording media or storage media managed by an app store that distributes applications, a site that supplies or distributes various other software, and a server.

The methods, acts or techniques of this disclosure may be implemented by various means. For example, these techniques may be implemented in hardware, firmware, software, or combinations thereof. Those skilled in the art will appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchange of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and the design requirements imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementations should not be interpreted as causing a departure from the scope of the present disclosure.

In a hardware implementation, the processing units used to perform the techniques may include one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs) ), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, electronic devices, and other electronic units designed to perform the functions described in this disclosure. , a computer, or a combination thereof.

Accordingly, the various illustrative logical blocks, modules, and circuits described in connection with this disclosure may be incorporated into a general-purpose processor, DSP, ASIC, FPGA or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or may be implemented or performed in any combination of those designed to perform the functions described in A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, eg, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other configuration.

In firmware and/or software implementation, the techniques include random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), PROM ( on a computer readable medium, such as programmable read-only memory (EPROM), erasable programmable read-only memory (EPROM), electrically erasable PROM (EEPROM), flash memory, compact disc (CD), magnetic or optical data storage device, or the like. It can also be implemented as stored instructions. Instructions may be executable by one or more processors and may cause the processor(s) to perform certain aspects of the functionality described in this disclosure.

If implemented in software, the techniques may be stored on or transmitted over as one or more instructions or code on a computer readable medium. Computer readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A storage media may be any available media that can be accessed by a computer. By way of non-limiting example, such computer readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or desired program code in the form of instructions or data structures. It can be used for transport or storage to and can include any other medium that can be accessed by a computer. Also, any connection is properly termed a computer-readable medium.

For example, if the software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, the coaxial cable , fiber optic cable, twisted pair, digital subscriber line, or wireless technologies such as infrared, radio, and microwave are included within the definition of medium. Disk and disc as used herein include CD, laser disc, optical disc, digital versatile disc (DVD), floppy disk, and blu-ray disc, where disks are usually magnetic data is reproduced optically, whereas discs reproduce data optically using a laser. Combinations of the above should also be included within the scope of computer readable media.

A software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium can be coupled to the processor such that the processor can read information from or write information to the storage medium. Alternatively, the storage medium may be integral to the processor. The processor and storage medium may reside within an ASIC. An ASIC may exist within a user terminal. Alternatively, the processor and storage medium may exist as separate components in a user terminal.

Although the embodiments described above have been described as utilizing aspects of the presently-disclosed subject matter in one or more stand-alone computer systems, the disclosure is not limited thereto and may be implemented in conjunction with any computing environment, such as a network or distributed computing environment. . Further, aspects of the subject matter in this disclosure may be implemented in a plurality of processing chips or devices, and storage may be similarly affected across multiple devices. These devices may include PCs, network servers, and portable devices.

Although the present disclosure has been described in relation to some embodiments in this specification, various modifications and changes may be made without departing from the scope of the present disclosure that can be understood by those skilled in the art. Moreover, such modifications and variations are intended to fall within the scope of the claims appended hereto.

110: user 120: user terminal
210: user terminal 220: network
230: information processing system 240: very large language model

Claims (17)

A document summarization method, performed by at least one processor, comprising:
receiving a document including a first set of sentences;
inputting the first sentence set to a category analyzer and extracting a second sentence set belonging to a first category, wherein the second sentence set is a subset of the first sentence set;
clustering the second sentence set belonging to the first category into a first cluster set;
extracting a third sentence set from the first cluster set, the third sentence set being a subset of the second sentence set; and
generating a first summary sentence associated with a first category of the document by inputting the third set of sentences into a language model;
Including, document summary method. According to claim 1,
inputting the first sentence set to a category analyzer and extracting a fourth sentence set belonging to a second category, wherein the fourth sentence set is a subset of the first sentence set;
clustering the fourth sentence set belonging to the second category into a second cluster set;
extracting a fifth sentence set from the second cluster set, the fifth sentence set being a subset of the fourth sentence set; and
generating a second summary sentence associated with a second category of the document by inputting the fifth set of sentences to the language model;
Further comprising a document summary method. According to claim 1,
The step of clustering the second sentence set into a first cluster set,
determining a score of each of the second sentence sets based on at least one of a sentence length, a length of a part associated with the first category in the sentence, whether a redundant morpheme exists, or whether a symbol is included; and
Clustering sentences having a score equal to or higher than a predetermined threshold among the second sentence set into a first cluster set.
Including, document summary method. According to claim 1,
Extracting the third set of sentences,
extracting one sentence from each of the first cluster set;
Including, document summary method. According to claim 1,
Extracting the third set of sentences,
selecting top n clusters (n is 2 or more) from the first cluster set based on the cluster size; and
Extracting one sentence from each of the selected top n clusters
Including, document summary method. According to claim 1,
The third sentence set is sorted in ascending order based on sentence length and input to the language model. According to claim 1,
The summary sentence associated with the first category is generated by inputting the third sentence set and a summary sample associated with the first category together into the language model. According to claim 1,
The summary sentence associated with the first category is generated by inputting the third sentence set, a summary sample associated with the first category, and information about the first category to the language model. According to claim 1,
verifying whether the first summary sentence is authentic by inputting the first summary sentence and the third set of sentences to a fact check unit; and
Regenerating the first summary sentence by inputting the third set of sentences to a language model in response to determining that the summary sentence associated with the first category is false.
Further comprising a document summary method. According to claim 9,
The step of verifying whether the summary sentence associated with the first category is authentic,
determining a relationship between the third set of sentences and the first summary sentence using a natural language inference (NLI) model; and
In response to determining that the third set of sentences and the first summary sentence are not entailment, determining the first summary sentence to be false.
Including, document summary method. According to claim 10,
Wherein the fact checker determines a relationship between two sentences of the third set of sentences and the first summary sentence. According to claim 9,
wherein the regenerated first summary sentence is generated more similar to at least one of the third set of sentences compared to the first summary sentence. According to claim 12,
verifying authenticity of the regenerated first summary sentence by inputting the regenerated first summary sentence and the third sentence set to the fact check unit; and
In response to determining that the first summary sentence is true, outputting the regenerated first summary sentence.
Further comprising a document summary method. According to claim 1,
determining whether the first summary sentence is a normal sentence; and
Correcting the first summary sentence by inputting the first summary sentence to the language model in response to determining that the first summary sentence is an abnormal sentence.
Further comprising a document summary method. According to claim 1,
the document includes a plurality of user reviews associated with a particular product;
Wherein the first summary sentence is generated by inputting the third sentence set, a summary sample associated with the first category, information on the first category, and the name of the specific product to the language model. A computer program stored in a computer-readable recording medium to execute the method according to any one of claims 1 to 15 on a computer. As a document summary system,
communication module;
Memory; and
at least one processor connected to the memory and configured to execute at least one computer readable program contained in the memory
including,
The at least one program,
Receive a document containing a first set of sentences;
inputting the first sentence set to a category analyzer to extract a second sentence set belonging to the first category, the second sentence set being a subset of the first sentence set;
clustering the second set of sentences belonging to the first category into a first cluster set;
extracting a third set of sentences from the first set of clusters, the third set of sentences being a subset of the second set of sentences;
and instructions for inputting the third set of sentences into a language model to generate a first summary sentence associated with the first category of the document.

Images