KR102506778B1 - Method and apparatus for analyzing risk of contract - Google Patents

Abstract

According to an embodiment of the present disclosure, a contract risk analysis method performed by a computing device including at least one processor is disclosed. The method includes: determining a type of input contract; obtaining a corresponding answer to at least one question included in the checklist based on a checklist corresponding to the input contract and the determined type through a learned question-and-answer model; and generating a risk analysis result by comparing the corresponding answer with a reference answer to the query.

Description

Contract risk analysis method and device {METHOD AND APPARATUS FOR ANALYZING RISK OF CONTRACT}

The present invention relates to a method for analyzing contract risk, and more particularly, to a method for analyzing contract risk using an artificial neural network.

There are various methods of text analysis in the field of natural language processing. In addition, as technology related to artificial neural networks has recently developed, text analysis methods based on artificial neural network models are also in the limelight. Through natural language processing technology, a computing device can understand the contents of a legal document and can further create a legal document.

In general, legal services are services that require high costs. For example, if the amount of the claim is small, the amount to be paid to the legal representative representing the case accounts for a relatively high percentage of the amount of the claim. In this case, even though legal judgment by experts is required, there may be a problem of not being able to use legal services due to the burden of cost. Such problems cause the need to create an environment in which a large number of people can receive legal services by providing low-cost legal services.

Therefore, there has been a continuous demand in the industry for a method for dramatically shortening the review time of a contract written through natural language processing technology and a method for reducing contract review costs for small business owners or SMEs who are vulnerable to legal contracts accordingly.

Korean registered patent KR10-2022412 discloses an automatic legal document creation system and method.

The present disclosure has been made in response to the above background art, and aims to provide a method for analyzing contract risk using an artificial neural network.

According to an embodiment of the present disclosure for realizing the above object, a contract risk analysis method performed by a computing device including at least one processor is disclosed. The method includes: determining a type of input contract; obtaining a corresponding answer to at least one question included in the checklist based on a checklist corresponding to the input contract and the determined type through a learned question-and-answer model; and generating a risk analysis result by comparing the corresponding answer with a reference answer to the query.

In an alternative embodiment, the type of the input contract may be determined based on a learned classification model independently of the question-and-answer model.

In an alternative embodiment, the classification model may include a higher classification model for matching an input contract with one higher type of a plurality of higher types; and a lower classification model for matching the input contract with one of a plurality of lower types included in the matched higher type.

In an alternative embodiment, the higher class classification model uses the input contract as an input and generates one higher type corresponding to the input contract as output information, and a plurality of higher level types that can be output from the higher class classification model. A plurality of sub-classification models exist to correspond to each of the types, and a sub-classification model for matching the input contract to one sub-type among the plurality of sub-classification models is based on the output information of the upper classification model. can be determined

In an alternative embodiment, the type of the input contract may include: a classification result obtained by inputting a plurality of text tokens included in the input contract into the learned classification model; and a weight based on the positions of the plurality of text tokens.

In an alternative embodiment, the position-based weight may include: a first weight assigned to a first text token of the plurality of text tokens at a preceding position within the input contract; and a second weight assigned to a second text token existing in a trailing position in the input contract among the plurality of text tokens, and the first weight is set to have a higher value than the second weight. It can be.

In an alternative embodiment, the input contract is divided into a preceding position area and a following position area, and a first weight assigned to a first text token present in the preceding position area corresponds to a second text token present in the following position area. It has a higher value than the second weight assigned to , and at least a portion of the preceding location area and at least a portion of the following location area may overlap.

In an alternative embodiment, the checklist includes: a checklist of essential provisions for checking whether elements that must be included according to the type of contract are included in the input contract; Alternatively, a poison clause checklist for checking whether poison elements according to the type of contract are included in the input contract may be included.

In an alternative embodiment, the mandatory clause checklist may include a first query generated based on a legal provision corresponding to the type of contract or a second query generated based on a precedent.

In an alternative embodiment, the question-and-answer model may include text including contract details; one or more inquiries about the terms of the agreement; and at least one learning data including reference answers labeled in the text to correspond to the one or more queries.

In an alternative embodiment, the question-answering model may include: inputting text including contract details and one or more questions about the contract contents into the question-answer model; obtaining a corresponding answer to the question in the text from the question answering model; and updating at least one parameter included in the question-answering model by comparing the corresponding answer with a reference answer to the question.

In an alternative embodiment, the one or more queries for the terms of the contract may include a replacement query in which at least some of the text tokens included in the other queries are substituted with different values.

In an alternative embodiment, the risk analysis result may include an analysis result of whether a mandatory provision is omitted or an analysis result of whether a poison clause is included.

In an alternative embodiment, the risk analysis result may be generated based on a text token-based similarity analysis result between the corresponding answer and the reference answer.

In an alternative embodiment, the risk analysis result may be generated based on an output value obtained as the corresponding answer and the reference answer are input to a risk determination model independently learned from the question answering model.

In an alternative embodiment, the risk analysis result may include a text token-based similarity analysis result for the corresponding answer and the reference answer; And it may be generated based on a result of applying weights to output values of the risk determination model learned for the corresponding answer and the reference answer, respectively.

In an alternative embodiment, generating contract summary information based on the corresponding answers obtained from the learned question-and-answer model; Alternatively, the method may further include generating recommendation information for a contract based on the risk analysis result.

A computer program stored in a computer readable storage medium is disclosed according to an embodiment of the present disclosure for realizing the above object. The computer program, when executed on one or more processors, is configured to perform the following operations for analyzing contract risk, the operations comprising: determining a type of an input contract; obtaining an answer corresponding to at least one question included in the checklist based on the checklist corresponding to the input contract and the determined type through a learned question-and-answer model; and generating a risk analysis result by comparing the corresponding answer with a reference answer to the query.

A contract risk analysis apparatus is disclosed according to an embodiment of the present disclosure for realizing the above object. The device may include one or more processors; Memory; And a network, and the one or more processors determine the type of the input contract, and include the checklist based on the checklist corresponding to the input contract and the determined type through a learned question-and-answer model. Corresponding answers to at least one query may be obtained, and a risk analysis result may be generated by comparing the corresponding answers with a reference answer to the query.

The present disclosure may provide a method of analyzing contract risk using an artificial neural network.

1 is a block diagram of a computing device for analyzing contract risk according to an embodiment of the present disclosure.
2 is a schematic diagram illustrating a network function according to an embodiment of the present disclosure.
3 is a flowchart of a risk analysis method according to an embodiment of the present disclosure.
4 is a flowchart of a method for learning a question-answering model according to an embodiment of the present disclosure.
5 is a simplified and general schematic diagram of an exemplary computing environment in which embodiments of the present disclosure may be implemented.

Various embodiments are now described with reference to the drawings. In this specification, various descriptions are presented to provide an understanding of the present disclosure. However, it is apparent that these embodiments may be practiced without these specific details.

The terms “component,” “module,” “system,” and the like, as used herein, refer to a computer-related entity, hardware, firmware, software, a combination of software and hardware, or an execution of software. For example, a component may be, but is not limited to, a procedure, processor, object, thread of execution, program, and/or computer running on a processor. For example, both an application running on a computing device and a computing device may be components. One or more components may reside within a processor and/or thread of execution. A component can be localized within a single computer. A component may be distributed between two or more computers. Also, these components can execute from various computer readable media having various data structures stored thereon. Components may be connected, for example, via signals with one or more packets of data (e.g., data and/or signals from one component interacting with another component in a local system, distributed system) to other systems and over a network such as the Internet. data being transmitted) may communicate via local and/or remote processes.

In addition, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or”. That is, unless otherwise specified or clear from the context, “X employs A or B” is intended to mean one of the natural inclusive substitutions. That is, X uses A; X uses B; Or, if X uses both A and B, "X uses either A or B" may apply to either of these cases. Also, the term "and/or" as used herein should be understood to refer to and include all possible combinations of one or more of the listed related items.

Also, the terms "comprises" and/or "comprising" should be understood to mean that the features and/or components are present. However, it should be understood that the terms "comprises" and/or "comprising" do not exclude the presence or addition of one or more other features, elements, and/or groups thereof. Also, unless otherwise specified or where the context clearly indicates that a singular form is indicated, the singular in this specification and claims should generally be construed to mean "one or more".

In addition, the term “at least one of A or B” should be interpreted as meaning “when only A is included”, “when only B is included”, and “when A and B are combined”.

Those skilled in the art will further understand that the various illustrative logical blocks, components, modules, circuits, means, logics, and algorithm steps described in connection with the embodiments disclosed herein may be implemented using electronic hardware, computer software, or combinations of both. It should be recognized that it can be implemented as To clearly illustrate the interchangeability of hardware and software, various illustrative components, blocks, configurations, means, logics, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented in hardware or as software depends on the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application. However, such implementation decisions should not be interpreted as causing a departure from the scope of this disclosure.

The description of the presented embodiments is provided to enable any person skilled in the art to use or practice the present invention. Various modifications to these embodiments will be apparent to those skilled in the art of this disclosure. The general principles defined herein may be applied to other embodiments without departing from the scope of this disclosure. Thus, the present invention is not limited to the embodiments presented herein. The present invention is to be accorded the widest scope consistent with the principles and novel features set forth herein.

1 is a block diagram of a computing device for analyzing contract risk according to an embodiment of the present disclosure.

The configuration of the computing device 100 shown in FIG. 1 is only a simplified example. In one embodiment of the present disclosure, the computing device 100 may include other components for performing a computing environment of the computing device 100, and only some of the disclosed components may constitute the computing device 100.

The computing device 100 may include a processor 110 , a memory 130 , and a network unit 150 .

The processor 110 may include one or more cores, and includes a central processing unit (CPU), a general purpose graphics processing unit (GPGPU), and a tensor processing unit (TPU) of a computing device. unit), data analysis, and processors for deep learning. The processor 110 may read the computer program stored in the memory 130 and process data for contract risk analysis according to the present disclosure. Processor 110 may determine the type of input contract. The processor 110 may obtain a corresponding answer to at least one question included in the checklist through a learned question-answering model based on the input contract and the checklist corresponding to the type determined for the input contract. The processor 110 may generate a risk analysis result by comparing a corresponding answer to the query obtained from the input contract with a standard answer to the query.

According to an embodiment of the present disclosure, the memory 130 may store any type of information generated or determined by the processor 110 and any type of information received by the network unit 150 .

According to an embodiment of the present disclosure, the memory 130 is a flash memory type, a hard disk type, a multimedia card micro type, or a card type memory (eg, SD or XD memory, etc.), RAM (Random Access Memory, RAM), SRAM (Static Random Access Memory), ROM (Read-Only Memory, ROM), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory) -Only Memory), a magnetic memory, a magnetic disk, and an optical disk may include at least one type of storage medium. The computing device 100 may operate in relation to a web storage that performs a storage function of the memory 130 on the Internet. The above description of the memory is only an example, and the present disclosure is not limited thereto.

The network unit 150 according to an embodiment of the present disclosure may include any wired/wireless communication network capable of transmitting and receiving data and signals of any type, and may be included in the network represented in the present disclosure.

In the present disclosure, the network unit 150 may be configured regardless of its communication mode, such as wired and wireless, and may be configured with various communication networks such as a personal area network (PAN) and a wide area network (WAN). can In addition, the network may be the known World Wide Web (WWW), or may use a wireless transmission technology used for short-range communication, such as Infrared Data Association (IrDA) or Bluetooth.

The techniques described herein may be used in the networks mentioned above as well as other networks.

The configuration of the computing device 100 shown in FIG. 1 is only a simplified example. In one embodiment of the present disclosure, the computing device 100 may include other components for performing a computing environment of the computing device 100, and only some of the disclosed components may constitute the computing device 100.

The computing device 100 according to the present disclosure may determine the type of input contract. In this disclosure, a contract may include a legal document, which is a document related to a legal act. According to an embodiment of the present disclosure, a contract including contents agreed upon between the parties may include a document in which the contents of the agreement are left in writing rather than verbally. The contract may contain the title of the contract, the indications of the parties, the preamble, the proximal indications, the defining clauses, the substantive clauses, the general clauses, the closing words, the signature line, the appendices, etc. The preamble may include references to the parties to the contract and how the contract was concluded. The text may include normal text rules and type-specific text rules. Ordinary text provisions may include provisions relating to the subject matter and contract price. Type-specific text rules may contain supplementary text to normal text rules. The end may include cancellation, contract period, jurisdiction, validation, date of creation, signature and seal, name and seal, etc. Types of contracts according to this disclosure may include, for example, contracts of sale, leases, gift contracts, labor contracts, power of attorney contracts, subcontracts, and the like. In one embodiment of the present disclosure, the computing device ( ) may classify the types of contracts into about 270 according to the characteristics of the contracts. The input contract may be input from the user to the computing device 100 in electronic form. The computing device 100 may determine the type of the input contract based on input contract type information received from the user. A user may select a type of contract or input a name indicating the type of contract through a separate user interface. The computing device 100 may extract the title part of the input contract and determine the type of contract according to the title.

The computing device 100 according to the present disclosure may determine the type of an input contract based on an artificial neural network-based classification model independently learned from the question-answering model. The classification model may be an artificial neural network model trained in a server different from the computing device 100 .

2 is a schematic diagram illustrating a network function according to an embodiment of the present disclosure. An operation of determining an input contract type according to the present disclosure may be performed by a classification model based on an artificial neural network including at least one node.

Throughout this specification, the terms model, artificial neural network, network function, and neural network may be used interchangeably with the same meaning. A neural network may consist of a set of interconnected computational units, which may generally be referred to as nodes. These nodes may also be referred to as neurons. A neural network includes at least one node. Nodes (or neurons) constituting neural networks may be interconnected by one or more links.

In a neural network, one or more nodes connected through a link may form a relative relationship of an input node and an output node. The concept of an input node and an output node is relative, and any node in an output node relationship with one node may have an input node relationship with another node, and vice versa. As described above, an input node to output node relationship may be created around a link. More than one output node can be connected to one input node through a link, and vice versa.

In a relationship between an input node and an output node connected through one link, the value of data of the output node may be determined based on data input to the input node. Here, a link interconnecting an input node and an output node may have a weight. The weight may be variable, and may be changed by a user or an algorithm in order to perform a function desired by the neural network. For example, when one or more input nodes are interconnected by respective links to one output node, the output node is set to a link corresponding to values input to input nodes connected to the output node and respective input nodes. An output node value may be determined based on the weight.

As described above, in the neural network, one or more nodes are interconnected through one or more links to form an input node and output node relationship in the neural network. Characteristics of the neural network may be determined according to the number of nodes and links in the neural network, an association between the nodes and links, and a weight value assigned to each link. For example, when there are two neural networks having the same number of nodes and links and different weight values of the links, the two neural networks may be recognized as different from each other.

A neural network may be composed of a set of one or more nodes. A subset of nodes constituting a neural network may constitute a layer. Some of the nodes constituting the neural network may form one layer based on distances from the first input node. For example, a set of nodes having a distance of n from the first input node may constitute n layers. The distance from the first input node may be defined by the minimum number of links that must be passed through to reach the corresponding node from the first input node. However, the definition of such a layer is arbitrary for explanation, and the order of a layer in a neural network may be defined in a method different from the above. For example, a layer of nodes may be defined by a distance from a final output node.

An initial input node may refer to one or more nodes to which data is directly input without going through a link in relation to other nodes among nodes in the neural network. Alternatively, in a relationship between nodes based on a link in a neural network, it may mean nodes that do not have other input nodes connected by a link. Similarly, the final output node may refer to one or more nodes that do not have an output node in relation to other nodes among nodes in the neural network. Also, the hidden node may refer to nodes constituting the neural network other than the first input node and the last output node.

In the neural network according to an embodiment of the present disclosure, the number of nodes in the input layer may be the same as the number of nodes in the output layer, and the number of nodes decreases and then increases again as the number of nodes progresses from the input layer to the hidden layer. can In addition, the neural network according to another embodiment of the present disclosure may be a neural network in which the number of nodes of the input layer may be less than the number of nodes of the output layer and the number of nodes decreases as the number of nodes increases from the input layer to the hidden layer. there is. In addition, the neural network according to another embodiment of the present disclosure is a neural network in which the number of nodes in the input layer may be greater than the number of nodes in the output layer, and the number of nodes increases as the number of nodes increases from the input layer to the hidden layer. can A neural network according to another embodiment of the present disclosure may be a neural network in the form of a combination of the aforementioned neural networks.

A deep neural network (DNN) may refer to a neural network including a plurality of hidden layers in addition to an input layer and an output layer. Deep neural networks can reveal latent structures in data. In other words, it can identify the latent structure of a photo, text, video, sound, or music (e.g., what objects are in the photo, what the content and emotion of the text are, what the content and emotion of the audio are, etc.). . Deep neural networks include convolutional neural networks (CNNs), recurrent neural networks (RNNs), auto encoders, generative adversarial networks (GANs), and restricted boltzmann machines (RBMs). machine), a deep belief network (DBN), a Q network, a U network, a Siamese network, a Generative Adversarial Network (GAN), and the like. The description of the deep neural network described above is only an example, and the present disclosure is not limited thereto.

In one embodiment of the present disclosure, the network function may include an autoencoder. An autoencoder may be a type of artificial neural network for outputting output data similar to input data. An auto-encoder may include at least one hidden layer, and an odd number of hidden layers may be disposed between input and output layers. The number of nodes of each layer may be reduced from the number of nodes of the input layer to an intermediate layer called the bottleneck layer (encoding), and then expanded symmetrically with the reduction from the bottleneck layer to the output layer (symmetrical to the input layer). Autoencoders can perform non-linear dimensionality reduction. The number of input layers and output layers may correspond to dimensions after preprocessing of input data. In the auto-encoder structure, the number of hidden layer nodes included in the encoder may decrease as the distance from the input layer increases. If the number of nodes in the bottleneck layer (the layer with the fewest nodes located between the encoder and decoder) is too small, a sufficient amount of information may not be conveyed, so more than a certain number (e.g., more than half of the input layer, etc.) ) may be maintained.

In some embodiments of the present disclosure, an artificial neural network model may be trained by at least one of supervised learning, unsupervised learning, semi supervised learning, or reinforcement learning. can Learning of the neural network may be a process of applying knowledge for the neural network to perform a specific operation to the neural network.

A neural network can be trained in a way that minimizes output errors. In the learning of the neural network, the learning data is repeatedly input into the neural network, the output of the neural network for the training data and the error of the target are calculated, and the error of the neural network is transferred from the output layer of the neural network to the input layer in the direction of reducing the error. It is a process of updating the weight of each node of the neural network by backpropagating in the same direction. In the case of teacher learning, the learning data in which the correct answer is labeled is used for each learning data (ie, the labeled learning data), and in the case of comparative teacher learning, the correct answer may not be labeled in each learning data. That is, for example, learning data in the case of teacher learning regarding data classification may be data in which each learning data is labeled with a category. Labeled training data is input to a neural network, and an error may be calculated by comparing an output (category) of the neural network and a label of the training data. As another example, in the case of comparative history learning for data classification, an error may be calculated by comparing input learning data with a neural network output. The calculated error is back-propagated in a reverse direction (ie, from the output layer to the input layer) in the neural network, and the connection weight of each node of each layer of the neural network may be updated according to the back-propagation. The amount of change in the connection weight of each updated node may be determined according to a learning rate. The neural network's computation of input data and backpropagation of errors can constitute a learning cycle (epoch). The learning rate may be applied differently according to the number of iterations of the learning cycle of the neural network. For example, a high learning rate may be used in the early stage of neural network training to increase efficiency by allowing the neural network to quickly obtain a certain level of performance, and a low learning rate may be used in the late stage to increase accuracy.

In neural network learning, generally, training data can be a subset of real data (ie, data to be processed using the trained neural network), and therefore, errors for training data are reduced, but errors for real data are reduced. There may be incremental learning cycles. Overfitting is a phenomenon in which errors for actual data increase due to excessive learning on training data. For example, a phenomenon in which a neural network that has learned a cat by showing a yellow cat does not recognize that it is a cat when it sees a cat other than yellow may be a type of overfitting. Overfitting can act as a cause of increasing the error of machine learning algorithms. Various optimization methods can be used to prevent such overfitting. To prevent overfitting, methods such as increasing the training data, regularization, inactivating some nodes in the network during learning, and using a batch normalization layer should be applied. can

According to an embodiment of the present disclosure, the computing device 100 may input at least a part of the text included in the input contract into the learned classification model and determine the type of the input contract according to an output value of the probability-based classification model. The learned classification model may be an artificial neural network model trained to perform a classification task for determining the type of contract. As a result of inputting at least some text included in the contract into the learned classification model, the computing device 100 may obtain a probability distribution for each classification class included in the plurality of classification classes for the contract. The plurality of classification classes are first type, second type, . . . , and the Nth type. The computing device 100 may determine the type of contract according to the classification class having the highest probability value in the probability distribution.

According to an embodiment of the present disclosure, a classification model for classifying input contracts by the computing device 100 may include an upper classification model and a lower classification model. The higher classification model may be a model for matching an input contract with one higher type among a plurality of higher types. For example, the upper type may include a sales contract type, a lease contract type, and the like. The lower classification model may be a model for matching the input contract with one of a plurality of lower types included in the higher type matched to the input contract by the higher classification model. For example, the upper type of sales contract type may include a land sales contract type, a shopping mall sales contract type, a farmland sales contract type, a housing sales contract type, a machine sales contract type, and the like as subtypes. As another example, subtypes included in the upper type of lease agreement may include land lease agreements, car rental agreements, commercial rental agreements, housing rental agreements, and the like. The upper classification model may generate one higher type corresponding to the input contract as output information by using the input contract as an input. In some embodiments according to the present disclosure, a plurality of lower classification models may exist to correspond to each of a plurality of higher types output from the higher classification model. For example, when an upper classification model determines an input contract as a sales contract type, a lower classification model for determining whether the input contract is a land sales contract or a housing sales contract among sales contracts may exist. In another embodiment, when the upper classification model determines the type of input contract as a lease contract, a lower classification model for determining whether the type of input contract is a land lease contract or a commercial lease contract may separately exist. The computing device 100 may determine a lower classification model for matching the input contract to one lower type among a plurality of lower classification models based on output information of the upper classification model. In the above example, the computing device 100 of the present disclosure determines the type of the input contract based on a classification model including an upper classification model and a lower classification model, thereby primarily classifying the type of contract according to the upper type, and then , it can be classified secondarily according to the subtype included in each supertype. This has the effect of approaching the problem of contract classification in a multidimensional manner rather than a one-dimensional approach. For example, differences between contracts belonging to the same parent category but differing only in subcategories may be more difficult to distinguish than differences between contracts in different subcategories. In this case, it is more efficient to first classify the upper category in a large range and then divide the lower types included in the common upper type secondarily, as in the classification model structure according to the present disclosure. Accordingly, the classification model of the present disclosure has an advantage of being able to more accurately and precisely distinguish differences between contracts on a contract type structure having a multidimensional structure.

According to an embodiment of the present disclosure, the type of the input contract determined by the computing device 100 may include a classification result obtained by inputting a plurality of text tokens included in the input contract into a learned classification model and locations of the plurality of text tokens. It can be determined based on a weight based on . A text token according to an embodiment of the present disclosure may include a natural language unit used for natural language processing. The text token may include, for example, a character unit, a word unit, an auxiliary word, a morpheme unit, a word unit, a phrase unit, and the like. Generating text tokens in the input contract in this way may mean segmentation/tokenization of the input contract or sentences of the input contract into arbitrary units. The text token may be generated according to any criteria pre-determined from the input contract. For example, the text token may be generated based on a word of text included in the input contract. The position in the position-based weight of the plurality of text tokens may be based on a word reference distance away from the starting point of the document. A position in the position-based weight may mean a predetermined specific part of a document, and a relatively high weight may be assigned to such a position. For example, when the first text token included in the input contract acts as the most significant factor in determining the type of the input contract, the computing device 100 determines the classification model learned for the text token appearing first in the input contract. It can be set to determine the type of input contract by assigning the greatest weight to the resulting value.

The weight based on the position is a first weight assigned to a first text token existing in a preceding position in the input contract among a plurality of text tokens included in the input contract and an input contract among a plurality of text tokens included in the input contract. and a second weight assigned to a second text token existing at a trailing position in the text token, but the first weight may be set to have a higher value than the second weight. The preceding position and the following position may be positions determined according to a distance from a starting point of the document. For example, “If the lessor or lessee does not fulfill his/her obligations under this Agreement and related laws, he or she must compensate the other party for the damages caused thereby.” If there is a sentence, a higher weight may be assigned to each text token generated on the basis of a word as it is positioned in front of the sentence. That is, higher weights can be assigned to text tokens existing in the preceding position, such as 0.9 for 'lessor', 0.9 for 'lessee', 0.8 for 'debt', and 0.8 for 'fulfillment'. This may be a result reflecting the domain characteristic that the more text tokens present at the front of the input contract, the more significant the factor in determining the type of contract.

In one embodiment of the present disclosure, the computing device 100 may divide the input contract into a preceding location area and a subsequent location area in order to determine the type of the input contract. The computing device 100 may assign a higher value to the first weight assigned to the first text token in the preceding location area than the second weight assigned to the second text token in the subsequent location area. In addition, at least a portion of the preceding location area and at least a portion of the following location area may overlap. For example, if the number of text tokens for determining the type of the input contract is 350, the computing device 100 sets the preceding location area to the first 200 tokens and the trailing location area to the later 200 tokens. can In this case, the last 50 tokens of the preceding location area and the front 50 tokens of the following location area may overlap each other. The computing device 100 may obtain a first probability distribution for the type of contract obtained as 200 tokens corresponding to the preceding location area are sequentially or in parallel input into the learned classification model. In addition, the computing device 100 may obtain a second probability distribution for the type of contract obtained as 200 tokens corresponding to the trailing location area are sequentially or in parallel input into the learned classification model. Finally, the computing device 100 may obtain a final probability distribution by multiplying the first probability distribution by the first weight, multiplying the second probability distribution by the second weight, and summing the results of the two multiplications. The example of the number of text tokens and the number of overlapping tokens is only an example for explanation and does not limit the present disclosure. The computing device 100 may determine the type of input contract based on the final probability distribution. The computing device 100 overlaps at least a portion of the preceding location area with at least a portion of the following location area so that context is not lost in the process of determining the type of contract, and the result value of the classification model for the preceding location area and the following location area are overlapped. Excessive variation in result values of the classification model for the location area can be prevented. The computing device 100 according to the present disclosure determines the type of an input contract based on a plurality of text tokens included in the input contract, and assigns a relatively high weight to the first text token present in the front part, and selects the first text token. The type of the input contract may be determined by using a second text token that exists later.

The computing device 100 according to the present disclosure may obtain a corresponding answer to at least one question included in the checklist based on the input contract and the checklist corresponding to the determined type through the learned question-answering model. can The checklist is a checklist of essential provisions for checking whether elements that must be included according to the type of contract are included in the input contract or whether exclusive elements according to the type of contract are included in the input contract. A toxin clause checklist may be included.

The essential clause checklist according to the present disclosure may include a first query generated based on a legal provision corresponding to a type of contract or a second query generated based on a precedent. For example, when the input contract corresponds to an employment contract, the mandatory clause checklist may include a first query generated based on the Labor Standards Act, which is a legal provision related to an employment contract. The first query created based on the Labor Standards Act is “From when to when is the term of the labor contract?”, “Has the labor contract issued in accordance with the Labor Standards Act?” or “What is your monthly salary?” The same or similar queries may be included. In addition, if the input contract corresponds to an employment contract, the second query generated based on the precedent asks, “Is the wage paid above the minimum wage?” etc. may be included.

The essential clause checklist according to the present disclosure may include questions corresponding to each legal requirement included in the legal clause corresponding to the type of contract. For example, Article 12 of the Subcontracting Act (prohibition of claims for unfair payment of goods purchase price, etc.) - “The principal contractor makes the subcontractor purchase goods necessary for manufacturing, repairing, constructing, or performing services from the subcontractor, or In the case of using equipment, etc., no act falling under any of the following subparagraphs shall not be performed without a justifiable reason.” In order to determine whether the requirements were complied with, the checklist of essential provisions according to this disclosure includes “Did the Contractor have the subcontractor purchase goods necessary for manufacturing, repairing, constructing, or performing services from itself?”, “Did the Contractor purchase from itself?” Does it fall under the case of using the equipment of the company?”, “Is there no justifiable reason?” or “Have you committed any of the following acts?” etc. may be included. As described above, the essential clause checklist according to the present disclosure includes a question corresponding to legal requirements, so that the computing device 100 determines whether a specific word is included in the input contract as well as a requirement to generate a specific legal effect. You can determine if it is included in the contract. The foregoing example is only an example for explanation, and the present disclosure may include various kinds of essential clause checklists used to confirm contents that must be included in each type of contract.

The poison clause checklist according to the present disclosure may be based on a dictionary of poison terms corresponding to the type of contract. The toxic term dictionary corresponding to the type of the contract may include at least one negative term generally having a toxic element in the content of the contract. For example, in subcontracting contracts, a toxic terminology might include the term “deferred payment”. In the poison clause checklist for subcontract contracts, “Is there a clause related to delay compensation?” created based on the poison term “deferred compensation”. or “Does the provision regarding damages exist only for one of the parties?” It may contain queries such as As another example, terms such as “aggregate transfer” may be included in a dictionary of exclusive terms in contracts related to intellectual property rights. In this case, the poison clause checklist was created based on the poison term “batch transfer”, “Is there a clause on lump-sum transfer of intellectual property rights without conditions?” may include a query. The foregoing example is merely an example for explanation and does not limit the present disclosure.

The computing device 100 according to the present disclosure may obtain a corresponding answer from the input contract by using a checklist of essential provisions or a checklist of foreign terms prepared based on a dictionary of foreign terms corresponding to the type of contract. The toxin clause checklist may include a query about whether or not toxin terms are included. The poison clause checklist may include a query capable of determining the nature of a sentence containing a poison term. Determination of the nature of the sentence may be performed based on a learned question-answering model. The question-answering model will be described in detail later.

Hereinafter, a question answering model according to the present disclosure will be described. Some of the contents of the structure and learning method of the artificial neural network-based question-answering model according to the present disclosure partially overlap with the contents of the classification model described above with reference to FIG. 2, so the following will focus on the differences.

A question-answering model according to the present disclosure includes at least one node, and at least one parameter whose value can be changed may be assigned to the node. The question-answering model may be learned based on at least one piece of training data including text including contract details, one or more questions about the contract details, and reference answers labeled in the text to correspond to the one or more questions. The learning data may include, for example, first text data such as “(input contract)_'How much is the contract price?'_'100 million won in gold'”. The learning data may include second text data such as, for example, “(Input contract)_'From when to when is the contract period?'_From November 11, 2020 to November 11, 2022'” .

One or more queries about contract contents included in the learning data may include a replacement query in which at least a part of text tokens included in other queries are substituted with other values. In a first embodiment according to the present disclosure, the alternative query may be generated by changing the word order of text tokens of other queries. For example, when there is a query such as “How much is the contract price?”, the computing device 100 changes the word order of the text token to ask “How much? The contract price”, “How much is the contract? You can create an alternative query, such as "how much is the price". The computing device 100 according to the present disclosure generates a plurality of alternative queries from one query through the first embodiment, so that when the same word is included, the meaning is maintained even if the word order is changed, taking advantage of the characteristics of the Korean language. It is possible to increase the number of learning data by multiplying.

In a second embodiment according to the present disclosure, the replacement query may be generated by replacing at least one of the text tokens of another query with a similar word. For example, in a query such as “How much is the contracted amount?”, the computing device 100 may ask “How much is the contracted amount?”, “How much is the contracted amount?”, “What is the contracted payment amount?”, “How much is the contract price?” You can create alternative queries such as

In a third embodiment according to the present disclosure, the replacement query may be generated from another query through a separate translation model. The translation model may be a model capable of bi-directional translation between the first language and the second language. In this case, the replacement query according to the present disclosure may be obtained by translating into the second language through the translation model and then translating it into the first language. For example, in the case of the Korean-English bidirectional translation model, the query “What is the contract amount?” can be translated into an English sentence “what is the contract amount?”, and then the English sentence is translated back into Korean. would yield an alternative query such as “what is the contract amount?”

In the second or third embodiments, the computing device 100 uses a method of replacing a specific word with a similar word or a method of generating a similar sentence for a specific sentence using a translation model, etc. By generating a plurality of replacement queries from a query, the number of training data can be increased.

In the fourth embodiment according to the present disclosure, the replacement query may be generated by discarding at least some of the text tokens of other queries or replacing them with noise values. For example, in a query such as “How much is the contracted amount?”, the computing device 100 responds with “How much is the contracted amount?”, “What is the contracted amount?”, “When is the contracted amount?” You can create alternative queries such as In the fourth embodiment, the present disclosure enables a question-answering model to be learned by learning data containing noise, so that even if a question sentence input to a question-answering model includes a somewhat ambiguous expression, the question-answering model is able to learn context from the corresponding question. can be recognized and the corresponding answer can be output. That is, a learned question-answering model that is robust to noise can be obtained.

The question-answering model of the present disclosure includes the steps of inputting a text including contract details and one or more questions about the contract contents into a question-answering model, obtaining a corresponding answer to the query from the text from the question-answering model, and Learning may be performed by comparing an answer with a reference answer to the question and updating at least one parameter included in the question-answering model.

A question-answering model according to the present disclosure may be pretrained to understand the linguistic structure of an input sentence. A question-answering model according to the present disclosure may be pre-learned based on a dictionary of legal terms. A question-answering model may be pre-trained to generate word embedding vectors considering the context of a legal term. The dictionary learning method may be based on a semi-supervised learning method of masking some text tokens included in a sentence in a sentence including legal terms and matching the masked text tokens through output of a question-answer model. . The dictionary learning method may include a learning method for inputting two arbitrary sentences included in a legal document and matching whether a following sentence is a sentence that can follow a preceding sentence. In the present disclosure, the method of learning a question-answering model to obtain a corresponding answer to at least one question from an input contract may be subsequently performed on the question-and-answering model for which pre-learning has been completed as described above. Learning of the question-answering model may be performed by the computing device 100, and after learning is performed by a separate external server, the computing device 100 of the present disclosure transmits information on the learned question-and-answer model to the network unit ( 150) can also be obtained.

4 is a flowchart of a method for learning a question-answering model according to an embodiment of the present disclosure. In a learning process, the question-answering model may receive text including contract details and one or more questions about contract contents (S410). The text including the contents of the contract may be at least part of the input contract. At least part of the input contract may be selected based on units such as paragraph units, page units, or arbitrary text length units. The one or more queries regarding the contents of the contract may be queries included in a checklist according to the present disclosure. The computing device 100 may acquire a corresponding answer to the query in the text from the question response model (S430). In order to train the question-answering model, the computing device 100 may compare a corresponding answer with a reference answer to the question and update at least one parameter included in the question-answering model (S450). Updating the at least one parameter may be performed by a backpropagation technique as described above.

For example, in step S410, the question-and-answer model in the learning process, as an input, “The validity period of this contract is 12 months from the date of signing the contract, and if there is no notice of cancellation until 3 months before the end of the contract, it is automatically extended by one year, , It is the same after that” along with a text such as “How long is the validity period of the contract?” may be received. At this time, the question-answering model being learned outputs “contract end” as a corresponding answer in step S430, and the computing device 100 may obtain the corresponding answer. The question-answering model may output not only the text itself corresponding to the answer to the question in the input text, but also the position of the text token, but the present disclosure is not limited thereto. In step S450, the computing device 100 compares the standard answer "12 months" and the corresponding answer "contract termination" to the question "How long is the validity period of the contract?" parameters can be updated. When the question-answering model according to the present disclosure is completed, the computing device 100 may input text and a question into the question-answering model and obtain an answer to the question from the text.

The computing device 100 according to the present disclosure may generate a risk analysis result by comparing a corresponding answer based on the learned question answering model with a reference answer to the question. The corresponding answers may be obtained based on a checklist including one or more questions according to input contracts and types of input contracts through a learned question-answering model.

Criteria answers according to the present disclosure may exist according to the queries included in the checklist. The standard answer may include answer text for true/false determination according to the query. For example, the standard answer to the question “Is there a provision that only one of the parties is obligated to fulfill?” could be “no”. The standard answer may include standard answer contents according to the query. For example, a standard answer to the question “How much are working hours?” may be “Working hours are based on 40 hours per week, 8 hours per day, excluding break time.” The standard answer may include a regular expression for a grammatical structure of an answer text according to a query. For example, the standard answer to the question “What is the basic wage?” can be “{number} won” or “{number} won”. {number} can be a regular expression to match a positive natural number. When the standard answer includes a regular expression expression, the computing device 100 provides a standard answer when the corresponding answer obtained from the input contract through the question-and-answer model to the question "What is the basic wage?" is "50,000,000 won in gold". It can be judged to be consistent with “gold {number} won’.

The risk analysis result generated by the computing device 100 according to the present disclosure by comparing the corresponding answer and the standard answer may include an analysis result on whether essential provisions are omitted or whether poison clauses are included. In one embodiment, the input contract is a housing lease contract, the query is "How long is the lease term?", the response obtained through the learned question-and-answer model is "until November 22, 2022", and the standard answer is "delivery date" to {year} year, {month} month, {day} day", the computing device 100 may generate an analysis result indicating that the point of time of the rental period is omitted through comparison of both answers.

A risk analysis result according to the present disclosure may be generated based on a text token-based similarity analysis result between a corresponding answer and a standard answer. The computing device 100 of the present disclosure may calculate the frequency of text tokens included in answers for text token-based similarity analysis. The text token-based similarity analysis method according to the present disclosure may include, for example, a TF-IDF technique, an LDA modeling technique, and the like.

In some embodiments according to the present disclosure, the computing device 100 may generate a TF-IDF matrix including element values based on the frequency of each text token based on a plurality of answers. The TF-IDF technique increases the importance of text tokens that frequently appear in one document and lowers the importance of text tokens that frequently occur in the entire document. TF (Term Frequency) is a value indicating how often a specific term appears in a document, and the higher this value is, the more important it is in the document. DF (Document Frequency) is a value that indicates how often a specific term appears in the entire document. This value is The higher the value, the lower the importance of the term. The reciprocal of the DF value is called IDF, and TF-IDF is a value obtained by multiplying TF and IDF. The computing device 100 may calculate a text token-based similarity analysis result for an answer set including one or more standard answers and one or more corresponding answers pre-generated by a question-and-answer model. The computing device 100 may identify the number of times a specific text token appears in a specific answer in order to calculate the TF value. That is, when there are N answers and M text tokens exist, the computing device 100 determines that i (i is 0 or more and less than M) th text token j (j is 0 or more and less than N) for each of the NxM number of matrix elements. )th answer, it is possible to assign an element value based on the frequency by identifying how many times it appears. The computing device 100 may set the frequency of occurrence of words as the TF value, or may adjust the TF value using a log function. Computing device 100 may calculate an IDF value for each text token and each answer pair based on the number of answers that include that particular text token and the total number of answers. The computing device 100 may generate a TF-IDF matrix in which a TF-IDF value is assigned to an answer-text token pair consisting of each text token and each answer based on the above-described TF and IDF values. A row in the TF-IDF matrix may represent the distribution of text tokens included in each answer. In the TF-IDF matrix, a column may indicate the number of times each text token is used in each answer. The answer may include a corresponding answer obtained by a question-answering model or a standard answer to a question of the computing device . A row of the TF-IDF matrix may be used as a vector representing the distribution of text tokens included in the answer. The foregoing example is only an example for explanation and does not limit the present disclosure.

In some embodiments according to the present disclosure, computing device 100 may generate an LDA matrix including element values based on the frequency of each text token based on a plurality of answers. The Latent Dirichlet Allocation (LDA) model is a generative probabilistic model that infers latent themes in a document. Topic extraction according to the LDA modeling technique is as follows. First, each word is assigned to a specific topic, and each topic can be expressed as a probability distribution of words. Therefore, according to the LDA modeling method, the probability of each topic of a document is determined according to the existence ratio of words in the document. In order to apply LDA topic modeling, the number of topics must be determined in advance, and in the present disclosure, the number of about 20 topics may be predetermined according to the classification of legal documents. The topic may express a type of contract. The computing device 100 may utilize the above-described TF-IDF matrix as an input to generate the LDA matrix. The computing device 100 may obtain a probability distribution vector for one or more topics of each answer by generating an LDA matrix for the answer set. In this disclosure, "probability distribution by topic for each answer" and "LDA matrix" may be used interchangeably.

The computing device 100 may compare a corresponding answer to a query included in an input contract acquired from a question-answering model learned based on the text token-based similarity analysis matrix as described above with a reference answer to the query. The text token-based similarity analysis matrix may include the above-described TF-IDF matrix or LDA matrix. The comparison between the corresponding answer and the standard answer may be a comparison between a text token vector for the corresponding answer and a text token vector for the standard answer existing in the matrix. In one embodiment, computing device 100 may compute a cosine similarity between a text token vector for a corresponding answer and a text token vector for a reference answer. If the cosine similarity calculated between the corresponding answer text token vector for the first query included in the mandatory clause checklist and the standard answer text token vector for the first query is equal to or greater than a predetermined threshold value, the computing device 100 finally After it is determined that both answers to the first question are similar within the margin of error, a risk analysis result including content that essential provisions related to the first question are not omitted may be generated. In addition, if the cosine similarity calculated between the corresponding answer text token vector for the second query included in the mandatory clause checklist and the standard answer text token vector for the second query is less than a predetermined threshold value, the computing device 100 finally It is possible to create a risk analysis result including the content that the content of the essential clause related to the second query is omitted from the input contract. The predetermined threshold may be set differently according to the type of contract.

A risk analysis result according to the present disclosure may be generated based on an output value obtained as a corresponding answer and a standard answer are input to a risk judgment model independently learned from a question answering model. Some of the contents of the structure and learning method of the artificial neural network-based risk judgment model according to the present disclosure are partially overlapped with the contents of the classification model described above with reference to FIG. 2, so the differences will be mainly described below. The risk judgment model may be trained to receive two sentences and output a probability value related to similarity between the two sentences. For example, a risk judgment model may receive “sentence 1_sentence 2” as input and return a value of 1 if sentence 1 and sentence 2 are completely semantically identical, and a value of 0 if they are not similar at all. In addition, the risk judgment model may return a value between 0 and 1 depending on the degree of similarity between the two sentences. The risk judgment model may calculate similarity based on morphological similarity of words used in sentences. The risk judgment model may generate an embedding vector for a sentence considering the context of the sentence, and calculate a similarity between the two sentences based on the generated embedding vector. The risk judgment model according to the present disclosure may be pre-learned to understand the linguistic structure of an input sentence similarly to the question answering model according to the present disclosure, and since the corresponding content is redundant with the above description, it will be omitted below.

The risk analysis result according to the present disclosure may be generated based on a result of applying weights to output values of the learned risk judgment model for the text token-based similarity analysis result for the corresponding answer and the standard answer and the corresponding answer and the standard answer, respectively. In an embodiment according to the present disclosure, the weight may be set as a weight for checking whether a result value equal to or greater than a predetermined value is displayed for each of the text token-based similarity analysis result and the output value of the learned risk determination model. For example, the computing device 100 sets a first threshold for the text token-based similarity analysis result, multiplies by 1 only when the text token-based similarity analysis result exceeds the first threshold, and multiplies by 0 otherwise. can In addition, the computing device 100 may set a second threshold value for the output value of the learned risk determination model, and then multiply by 1 when the output value of the learned risk determination model exceeds the second threshold value and multiply by 0 otherwise. there is. The computing device 100 may generate a risk analysis result by summing the text token-based similarity analysis result and the output value of the learned risk determination model based on the above weight. In this case, the computing device 100 may generate a result that the risk exists when it is determined that the risk exists by one of the text token-based similarity analysis result or the learned risk judgment model, thereby analyzing the risk for the input contract. can be done more conservatively.

In another embodiment according to the present disclosure, the weight may be set as a weight for determining a relative weight between a text token-based similarity analysis result and an output value of a learned risk determination model. The first weight for the text token-based similarity and the second weight for the learned risk judgment model may be relatively determined in consideration of analysis accuracy and the like. For example, the second weight may be set in proportion to the F1 score of the learned risk determination model. As the accuracy of the learned risk judgment model increases and the F1 score of the calculated model increases, the second weight may be set higher than the first weight. According to the present disclosure, the user properly adjusts the relative ratio of the first weight and the second weight to efficiently combine the answer similarity based on the text token vector and the answer similarity based on the model's calculation result to perform the final risk analysis. . If the accuracy of the risk judgment model is low, the text token-based similarity analysis result may have higher accuracy in the risk analysis of the answer. In this case, an accurate risk analysis result can be created by placing a high weight on the text token-based similarity analysis result. In addition, when the learning continuously progresses and the accuracy of the risk determination model is improved, the computing device 100 may generate an accurate risk analysis result by placing a second weight on the output value of the risk determination model higher than the first weight.

The contract risk analysis method according to the present disclosure may further include generating contract summary information based on corresponding answers obtained from the learned question-and-answer model. The contract summary information may include an answer to at least one question included in a checklist corresponding to the input contract. Since the computing device 100 according to the present disclosure determines the type of input contract and obtains corresponding answers based on the corresponding checklist, corresponding answers to questions included in the checklist can be obtained. Accordingly, the computing device 100 may generate a risk analysis result by comparing a corresponding answer to a query of an input contract with a standard answer, and may generate contract summary information including a corresponding answer to the contract. The step of generating the contract summary information by the computing device 100 may be performed separately from the step of generating the risk analysis result by comparing the corresponding answer with the standard answer. When the computing device 100 of the present disclosure generates contract summary information based on the corresponding answers, the user can determine what information is included in the contract for clauses that must be included in addition to the risk analysis performed based on the mandatory clause checklist. You can check summary information. In this way, when the computing device 100 generates contract summary information separately from risk analysis, there is an advantage in that it can remove noise from the contract and provide information on items that are necessarily required according to the contract type. .

The contract risk analysis method according to the present disclosure may further include generating recommendation information for the contract based on the risk analysis result. The recommendation information may include information related to missing essential provisions and information related to included poison provisions. The computing device 100 may generate recommendation information including contents of a corrected contract when essential clauses are omitted in the input contract, or recommendation information recommending excluding poisonous clauses in the input contract. Recommendation information for a contract according to the present disclosure may be generated based on a standard answer to each query, which is a basis for generating a risk analysis result. As an example, in relation to the 'commission' included in the subcontract contract, the standard answer of the computing device 100 is “{number} won (including VAT)”, but the corresponding answer obtained from the Q&A model is “{number} won” If the content of “(including VAT)” is omitted, the computing device 100 may generate recommendation information including the clause “(including VAT)” based on the standard answer as described above. . In addition, when it is determined that a specific sentence included in the input contract is a poison clause based on the poison clause checklist, the computing device 100 may display the sentence and generate recommendation information recommending deletion or modification of the sentence.

3 is a flowchart of a risk analysis method according to an embodiment of the present disclosure. The computing device 100 according to the present disclosure may determine the type of input contract (S310). The type of the input contract may be determined by a trained artificial neural network-based classification model. The classification model may receive one or more text tokens included in the input contract sequentially or in parallel and determine the type based on this. The classification model may determine the type of the contract by using a plurality of text tokens included in the input contract as inputs, and may determine the type of the input contract additionally based on a weight based on the position of the text token in the contract. The computing device 100 may determine a corresponding checklist according to the type of the input contract. The checklist is a checklist of essential provisions for checking whether elements that must be included according to the type of contract are included in the input contract or whether exclusive elements according to the type of contract are included in the input contract. A toxin clause checklist may be included. The computing device 100 may acquire an answer corresponding to at least one question included in the checklist based on the checklist corresponding to the input contract and the determined type through the learned question-answering model (S330). there is. The question-answering model may include a node to which at least one updatable parameter is assigned, and may be a model learned separately from the classification model. The question-answering model may be a model based on an artificial neural network trained to extract an answer to a corresponding question from the sentence when a sentence and a question are entered. The question-answering model may be trained in advance to understand the language structure in documents related to legal terminology. The computing device 100 may generate a risk analysis result by comparing the corresponding answer with a reference answer to the query (S350). The risk analysis result may include an analysis result on omission of essential provisions or an analysis result on whether poison provisions are included. The risk analysis result may be performed based on a text token-based similarity analysis result. The text token-based similarity analysis result may be based on a TF-IDF matrix or an LDA matrix. The risk analysis result may be generated based on an output value obtained as the corresponding answer and the reference answer are input to a risk determination model independently learned from the question answering model. In addition, the risk analysis result may be generated based on an ensemble result of a text token-based similarity analysis result and an output value of a learned risk judgment model. The contract risk analysis method according to the present disclosure including the above flow manages contents that must be confirmed in an input contract through a checklist, and extracts responses to questions included in the checklist through a learned question-and-answer model. And, since risk is analyzed by comparing this with the standard answer, it has the effect of efficiently identifying whether essential provisions are omitted and poisonous provisions are included.

5 is a simplified and general schematic diagram of an exemplary computing environment in which embodiments of the present disclosure may be implemented. Although the present disclosure has been described above as being generally embodied by a computing device, those skilled in the art will understand that the present disclosure may be combined with computer-executable instructions and/or other program modules that may be executed on one or more computers and/or may be implemented in hardware and software. It will be appreciated that it can be implemented as a combination.

Generally, program modules include routines, programs, components, data structures, etc. that perform particular tasks or implement particular abstract data types. It will also be appreciated by those skilled in the art that the methods of the present disclosure may be used in single-processor or multiprocessor computer systems, minicomputers, mainframe computers, as well as personal computers, handheld computing devices, microprocessor-based or programmable consumer electronics, and the like (each of which is It will be appreciated that other computer system configurations may be implemented, including those that may be operative in connection with one or more associated devices.

The described embodiments of the present disclosure may also be practiced in distributed computing environments where certain tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

Computers typically include a variety of computer readable media. Computer readable media can be any medium that can be accessed by a computer, including volatile and nonvolatile media, transitory and non-transitory media, removable and non-transitory media. Includes removable media. By way of example, and not limitation, computer readable media may include computer readable storage media and computer readable transmission media. Computer readable storage media are volatile and nonvolatile media, transitory and non-transitory, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. includes media Computer readable storage media may include RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital video disk (DVD) or other optical disk storage device, magnetic cassette, magnetic tape, magnetic disk storage device or other magnetic storage device. device, or any other medium that can be accessed by a computer and used to store desired information.

A computer readable transmission medium typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism. Including all information delivery media. The term modulated data signal means a signal that has one or more of its characteristics set or changed so as to encode information within the signal. By way of example, and not limitation, computer readable transmission media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared, and other wireless media. Combinations of any of the above are also intended to be included within the scope of computer readable transmission media.

An exemplary environment 1100 implementing various aspects of the present disclosure is shown including a computer 1102, which includes a processing unit 1104, a system memory 1106, and a system bus 1108. do. System bus 1108 couples system components, including but not limited to system memory 1106 , to processing unit 1104 . Processing unit 1104 may be any of a variety of commercially available processors. Dual processor and other multiprocessor architectures may also be used as the processing unit 1104.

System bus 1108 may be any of several types of bus structures that may additionally be interconnected to a memory bus, a peripheral bus, and a local bus using any of a variety of commercial bus architectures. System memory 1106 includes read only memory (ROM) 1110 and random access memory (RAM) 1112 . A basic input/output system (BIOS) is stored in non-volatile memory 1110, such as ROM, EPROM, or EEPROM, and is a basic set of information that helps transfer information between components within computer 1102, such as during startup. contains routines. RAM 1112 may also include high-speed RAM, such as static RAM, for caching data.

The computer 1102 may also include an internal hard disk drive (HDD) 1114 (eg, EIDE, SATA) - the internal hard disk drive 1114 may also be configured for external use within a suitable chassis (not shown). Yes—a magnetic floppy disk drive (FDD) 1116 (e.g., for reading from or writing to a removable diskette 1118), and an optical disk drive 1120 (e.g., a CD-ROM) for reading disc 1122 or reading from or writing to other high capacity optical media such as DVDs). The hard disk drive 1114, magnetic disk drive 1116, and optical disk drive 1120 are connected to the system bus 1108 by a hard disk drive interface 1124, magnetic disk drive interface 1126, and optical drive interface 1128, respectively. ) can be connected to The interface 1124 for external drive implementation includes at least one or both of USB (Universal Serial Bus) and IEEE 1394 interface technologies.

These drives and their associated computer readable media provide non-volatile storage of data, data structures, computer executable instructions, and the like. In the case of computer 1102, drives and media correspond to storing any data in a suitable digital format. Although the description of computer readable media above refers to HDDs, removable magnetic disks, and removable optical media such as CDs or DVDs, those skilled in the art can use zip drives, magnetic cassettes, flash memory cards, cartridges, etc. It will be appreciated that other tangible media readable by the computer, such as the like, may also be used in the exemplary operating environment and any such media may include computer executable instructions for performing the methods of the present disclosure.

A number of program modules may be stored on the drive and RAM 1112, including an operating system 1130, one or more application programs 1132, other program modules 1134, and program data 1136. All or portions of the operating system, applications, modules and/or data may also be cached in RAM 1112. It will be appreciated that the present disclosure may be implemented in a variety of commercially available operating systems or combinations of operating systems.

A user may enter commands and information into the computer 1102 through one or more wired/wireless input devices, such as a keyboard 1138 and a pointing device such as a mouse 1140. Other input devices (not shown) may include a microphone, IR remote control, joystick, game pad, stylus pen, touch screen, and the like. Although these and other input devices are often connected to the processing unit 1104 through an input device interface 1142 that is connected to the system bus 1108, a parallel port, IEEE 1394 serial port, game port, USB port, IR interface, may be connected by other interfaces such as the like.

A monitor 1144 or other type of display device is also connected to the system bus 1108 through an interface such as a video adapter 1146. In addition to the monitor 1144, computers typically include other peripheral output devices (not shown) such as speakers, printers, and the like.

Computer 1102 may operate in a networked environment using logical connections to one or more remote computers, such as remote computer(s) 1148 via wired and/or wireless communications. Remote computer(s) 1148 may be a workstation, computing device computer, router, personal computer, handheld computer, microprocessor-based entertainment device, peer device, or other common network node, and generally includes It includes many or all of the components described for, but for simplicity, only memory storage device 1150 is shown. The logical connections shown include wired/wireless connections to a local area network (LAN) 1152 and/or a larger network, such as a wide area network (WAN) 1154 . Such LAN and WAN networking environments are common in offices and corporations and facilitate enterprise-wide computer networks, such as intranets, all of which can be connected to worldwide computer networks, such as the Internet.

When used in a LAN networking environment, computer 1102 connects to local network 1152 through wired and/or wireless communication network interfaces or adapters 1156. Adapter 1156 may facilitate wired or wireless communications to LAN 1152, which also includes a wireless access point installed therein to communicate with wireless adapter 1156. When used in a WAN networking environment, computer 1102 may include a modem 1158, be connected to a communicating computing device on WAN 1154, or establish communications over WAN 1154, such as over the Internet. have other means. A modem 1158, which may be internal or external and a wired or wireless device, is connected to the system bus 1108 through a serial port interface 1142. In a networked environment, program modules described for computer 1102, or portions thereof, may be stored on remote memory/storage device 1150. It will be appreciated that the network connections shown are exemplary and other means of establishing a communication link between computers may be used.

Computer 1102 is any wireless device or entity that is deployed and operating in wireless communication, eg, printers, scanners, desktop and/or portable computers, portable data assistants (PDAs), communication satellites, wireless detectable tags associated with It operates to communicate with arbitrary equipment or places and telephones. This includes at least Wi-Fi and Bluetooth wireless technologies. Thus, the communication may be a predefined structure as in conventional networks or simply an ad hoc communication between at least two devices.

Wi-Fi (Wireless Fidelity) makes it possible to connect to the Internet without wires. Wi-Fi is a wireless technology, such as a cell phone, that allows such devices, eg, computers, to transmit and receive data both indoors and outdoors, i.e. anywhere within coverage of a base station. Wi-Fi networks use a radio technology called IEEE 802.11 (a, b, g, etc.) to provide secure, reliable, and high-speed wireless connections. Wi-Fi can be used to connect computers to each other, to the Internet, and to wired networks (using IEEE 802.3 or Ethernet). Wi-Fi networks can operate in the unlicensed 2.4 and 5 GHz radio bands, for example, at 11 Mbps (802.11a) or 54 Mbps (802.11b) data rates, or in products that include both bands (dual band) .

Those skilled in the art will understand that information and signals may be represented using any of a variety of different technologies and techniques. For example, data, instructions, instructions, information, signals, bits, symbols and chips that may be referenced in the above description are voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields s or particles, or any combination thereof.

Those skilled in the art will understand that the various illustrative logical blocks, modules, processors, means, circuits, and algorithm steps described in connection with the embodiments disclosed herein are electronic hardware, (for convenience) , may be implemented by various forms of program or design code (referred to herein as software) or a combination of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and the design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

Various embodiments presented herein may be implemented as a method, apparatus, or article of manufacture using standard programming and/or engineering techniques. The term article of manufacture includes a computer program, carrier, or media accessible from any computer-readable storage device. For example, computer-readable storage media include magnetic storage devices (eg, hard disks, floppy disks, magnetic strips, etc.), optical disks (eg, CDs, DVDs, etc.), smart cards, and flash memory devices (eg, EEPROM, cards, sticks, key drives, etc.), but are not limited thereto. Additionally, various storage media presented herein include one or more devices and/or other machine-readable media for storing information.

It is to be understood that the specific order or hierarchy of steps in the processes presented is an example of exemplary approaches. Based upon design priorities, it is to be understood that the specific order or hierarchy of steps in the processes may be rearranged within the scope of this disclosure. The accompanying method claims present elements of the various steps in a sample order, but are not meant to be limited to the specific order or hierarchy presented.

The description of the presented embodiments is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art of this disclosure, and the general principles defined herein may be applied to other embodiments without departing from the scope of this disclosure. Thus, the present disclosure is not to be limited to the embodiments presented herein, but is to be interpreted in the widest scope consistent with the principles and novel features presented herein.

Claims (19)

A contract risk analysis method performed by a computing device including at least one processor,
determining the type of input contract;
obtaining a corresponding answer to at least one question included in the checklist based on a checklist corresponding to the input contract and the determined type through a learned question-and-answer model; and
generating a risk analysis result by comparing the corresponding answer with a reference answer to the query;
including,
The risk analysis result is,
text token-based similarity analysis results for the corresponding answer and the reference answer; and
an output value of a risk judgment model learned with respect to the corresponding answer and the reference answer;
It is generated based on the result of applying a weight to each,
The weight is calculated based on the accuracy of the risk judgment model,
How to analyze contract risk.
According to claim 1,
The type of the input contract is,
Determined based on a classification model learned independently of the question response model,
How to analyze contract risk.
According to claim 2,
The classification model,
an upper classification model for matching an input contract with one of a plurality of higher types; and
a lower classification model for matching the input contract with one of a plurality of lower types included in the matched upper type;
including,
How to analyze contract risk.
According to claim 3,
The upper classification model uses the input contract as an input and generates one upper type corresponding to the input contract as output information;
A plurality of lower classification models exist to correspond to each of a plurality of higher types that can be output from the upper classification model, and
Among the plurality of lower classification models, a lower classification model for matching the input contract to one sub type is determined based on output information of the upper classification model.
How to analyze contract risk.
According to claim 2,
The type of the input contract is,
a classification result obtained by inputting a plurality of text tokens included in an input contract into the learned classification model; and
a weight based on the position of the plurality of text tokens;
determined based on
How to analyze contract risk.
According to claim 5,
The weight based on the location is:
a first weight assigned to a first text token existing in a leading position in the input contract among the plurality of text tokens; and
a second weight assigned to a second text token existing in a trailing position in the input contract among the plurality of text tokens;
includes, and
The first weight is set to have a higher value than the second weight,
How to analyze contract risk.
According to claim 5,
The input contract is divided into a preceding location area and a subsequent location area,
a first weight assigned to a first text token in the preceding location area has a higher value than a second weight assigned to a second text token in the following location area; and
At least a portion of the preceding location area and at least a portion of the following location area overlap.
How to analyze contract risk.
According to claim 1,
The above checklist is:
a checklist of essential provisions for checking whether elements that must be included according to the type of contract are included in the input contract; or
a poison clause checklist for checking whether or not poisonous elements according to the type of contract are included in the input contract;
including,
How to analyze contract risk.
According to claim 8,
The above essential checklist is:
Including a first query generated based on a legal provision corresponding to the type of contract or a second query generated based on a precedent,
How to analyze contract risk.
According to claim 1,
The question answering model,
text containing the terms of the agreement;
one or more inquiries about the terms of the agreement; and
criteria answers labeled within the text to correspond to the one or more queries;
Learning based on at least one learning data including
How to analyze contract risk.
According to claim 1,
The question answering model,
inputting text including contract details and at least one query about the contract contents into the question answering model;
obtaining a corresponding answer to the question in the text from the question answering model; and
updating at least one parameter included in the question-answering model by comparing the corresponding answer with a reference answer to the question;
learned by
How to analyze contract risk.
According to claim 10,
One or more inquiries about the contents of the above contract,
a replacement query in which at least some of the text tokens included in the other query are replaced with other values;
including,
How to analyze contract risk.
According to claim 1,
The risk analysis result is,
Including the analysis result of whether essential provisions are omitted or whether poison provisions are included,
How to analyze contract risk.
According to claim 1,
The risk analysis result is,
Generated based on the text token-based similarity analysis result between the corresponding answer and the reference answer,
How to analyze contract risk.
According to claim 1,
The risk analysis result is,
Generated based on output values obtained as the corresponding answer and the reference answer are input to the risk judgment model learned independently of the question response model,
How to analyze contract risk.
delete According to claim 1,
generating contract summary information based on the corresponding answers obtained from the learned question-and-answer model; or
generating recommendation information for a contract based on a result of the risk analysis;
Including more,
How to analyze contract risk.
A computer program stored on a computer readable storage medium, which, when executed on one or more processors, causes the following operations for analyzing contract risk, the operations:
operation to determine the type of input contract;
obtaining an answer corresponding to at least one question included in the checklist based on the checklist corresponding to the input contract and the determined type through a learned question-and-answer model; and
generating a risk analysis result by comparing the corresponding answer with a reference answer to the query;
including,
The risk analysis result is,
text token-based similarity analysis results for the corresponding answer and the reference answer; and
an output value of a risk judgment model learned with respect to the corresponding answer and the reference answer;
It is generated based on the result of applying a weight to each,
The weight is calculated based on the accuracy of the risk judgment model,
A computer program stored on a computer-readable storage medium.
As a contract risk analysis device,
one or more processors;
Memory; and
network unit;
contains, and
The one or more processors,
determine the type of input contract;
Obtaining a corresponding answer to at least one question included in the checklist based on the checklist corresponding to the input contract and the determined type through a learned question-answering model, and
A risk analysis result is generated by comparing the response answer with the standard answer to the query,
The risk analysis result is,
text token-based similarity analysis results for the corresponding answer and the reference answer; and
an output value of a risk judgment model learned with respect to the corresponding answer and the reference answer;
It is generated based on the result of applying a weight to each,
The weight is calculated based on the accuracy of the risk judgment model,
Contract risk analysis device.

Images