KR102205430B1 - Learning method using artificial neural network - Google Patents

Abstract

According to one embodiment of the present invention, a learning method using an artificial neural network circuit may comprise the steps of: receiving a document including a plurality of natural languages and converting the plurality of natural languages included in the received document into tokens; generating a plurality of partial tokens by dividing the tokens; generating embedding vectors by converting the plurality of partial tokens into a vector; generating an embedding vector inferred by performing RNN encoding and RNN decoding on the embedding vectors; generating an inverted embedding vector by inverting an order of the embedding vectors; calculating a difference between the inferred embedding vector and the inverted embedding vector; generating a backpropagation value for reducing the difference; and performing the RNN encoding and the RNN decoding again using the back propagation value. Therefore, the usability is improved.

Description

Learning method using artificial neural network {LEARNING METHOD USING ARTIFICIAL NEURAL NETWORK}

The present invention relates to a learning method using an artificial neural network and a method for recommending similar documents using the method.

With recent advances in technology, interest in natural language processing (NLP) using machine learning is increasing. Natural language processing is a technology that enables computers to recognize and process the language we use, and is used in automatic translation and search engines.

Representative algorithms for natural language processing include BERT (Bidirectional Encoder Representations from Transformers) and GPT (Generative Pre-Training model).BERT and GPT are natural language processing algorithms that reflect context in units of sentences. It shows the best performance when dealing with composed sentences.

Conventional algorithms such as BERT use a transformer algorithm to grasp the context. Conventionally, a transformer algorithm was tuned and used to determine the context of a document unit, but increasing the number of heads to handle the context of a document unit increases memory complexity, and increasing the number of vocabularies increases versatility. There is a problem with this dropping.

The problem to be solved by the present invention is to provide a method of generating a vector reflecting the context of a document unit in order to perform learning using an artificial neural network.

However, the problem to be solved by the present invention is not limited to those mentioned above, and another problem to be solved that is not mentioned may be clearly understood by those of ordinary skill in the art to which the present invention belongs from the following description. will be.

A learning method using an artificial neural network circuit according to an embodiment of the present invention includes: receiving a document including a plurality of natural languages and converting a plurality of natural languages included in the received document into tokens; Dividing the token to generate a plurality of partial tokens; Converting the plurality of partial tokens into vectors to generate an embedding vector; Generating an inferred embedding vector by performing RNN (recurrent neural network) encoding and RNN decoding on the embedding vector; Generating an inverted embedding vector by reversing the order of the embedding vectors; Calculating a difference between the inferred embedding vector and the inverted embedding vector; Generating a backpropagation value for reducing the difference; And re-performing the RNN encoding and the RNN decoding using the backpropagation value.

The method may further include adjusting weights of the RNN encoding and the RNN decoding according to the backpropagation value.

The generating of the embedding vector may include: generating a plurality of embedding vector lists by performing a transformer algorithm on each of the plurality of partial tokens; And generating the embedding vector by combining the plurality of embedding vector lists.

Depending on the length of the partial token and the stride to generate the partial token, some of the plurality of partial tokens overlap, and the embedding vector is the embedding vector corresponding to the overlapping portion of the plurality of partial tokens It can be generated by performing an average value operation on vectors in lists.

In the step of calculating the difference between the inferred embedding vector and the inverted embedding vector, the difference may be calculated using any one of the Smooth-L1 algorithm, the L1 algorithm, the L2 algorithm, and the Huber Loss algorithm.

According to an embodiment of the present invention, it is possible to grasp a context in a document unit (about 50,000 tokens).

In addition, in the present invention, since the model learned through the BERT algorithm is used as it is, it has high versatility, and since a tricky BERT fine-tuning procedure is performed instead in the RNN learning process, the usability is improved.

1 is a block diagram of a DRAE that performs a vector generation algorithm reflecting the context of a document unit to perform learning using an artificial neural network.
2 is a detailed block diagram of an embedding vector generator shown in FIG. 1.
3 shows a method of generating a plurality of partial tokens in the token divider illustrated in FIG. 2.
4 shows a method of generating an embedding vector in the embedding vector association unit shown in FIG. 2.
5A and 5B are flowcharts illustrating a method of generating a vector reflecting the context of a document unit in order to perform learning using an artificial neural network according to an embodiment of the present invention.
6 is a flowchart illustrating a method of recommending a similar document according to an embodiment of the present invention.
7 is a flowchart illustrating a method of recommending a similar document through a natural language query according to an embodiment of the present invention.
8 shows an apparatus for performing a method according to an embodiment of the present invention.

Advantages and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in a variety of different forms, only the present embodiments are intended to complete the disclosure of the present invention, and the general knowledge in the art It is provided to completely inform the scope of the invention to those who have it, and the invention is only defined by the scope of the claims.

In describing embodiments of the present invention, if it is determined that a detailed description of a known function or configuration may unnecessarily obscure the subject matter of the present invention, a detailed description thereof will be omitted. In addition, terms to be described later are terms defined in consideration of functions in an embodiment of the present invention, which may vary according to the intention or custom of users or operators. Therefore, the definition should be made based on the contents throughout this specification.

1 is a block diagram of a DRAE that performs a vector generation algorithm reflecting the context of a document unit to perform learning using an artificial neural network.

Referring to FIG. 1, a document recurrent autoencoder (DRAE) 10 includes a first embedding vector generator 100, a second embedding vector generator 110, an RNN encoder-decoder (Recurrent Neural Network, RNN). , Encoder 120, a reverser 150, and a cost function calculation unit 160 may be included.

The first embedding vector generation unit 100 receives a document including a plurality of natural languages, converts a plurality of natural languages included in the received document into tokens, and embeds an embedding vector (EV) for the converted token. Can be created. The first embedding vector generator 100 may transmit the generated embedding vector EV to the RNN encoder-decoder 120.

The second embedding vector generator 110 receives a document including the plurality of natural languages, converts a plurality of natural languages included in the received document into tokens, and generates an embedding vector (EV) for the converted token. I can. The second embedding vector generator 110 may transmit the generated embedding vector EV to the inverting unit 150.

In this specification, the DRAE 10 is shown to include two embedding vector generators 100 and 110, but is not limited thereto. For example, according to an embodiment, the DRAE 10 may include one embedding vector generator. When the DRAE 10 includes a single embedding vector generator, the embedding vector generator converts a plurality of natural languages included in the input document into tokens, generates an embedding vector for the converted token, and generates the embedding vector. It can be transmitted to the RNN encoder-decoder 120 and the inverting unit 150. A description of the embedding vector generation units 100 and 110 will be described in more detail in FIG. 2.

The RNN encoder-decoder 120 may include an RNN encoder 130 and an RNN decoder 140. The RNN encoder-decoder 120 may receive an embedding vector (EV) from the first embedding vector generator 100. In addition, the RNN encoder-decoder 120 receives an Encoder Initial Vector (EIV) as a vector value for initializing the RNN encoder 130, and initializes the decoder with a vector value for initializing the RNN decoder 140. A vector (Decoder Initial Vector) (DIV) may be received, and a backpropagation (BP) value may be received from the cost function calculation unit 160.

The encoder initialization vector EIV may mean a value for initializing the RNN encoder 130. The encoder initialization vector EIV may be set to a vector representing 0, but is not limited thereto. That is, according to an embodiment, the encoder initialization vector EIV may be set to a vector representing a value other than 0. The decoder initialization vector (DIV) may mean a value for initializing the RNN decoder 140. The decoder initialization vector DIV may be set to a vector representing 0, but is not limited thereto. That is, according to an embodiment, the decoder initialization vector DIV may be set to a vector representing a value other than 0.

The RNN encoder-decoder 120 generates a context vector (CV) by encoding the received embedding vector, and decodes the generated context vector (CV) to inferred embedding vector (IEV). (Or inferred Embedding Vector Logits) can be created. The RNN encoder-decoder 120 may transmit the inferred embedding vector (IEV) to the cost function calculation unit 160.

The RNN encoder 130 may encode the embedding vector (EV) using the encoder initialization vector (EIV) and generate a context vector (CV) as a result of the encoding. In addition, the RNN encoder 130 may perform a gradient descent algorithm with the RNN decoder 140 using the backpropagation value BP received from the cost function calculator 160. As the RNN encoder 130 performs one or more gradient descent algorithms, an error, which is the difference between the inferred embedding vector (IEV) and the inverted embedding vector (REV), is recognized as 0 (or 0). Possible value). The RNN encoder 130 may output a context vector of a document when an error, which is a difference between the embedding vector (IEV) and the inverted embedding vector (REV), is 0, and the memory (3 in FIG. 8) is It is possible to store the context vector of the document.

The RNN decoder 140 may decode the context vector (CV) using the backpropagation value (BP) and the decoder initialization vector (DIV), and generate an embedding vector (IEV) inferred from the decoding result. Here, the inferred embedding vector (IEV) may mean a vector predicted as the first input embedding vector (EV).

In addition, the RNN decoder 140 may generate an inferred embedding vector (IEV) using a transformer algorithm. There was a problem that the inverse function does not exist in the embedding vector generated using the conventional transformer algorithm.In the present invention, the RNN decoder 140 acquires the embedding vector of the token (TK) using the transformer algorithm, thereby estimating the token. You can solve the problem.

The RNN decoder 140 may transmit the inferred embedding vector (IEV) to the cost function calculator 160. The RNN decoder 140 may receive a backpropagation value BP from the cost function calculation unit 160 and may perform a gradient descent algorithm using the received backpropagation value BP.

The inverting unit 150 receives the embedding vector (EV) from the second embedding vector generation unit 110, and generates an inverted embedding vector (REV) by reversing the order of the numbers included in the received embedding vector (EV). can do. The inverting unit 150 may transmit the inverted embedding vector REV to the cost function calculating unit 160.

The conventional RNN encoder-decoder has poor learning performance for long sentences. This is because the length from the first point to the last point of the sentence is long, making it difficult to maintain the context within the sentence. Therefore, it is necessary to perform order reversal learning in which the estimated value of the first token of the document is composed of the last token of the document. If the document is reversed in units of tokens, it may not be possible to generate an inverted embedding vector correctly. The reason is that if the document is reversed in units of tokens, the context of the sentence is not maintained and a completely different embedding vector may be generated.

In the present invention, in order to solve this problem, the order of the embedding vector (EV) is reversed by using the inversion unit 150. Accordingly, the DRAE 10 of the present invention can generate an inverted embedding vector in which the context of the sentence is preserved, and solve the problem of the conventional RNN encoder-decoder that the learning performance for a long sentence is poor.

The cost function calculation unit 160 may receive an embedding vector (IEV) inferred from the RNN decoder 140 and may receive an inverted embedding vector (REV) from the inverting unit 150. The cost function calculation unit 160 calculates an error, which is a difference between the inferred embedding vector (IEV) and the inverted embedding vector (REV), and uses the backpropagation value (BP) as a feedback to reduce the error. It can be transmitted to 140.

The cost function calculation unit 160 uses a backpropagation method to reduce a difference between the inferred embedding vector (IEV) and the inverted embedding vector (REV). At this time, since the inferred embedding vector must be estimated, not the token number, the cost function calculation unit 160 cannot use Softmax & Log Likelihood, and thus learning is impossible. Therefore, in order to solve this problem, the cost function calculation unit 160 of the present invention learns by determining the backpropagation value (BP) using the Smooth-L1 algorithm, the L1 algorithm, the L2 algorithm, and/or the Huber Loss algorithm. This impossible conventional problem has been solved.

2 is a detailed block diagram of an embedding vector generator shown in FIG. 1.

Referring to FIG. 2, the embedding vector generation unit 100 or 110 (referred to as 100 for convenience) includes a token generator 200, a token divider 210, and a transformer embedding unit 220. ) And an Embedding Vector Concatenator 230.

The token generator 200 may receive a document including a plurality of natural languages and convert a plurality of natural languages included in the received document into a token TK. The token TK may be composed of a set of numbers, and the numbers in the token TK may be generated in at least one unit among words, syllables, phonemes, or combinations thereof in the document. The token generator 200 may transmit the generated token TK to the token divider 210.

The token divider 210 may generate a plurality of partial tokens PTK1 to PTK(k) including n consecutive numbers from among the numbers included in the token TK. Here, n may mean a natural number less than or equal to the number of numbers included in the token, and may mean an appropriate number of inputs to drive the transformer algorithm. In addition, k may mean a natural number of 2 or more. The token divider 210 may generate a plurality of partial tokens PTK1 to PTK(k) with a stride having a size of m (m is a natural number). In this case, depending on the length of the token TK, the length of the last partial token PTK(k) may be less than n. The token divider 210 may sequentially transmit the generated partial tokens PTK1 to PTK(k) to the transformer embedding unit 220.

According to an embodiment, when the size of m is smaller than n, some of the partial tokens may overlap. For example, if the token TK is {1, 2, 3, 4, 5, 6, 7, 8, 9, 10}, n is 4, and m is 2, the first partial token PTK1 is { 1, 2, 3, 4}, and the second partial token (PTK2) becomes {3, 4, 5, 6}, and thus, 3 and 4 are the first partial token PTK1 and the second partial token PTK2 ).

A method of generating a plurality of partial tokens will be described in detail with reference to FIG. 3.

The transformer embedding unit 220 may generate a plurality of embedding vector lists EVL1 to EVL(k) by performing a transformer algorithm on each of the plurality of partial torques PTK1 to PTK(k). The transformer embedding unit 220 generates a first embedding vector list EVL1 by performing a transformer algorithm on the first partial torque PTK1, and performs a transformer algorithm on the second partial torque PTK2 to perform a second embedding vector list. A third embedding vector list EVL3 may be generated by generating EVL2 and performing a transformer algorithm on the third partial torque PTK3. The transformer embedding unit 220 may transmit the generated embedding vector lists EVL1 to EVL(k) to the embedding vector association unit 230.

The embedding vector association unit 230 may generate an embedding vector EV by combining a plurality of embedding vector lists EVL1 to EVL(k). The embedding vector association unit 230 may transmit the generated embedding vector (EV) to the RNN encoder-decoder 120.

The embedding vector association unit 230 sorts the plurality of embedding vector lists EVL1 to EVL(k) in the order of index, and selects the same index among the plurality of embedding vector lists EVL1 to EVL(k). It is possible to create an embedding vector (EV) by combining the owned vectors. An embedding vector may be generated by performing an average value operation on vectors having the same index among a plurality of embedding vector lists EVL1 to EVL(k). For example, according to an embodiment, when the size of m is smaller than n, some of the embedding vector lists EVL1 to EVL(k) may be overlapped, and by performing an average value operation on the overlapping vectors, the embedding vector ( EV) can be generated. A method of generating the embedding vector (EV) will be described in more detail with reference to FIG. 4.

Conventional natural language processing algorithms have a problem in that they cannot extract an embedding vector reflecting the context of a document unit. In this specification, the token divider 210 generates partial tokens while preserving the context in the sentence, and the embedding vector association unit 230 combines the embedding vector lists so that the context of the generated token (partial tokens) is maintained. , It is possible to extract the embedding vector reflecting the context of the entire document.

3 shows a method of generating a partial token in the token divider shown in FIG. 2.

Referring to FIG. 3, the token divider may generate k (k is a natural number) partial tokens PTK1 to PTK(k) from a token consisting of L (L is a natural number) number. Each of the partial tokens PTK1 to PTK(k) may be composed of n numbers (n is a natural number less than L). The token divider can generate partial tokens with a stride of m (m is a natural number less than L).

The token divider may generate a first partial token PTK1 including an n-th number from the first number of the token. The token divider may generate the second partial token PTK2 with a stride of m from the first partial token PTK1. That is, the token divider may generate the second partial torque PTK2 including the (n+m)-th digit from the (1+m)-th digit of the token. In addition, the token divider may generate a third partial token PTK3 with a stride of m from the second partial token PTK2. That is, the token divider may generate the third partial torque including the (1+2m)-th digit to the (n+2m)-th digit. The token divider 210 may transmit the generated partial torques PTK1 to PTK(k) to the transformer embedding unit 220.

2 shows that the length of the k-th partial token (PTK(k)) is the same as the length of the (k-1)-th partial token (PTK(k-1)), but the length of the token TK (L ), the length of the k-th partial token (PTK(k)) and the length of the (k-1)-th partial token (PTK(k-1)) may be different.

4 shows a method of generating an embedding vector in the embedding vector association unit shown in FIG. 2.

Referring to FIG. 4A, when m and n are the same, embedding vector lists may not overlap. For example, when m and n are 5 and L is 10, the first embedding vector list EVL1 corresponds to indices 1 to 5, and the second embedding vector list EVL2 corresponds to indices 6 to 10. May not overlap. Accordingly, the embedding vector association unit may generate the embedding vector EV by sequentially combining the first embedding vector list EVL1 and the second embedding vector list EVL2.

Referring to FIG. 4B, when m is less than n, some of the embedding vector lists may overlap. For example, when m is 1, n is 5, and L is 10, the first embedding vector list (EVL1) corresponds to indices 1 to 5, and the second embedding vector list (EVL2) corresponds to indices 2 to 6 Accordingly, the first embedding vector list EVL1 and the second embedding vector list EVL2 may overlap at index 2. Accordingly, an average value of the value of the first embedding vector list EVL1 at index 2 and the value of the second embedding vector list EVL2 at index 2 may be determined as an embedding vector for index 2.

5A and 5B are flowcharts illustrating a method of generating a vector reflecting the context of a document unit in order to perform learning using an artificial neural network according to an embodiment of the present invention.

5A and 5B, the token generator 200 may receive a document including a plurality of natural languages and convert a plurality of natural languages included in the received document into a token TK (S500). The token divider 210 may generate a plurality of partial tokens PTK1 to PTK(k) including n consecutive numbers from among the numbers included in the token TK (S510). The transformer embedding unit 220 may generate a plurality of embedding vector lists EVL1 to EVL(k) by performing a transformer algorithm on each of the plurality of partial torques PTK1 to PTK(k) (S520). The embedding vector association unit 230 may generate an embedding vector EV by combining the plurality of embedding vector lists EVL1 to EVL(k) (S530). The embedding vector association unit 230 sorts the plurality of embedding vector lists EVL1 to EVL(k) in the order of index, and selects the same index among the plurality of embedding vector lists EVL1 to EVL(k). It is possible to create an embedding vector (EV) by combining the owned vectors.

The RNN encoder 130 may encode the embedding vector (EV) using the encoder initialization vector (EIV) and generate a context vector (CV) as a result of the encoding (S540). The RNN decoder 140 may decode the context vector (CV) using the backpropagation value (BP) and the decoder initialization vector (DIV), and generate an embedding vector (IEV) inferred as a result of the decoding (S550).

The inverting unit 150 receives the embedding vector (EV) from the embedding vector association unit 230 and reverses the order of numbers included in the received embedding vector (EV) to generate an inverted embedding vector (REV). have

The cost function calculation unit 160 may calculate an error, which is a difference between the inferred embedding vector (IEV) and the inverted embedding vector (REV), and determine whether the error is 0 (zero) ( S560).

When the error is not 0 (NO in S560), the cost function calculation unit 160 transmits a backpropagation value BP for making the error 0 to the RNN decoder 140, and the RNN encoder ( 130) and the RNN decoder 140 may perform a backpropagation algorithm using the backpropagation value BP (S570). The RNN encoder 130 and the RNN decoder may repeat steps S540 to S570 until a difference between the embedding vector inferred using the weight changed according to the backpropagation algorithm and the inverted embedding vector becomes 0. . By repeating steps S540 to S570, the DRAE 10 (or RNN encoder-decoder 120) may be learned.

If the error is 0 (zero) (YES in S560), the RNN encoder 130 outputs a context vector (CV) when the error is 0 (S580), and the memory (3 in FIG. 8) is an RNN encoder ( 130), the output context vector (CV) may be stored. The context vector CV may be a result of training the DRAE 10 (or the RNN encoder-decoder 120).

6 is a flowchart illustrating a method of recommending a similar document according to an embodiment of the present invention.

The context vector (CV) of the reference document to be searched for similar documents may be determined by using the DRAE 10 (S600). A method of determining the context vector (CV) of the reference document may be performed by the method described in FIGS. 1 to 5B. Thereafter, the context vector of each of the documents included in the document group to be compared with the reference document and similarity is determined (S610), and a distance between the context vector of the reference document and the context vector of each of the documents included in the document group may be measured. There is (S620).

As a result of the measurement, as the distance between the context vector of the reference document and the context vector of the document included in the document group is closer, it is determined that the similarity is high, and several documents with the closest distance may be output in order of distance (S630).

7 is a flowchart illustrating a method of recommending a similar document through a natural language query according to an embodiment of the present invention.

The context vector (CV) of a query composed of a natural language to be searched may be determined by using the DRAE 10 (S700). Here, the query may be a simple list of words, or may be a document unit including one or more sentences or one or more paragraphs.

Thereafter, the context vector of each of the plurality of documents included in the document group to be compared with the natural language and similarity is determined (S710), and the distance between the context vector of the reference document and each of the documents included in the document group is measured. Can be (S720).

As a result of the measurement, it is determined that the similarity is higher as the distance between the context vector of the reference document and the context vector of the document included in the document group is closer, and several documents with the closest distance may be output in order of distance (S730).

8 shows an apparatus for performing a method according to an embodiment of the present invention.

The device 1 may include a processor 2, a memory 3, and an input/output device 4.

The processor 2 can control overall the operation of the device 1. The processor 2 may control operations of the memory 3 and the input/output device 4.

The memory 3 may store information on the DRAE 10. The memory 3 may store information on the context vector CV output from the RNN encoder 130 under the control of the processor 2. In order to recommend a similar document, the processor 2 may load information about the DRAE 10 and/or information about the context vector (CV) stored in the memory 3.

Combinations of each block of the block diagram attached to the present invention and each step of the flowchart may be performed by computer program instructions. Since these computer program instructions can be mounted on the encoding processor of a general-purpose computer, special purpose computer or other programmable data processing equipment, the instructions executed by the encoding processor of the computer or other programmable data processing equipment are each block of the block diagram or Each step of the flow chart will create a means to perform the functions described. These computer program instructions can also be stored in computer-usable or computer-readable memory that can be directed to a computer or other programmable data processing equipment to implement a function in a particular way, so that the computer-usable or computer-readable memory It is also possible to produce an article of manufacture in which the instructions stored in the block diagram contain instruction means for performing the functions described in each block or flow chart. Computer program instructions can also be mounted on a computer or other programmable data processing equipment, so that a series of operating steps are performed on a computer or other programmable data processing equipment to create a computer-executable process to create a computer or other programmable data processing equipment. It is also possible for the instructions to perform the processing equipment to provide steps for performing the functions described in each block of the block diagram and each step of the flowchart.

In addition, each block or each step may represent a module, segment, or part of code comprising one or more executable instructions for executing the specified logical function(s). It should also be noted that in some alternative embodiments, functions mentioned in blocks or steps may occur out of order. For example, two blocks or steps shown in succession may in fact be performed substantially simultaneously, or the blocks or steps may sometimes be performed in the reverse order depending on the corresponding function.

The above description is merely illustrative of the technical idea of the present invention, and those of ordinary skill in the art to which the present invention pertains will be able to make various modifications and variations without departing from the essential quality of the present invention. Accordingly, the embodiments disclosed in the present invention are not intended to limit the technical idea of the present invention, but to explain the technical idea, and the scope of the technical idea of the present invention is not limited by these embodiments. The scope of protection of the present invention should be interpreted by the following claims, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of the present invention.

10: DRAE
100, 110: embedding vector generator
120: RNN encoder-decoder
150: reverse part
160: cost function calculation unit

Claims (6)

In a method of learning a document recurrent auto encoder including a recurrent neural network (RNN) encoder and an RNN decoder performed by a document recurrent autoencoder learning device,
Receiving a document including a plurality of natural languages and converting a plurality of natural languages included in the received document into tokens;
Dividing the token to generate a plurality of partial tokens;
Converting the plurality of partial tokens into vectors to generate an embedding vector;
Generating an inferred embedding vector by performing RNN encoding and RNN decoding on the embedding vector;
Generating an inverted embedding vector by reversing the order of the embedding vectors;
Calculating a difference between the inferred embedding vector and the inverted embedding vector;
Generating a backpropagation value for reducing the difference; And
And re-performing the RNN encoding and the RNN decoding using the backpropagation value.
How to train the document rotation autoencoder. The method of claim 1,
Adjusting the weight of the RNN encoding and the RNN decoding according to the backpropagation value further comprising
How to train the document rotation autoencoder. The method of claim 1,
Generating the embedding vector,
Generating a plurality of embedding vector lists by performing a transformer algorithm on each of the plurality of partial tokens; And
Comprising the step of combining the plurality of embedding vector lists to generate the embedding vector
How to train the document rotation autoencoder. The method of claim 3,
Depending on the length of the partial token and the stride to generate the partial token, some of the plurality of partial tokens overlap,
The embedding vector is generated by performing an average value operation on vectors in the embedding vector lists corresponding to an overlapping part of the plurality of partial tokens.
How to train the document rotation autoencoder. The method of claim 1,
The step of calculating the difference between the inferred embedding vector and the inverted embedding vector includes calculating the difference using any one of Smooth-L1 algorithm, L1 algorithm, L2 algorithm, and Huber Loss algorithm.
How to train the document rotation autoencoder. As a computer-readable recording medium storing a computer program,
The computer program,
A computer-readable recording medium comprising instructions for causing a processor to perform the method according to any one of claims 1 to 5.

Images