JPWO2021009972A1 - Natural language processing method, natural language processing system, and natural language processing program - Google Patents

Abstract

The issue to be solved is to realize new natural language processing. A reception step that accepts the designation of a web page containing the first text of the original language, a translation step of acquiring the second text of the target language based on the first text, and at least a part of the second text. A natural language that causes a computer processor to perform a collection step of scraping as a query to obtain a third text of the target language and a determination step of determining the third text as a translation of the first text. Realize the processing method and its system and program.

Description

The present invention relates to a natural language processing method, a natural language processing system, and a natural language processing program.

Machine translation, which converts text written in any natural language (original language) into text written in any different natural language (target language), is a "machine translation" that does not include humans as the main body in the translation process. Is.

"Machine translation" generalizes translation processes that are close to specialized skills and reduces translation costs, but it also floods the Internet Web with unreadable translations, which is a factor that makes search engine optimization difficult, for example. It is one.

Therefore, in recent years, machine translation is required not only to improve translation accuracy, but also to realize "translation by humans" that can generate easy-to-read translations that include humans as the main body in the translation process.

According to Patent Document 1, a translation candidate generation unit including a plurality of machine translation devices for generating a translation of the second language for each input sentence of the first language, and a plurality of obtained second translation units. A machine that includes a translation improvement unit that transforms and improves the translation from each language translation, and an end judgment unit that selects the improved translation that satisfies certain conditions as the output sentence for the input sentence. Inventions relating to translation systems have been reported.

Japanese Patent No. 3919771

The translation improvement unit in the invention described in Patent Document 1 includes a translation selection unit for selecting either an initial candidate translation or a translation read from the translation storage unit, and has a low degree of literal translation and is close to a free translation. It can be understood that the sentence can be generated.

However, in order to reproduce the "human translation", it is necessary to refer not only to the recorded or improved information, but also to the latest text / context as a human creation. In this respect, it can be said that the invention described in Patent Document 1 has room for improvement.

The present invention is a problem to be solved to realize a novel natural language processing.

In order to solve the above problems, the present invention is a natural language processing method, which is a reception step for accepting a designation of a web page including the first text of the original language, and a first language of the target language based on the first text. A translation step of acquiring the second text, a collection step of scraping at least a part of the second text as a query to acquire the third text of the target language, and the first text of the third text. It is characterized by having a computer processor execute a decision step to determine as a translation of the text of.

In a preferred embodiment of the present invention, the processor executes an analysis step of acquiring a morpheme string of text, an evaluation step of acquiring a variance vector of the morpheme string, and a determination step of determining the similarity between the variance vectors. The translation step obtains the fourth text of the original language based on the third text, and the analysis step is the morpheme sequence of the first text and the fourth text of the fourth text. The morpheme sequence is acquired, the evaluation step acquires the dispersion vector of the first text, and the dispersion vector of the fourth text, and the determination step is the first determination determined by the determination step. When the similarity between the text and the variance vector of the fourth text exceeds the threshold value, the third text is determined as the translation.

In a preferred embodiment of the invention, the collection step is based on the web page if the similarity of the variance vector of the first text and the variance vector of the fourth text does not exceed the threshold. It is characterized in that the query is extended, the scraping is performed, and the third text is acquired.

In a preferred embodiment of the present invention, the evaluation step acquires the variance vector based on the trained model, and the trained model is a neural network model in which a part of the hidden layer or the output layer shows the variance vector. It is characterized by.

In order to solve the above problems, the present invention is a natural language processing system, which is a reception means for accepting designation of a web page including a first text of the original language, and a first language of a target language based on the first text. The translation means for acquiring the second text, the collecting means for obtaining the third text of the target language by scraping at least a part of the second text as a query, and the first text. It is characterized by having a determination means for determining as a translation of the text of.

In order to solve the above problems, the present invention is a natural language processing program, which is a reception means for receiving a designation of a web page containing a first text of the original language, and an object based on the first text. A translation means for acquiring the second text of the language, a collecting means for obtaining the third text of the target language by scraping at least a part of the second text as a query, and the third text. It is characterized in that it functions as a determination means for determining as a translation of the first text.

According to the present invention, a novel natural language processing can be realized.

A hardware configuration according to an embodiment of the present invention is shown. The block diagram which concerns on one Embodiment of this invention is shown. The block diagram of the database which concerns on one Embodiment of this invention is shown. The flowchart which concerns on one Embodiment of this invention is shown.

The natural language processing system, the natural language processing method, and the natural language processing program according to the embodiment of the present invention will be described below with reference to the drawings. The present invention is not limited to the following embodiment, and various configurations may be adopted. As an example, each means / step according to the present invention may appropriately perform messaging via e-mail, SMS, or the like, data input / output via API, or the like in order to realize its action and effect.

A natural language processing system, a natural language processing method, and a natural language processing program can exert similar effects. Further, the action and effect of each means and the action and effect of each step bearing the same name are the same.

The natural language processing program may be stored in a non-transient recording medium. A non-transient recording medium in which the natural language processing program is stored is used to install the natural language processing program on a computer device.

《Hardware configuration》
As shown in FIG. 1, the natural language processing system includes one or more servers 10, one or more cache servers 20, and one or more terminals 30.

The server 10 is a computer, and includes at least a calculation unit 11, a main storage unit 12, an auxiliary storage unit 13, an input unit 14, a display unit 15, and a communication unit 16. Each part is used to realize the action and effect of each means related to the server 10.

The cache server 20 is a computer, and includes at least a calculation unit 21, a main storage unit 22, an auxiliary storage unit 23, an input unit 24, a display unit 25, and a communication unit 26. Each part is used to realize the action and effect of each means related to the cache server 20.

The terminal 30 is a computer, and includes at least a calculation unit 31, a main storage unit 32, an auxiliary storage unit 33, an input unit 34, a display unit 35, and a communication unit 36. Each part is used to realize the action and effect of each means according to the terminal 30.

The server 10, the cache server 20, and the terminal 30 are connected to each other via a network with each communication unit. The network is composed of a public network and / or a private network, and there are no restrictions on communication protocols and the like.

Each part of the server 10, the cache server 20, and the terminal 30 will be described below.

The arithmetic units 11, 21 and 31 include a known processor such as a CPU. The main storage units 12, 22, and 32 include known volatile devices such as RAM. Auxiliary storage units 13, 23, and 33 include known non-volatile devices such as flash memory, and store an OS and a program. The auxiliary storage unit 13 may function as at least a part of the databases DB1, DB2, DB3, DB4, DB5, and DB6 described later. Input units 14, 24, and 34 are used for data input of a natural language processing system and the like. The input unit 34 includes an input device such as a keyboard and a touch panel. The display units 15, 25, and 35 are used for data display processing of a natural language processing system, and include a display device, a graphic controller, and the like. Communication units 16, 26, and 36 are used for communication processing. At least a part of the databases DB1, DB2, DB3, DB4, DB5, and DB6 may be an external database capable of communicating with the server 10.

The server 10 and the cache server 20 may have a known device configuration such as a workstation and may not include the display unit 15 or 25. In addition, the terminal 30 can adopt a known device configuration such as a smartphone or a laptop.

《Block diagram》
As shown in FIGS. 2 and 3, in the present invention, the databases DB1, DB2, DB3, DB4, DB5, and DB6, the server 10, the cache server 20, and the terminal 30 are organically combined. It is realized by being.

The database DB1 stores text information 1001 indicating each of the first text T1, the second text T2, the third text T3, and the fourth text T4. Further, the database DB1 further stores text meta information 1002 indicating each category or natural language of the first text T1, the second text T2, the third text T3, and the fourth text T4. Further, the database DB1 further stores morpheme string information 1003 indicating each of the morpheme strings P1, P2, P3, and P4. The database DB1 further stores the variance vector information 1004 indicating each of the variance vectors V1, V2, V3, and V4.

The database DB 2 stores information related to the web page 0 received by the reception means 101. In addition, the database DB2 includes link information 2001 corresponding to web page 0, category information 2002 indicating the category of web page 0, domain information 2003 which is a country code top-level domain indicated by link information 2001, and web page 0. Language information 2004 for identifying the natural language indicated by the text contained in is further stored.

The database DB 3 has the information of the web page 0 received by the reception means 101. Further, the database DB 3 has the style sheet information 3001 of the web page 0 and the script information 3002 indicating the script code of the web page 0.

The database DB4 takes the form of a user-defined dictionary and has word information 4001.

The database DB5 has vocabulary information 5001 for representing at least a part of the variance vector of the morpheme sequence based on the trained model 1031 described later. The vocabulary information 5001 includes one or more words.

The database DB 6 has query information 6001 indicating a query related to scraping, and link information 6002 indicating one or more web pages 2 which are scraping destinations.

The server 10 has at least a reception unit 101, an analysis unit 102, an evaluation unit 103, a translation unit 104, a collection unit 105, a determination unit 106, and a determination unit 107. Each means included in the server 10 may be realized by distributed processing by one or more servers 10. Each means included in the server 10 may be configured to be shared by each of one or more servers 10.

The reception means 101 accepts the designation of the web page 0 by the terminal 30. The reception means 101 stores the link information 2001, the category information 2002, the domain information 2003, and the language information 2004 on the database DB2. As used herein, the term "web page" refers to at least one of web pages 0, 1, and 2. Further, the "category" in the category information 2002 refers to IT, software, ASP, and the like.

The analysis means 102 morphologically analyzes the text, determines the morpheme string of the text, and stores the morpheme string in the database DB1. Further, the analysis means 102 uses the morphological analysis engine 1021 stored in the server 10 and / or the external server. Morphological analysis engine 1021 refers to a known morphological analysis engine. As used herein, the term "text" refers to at least one of a first text T1, a second text T2, a third text T3, and a fourth text T4. Further, the "morpheme string" in the description in the present specification refers to at least one of the morpheme strings P1, P2, P3, and P4.

The evaluation means 103 determines the variance vector based on at least a part of the morpheme sequence, and stores the variance vector in the database DB1. Further, the evaluation means 103 uses the trained model 1031 stored in the server 10 and / or the external server. The trained model 1031 refers to a known neural network model such as a CBoW model, a Skip-Gram model, a DoBW model, and a PV-DM model. The input value of the trained model 1031 is a numerical vector based on at least a part of the morpheme string, and is one or more one-hot vectors based on at least a part of the morpheme string acquired by the analysis means 102 and the vocabulary. A part of the hidden layer or the output layer of the trained model 1031 shows a variance vector. As used herein, the "dispersion vector" refers to at least one of the variance vectors V1, V2, V3, and V4. The variance vector of the morpheme sequence may be based on the variance vector of each word and phrase constituting the morpheme sequence. The training according to the trained model 1031 may be accelerated by applying a hierarchical softmax, negative sampling, an embedded layer, or the like.
Of course, the trained model 1031 uses words, peripheral words, and the like as teacher values as appropriate according to the type of neural network model.

The translation means 104 converts the text of the original language as the input value into the text of the target language as the output value, and stores the text of the target language in the database DB1. Further, the translation means 104 uses a translation engine 1041 stored in the server 10 and / or an external server.
Translation engine 1041 refers to a known translation engine. The translation engine 1041 may be rule-based or corpus-based. The translation engine 1041 may be based on statistical machine translation or neural machine translation. The "original language" and the "objective language" in the description herein refer to known natural languages such as Japanese and English, respectively. It goes without saying that the "original language" is the translation source language of the natural language processing according to the present invention, whereas the "objective language" is the translation destination language of the natural language processing according to the present invention.

The collecting means 105 performs scraping based on the query information 6001, acquires a text, and stores the text in the database DB1. Further, the collecting means 105 uses a scraper 1051 stored in the server 10 or an external server. Scraper 1051 refers to a known scraper. Further, the collecting means 105 may determine a query as an output value based on the text as an input value. The collecting means 105 may determine a part of the phrase of the text as an input value as a query. In addition, the collecting means 105 may extend the query based on various information on the web page and / or the text held by the collecting means 105. At this time, the collecting means 105 may extend the query based on the text on a predetermined web page corresponding to the category indicated by at least a part of the text as an input value. A category corresponding to at least a part of the text as an input value and a link indicating a predetermined web page are input by a user operation or the like and stored in any of various databases in the natural language processing system, as appropriate. Referenced by collection means 105. The collection target of "scraping" in the description in the present specification is all the information on a known web page, and includes not only text but also images and the like. At this time, the collecting means 105 acquires the text having a high search score from the texts on the known web page scored by the known search engine based on the query, and stores the text in the database DB1. The search score indicates the degree of matching between a text or a web page and a query, and a general conventional score used by a known search engine is exemplified as one embodiment. The collection means 105 may determine the collection target based on the query. The collecting means 105 may work with a known image analysis engine to estimate text from an image on a web page. Based on the word information 4001, the collecting means 105 may perform post-processing such as replacing a predetermined word / phrase included in the text data such as the first text T1 with a proper noun as a word alignment.

The determination means 106 determines two different variance vectors as input values and a similarity 300 between the variance vectors as an output value. The determination means 106 is based on a method of calculating a known similarity index / distance index such as cosine similarity, Pearson's correlation coefficient, deviation pattern similarity, Euclidean distance, standard Euclidean distance, Mahalanobis distance, Manhattan distance, and Minkowski distance. , Determine the similarity 300.

The determination means 107 stores the web page 1 including the text of the target language in the cache server 20. At this time, the determination means 107 determines the text as a translation of the first text T1. At this time, the collecting means 105 may hold the text. The determination means 107 converts the text in the original language on the web page 0 into the text based on at least a part of various information on the databases DB2 and DB3.

"flowchart"
As shown in FIG. 4, a series of processes according to the present invention includes the following steps. The permutation of each step shown in FIG. 3 is an example, and the permutation can be changed as appropriate unless otherwise specified.

The terminal 30 specifies at least a part of the URL of the web page 0 including the first text T1 (designation step S100). At this time, the terminal 30 may be able to specify at least a part of the text on the web page 0 as the text to be processed in natural language. Next, the reception means 101 accepts the designation related to the web page 0 including the first text T1 in the designation step S100 (reception step S101).

The analysis means 102 performs morphological analysis of the first text T1 and acquires the morpheme string P1 of the first text T1 (analysis step S102). At this time, the analysis means 102 may perform preprocessing as a word alignment that replaces a predetermined word / phrase included in the text data such as the first text T1 with a proper noun based on the word information 4001. As a result, in the natural language processing according to the present invention, it can be expected that the accuracy in determining the morpheme sequence will be improved.

The evaluation means 103 acquires the morpheme sequence P1 as an input value and the variance vector V1 as an output value corresponding to the trained model 1031 (evaluation step S103). At this time, the evaluation means 103 may determine the variance vector V1 based only on the nouns, verbs, and adjectives included in the morpheme sequence P1 and the like. This facilitates noise removal in the dispersion vector in the natural language processing according to the present invention.

The translation means 104 takes the first text T1 of the original language as an input value and acquires the second text T2 of the target language as an output value via the translation engine 1041 (translation step S104). Next, the collecting means 105 determines the query information 6001 based on at least a part of the second text T2, scrapes the scraped target including the web page 2 based on the query information 6001, and performs scraping on the scraping target of the target language. The text T3 is acquired (collection step S105). Next, the translation means 104 acquires the third text T3 of the target language as an input value and the fourth text T4 of the original language as an output value via the translation engine 1041 (translation step S106).

The analysis means 102 performs morphological analysis of the fourth text T4 of the original language determined in the translation step S106, and acquires the morpheme string P4 of the fourth text T4 (analysis step S107). Next, the evaluation means 103 acquires the variance vector V4 of the morpheme sequence P4 based on the trained model 1031 (evaluation step S108).

The determination means 106 acquires the similarity 300 of the variance vector V1 and the variance vector V4 (determination step S109). When the similarity 300 exceeds the threshold value 301, the determination means 107 stores the web page 1 including the third text T3 in the cache server 20 (determination step S110). At this time, the threshold value 301 may be a value arbitrarily set. When the similarity 300 does not exceed the threshold value 301, the collecting means 105 expands the query information 6001 based on the web page 0 and then scrapes to reacquire the third text T3 of the target language (collection step S105X). ..

The natural language processing system according to the embodiment of the present invention may take the form of a Web API (hereinafter, referred to as "email translation API") capable of inputting the first text T1 and an email address. In the email translation API, as an example, the terminal 30 POSTs the first text T1 and the email address. At this time, the receiving means 101 receives the first text T1 and the e-mail address input by the terminal 30. In the mail translation API, the first text T1 and the mail address are posted as a request by the determination means 107 of the terminal 30, and the third text T3 is returned as a response by the server 10. Here, the response is realized in the mode of transmission to the e-mail address of the e-mail having the third text T3 as the e-mail body. In addition, in one embodiment which takes the form of the mail translation API, at least a part of the configuration of the other embodiment can be appropriately adopted.

The natural language processing system in one embodiment of the present invention can be understood to take the form of Web API such that the first text T1 and the third text T3 are requests and responses, respectively.

According to the present invention, a novel natural language processing can be realized.

0 Web page 1 Web page 2 Web page 10 Server 11 Calculation unit 12 Main storage unit 13 Auxiliary storage unit 14 Input unit 15 Display unit 16 Communication unit 20 Cash server 21 Calculation unit 22 Main storage unit 23 Auxiliary storage unit 24 Input unit 25 Display Unit 26 Communication unit 30 Terminal 31 Calculation unit 32 Main memory unit 33 Auxiliary storage unit 34 Input unit 35 Display unit 36 Communication unit 101 Reception means 102 Analysis means 103 Evaluation means 104 Translation means 105 Collection means 106 Judgment means 107 Determining means 300 Similarity 301 Threshold 1001 Text information 1002 Text meta information 1003 Morphological string information 1004 Distributed vector information 1021 Morphological analysis engine 1031 Trained model 1041 Translation engine 1051 Scraper 2001 Link information 2002 Category information 2003 Domain information 2004 Language information 3001 Style sheet information 3002 Script information 4001 Word information 5001 Vocabulary information 6001 Query information 6002 Link information DB1 Database DB2 Database DB3 Database DB4 Database DB5 Database DB6 Database P1 Form element string P2 Form element string P3 Form element string P4 Form element string S100 Designation step S101 Reception step S102 Analysis step S103 Evaluation step S104 Translation step S105 Collection step S105X Collection step S106 Translation step S107 Analysis step S108 Evaluation step S109 Judgment step S110 Decision step T1 First text T2 Second text T3 Third text T4 Fourth text V1 Distributed vector V2 Distributed vector V3 Distributed vector V4 dispersion vector

Claims (6)

It ’s a natural language processing method.
A reception step that accepts the designation of a web page containing the first text in the original language,
A translation step to obtain the second text of the target language based on the first text, and
A collection step of scraping at least a part of the second text as a query to obtain the third text of the target language.
A determination step of determining the third text as a translation of the first text, and
A natural language processing method that causes a computer processor to execute. It ’s a natural language processing method.
An analysis step to get the morpheme string of the text,
The evaluation step of acquiring the variance vector of the morpheme sequence and
A determination step for determining the similarity between the variance vectors, and
To the processor
The translation step obtains a fourth text in the original language based on the third text.
The analysis step acquires the morpheme string of the first text and the morpheme string of the fourth text.
The evaluation step acquires the variance vector of the first text and the variance vector of the fourth text.
A claim in which the determination step determines the third text as the translation when the similarity between the variance vectors of the first text and the fourth text determined by the determination step exceeds a threshold value. The natural language processing method according to 1. The collection step extends the query based on the web page if the similarity of the variance vector of the first text and the variance vector of the fourth text does not exceed the threshold. The natural language processing method according to claim 2, wherein scraping is performed and the third text is obtained. The evaluation step acquires the variance vector based on the trained model and obtains the variance vector.
The natural language processing method according to claim 2 or 3, wherein the trained model is a neural network model in which a part of the hidden layer or the output layer shows the dispersion vector. It ’s a natural language processing system.
A reception means that accepts the designation of a web page containing the first text in the original language,
A translation means for obtaining a second text of the target language based on the first text, and
A collecting means for scraping at least a part of the second text as a query to obtain the third text of the target language.
A natural language processing system having a determination means for determining the third text as a translation of the first text. A natural language processing program that uses a computer
A reception means that accepts the designation of a web page containing the first text in the original language,
A translation means for obtaining a second text of the target language based on the first text, and
A collecting means for scraping at least a part of the second text as a query to obtain the third text of the target language.
A natural language processing program that functions as a determination means for determining the third text as a translation of the first text.

Images