KR20220107738A - Method, system, and computer readable record medium for ranking reformulated query - Google Patents

Abstract

A method, a system, and a computer-readable recording medium for ranking a reconstructed query are disclosed. The query ranking method comprises the steps of: generating, by the at least one processor, a first query candidate through query reformulation using a statistical machine translation model for an original query; generating, by the at least one processor, a second query candidate through query reconstruction using a bidirectional encoder representations from transformer (BERT) based machine translation model with respect to the original query; and ranking, by the at least one processor, the first query candidate and the second query candidate using a feature related to a search. The present invention uses a BERT model along with an SMT model.

Description

Method, system, and computer-readable recording medium for ranking reconstructed queries

The description below relates to a query reformulation (referred to as 'QR') technique for providing optimal search results.

A user may conduct a search to obtain desired information through a site such as a search engine. The user may obtain desired information by inputting a query into a query input window of a search engine through the user terminal and checking a search result output for the query.

In a query-term matching-based search system, a query reconstruction technique capable of providing an optimal search result by adding reconstructed queries having similar meanings to a user's query is used.

For example, in Korea Patent Application Laid-Open No. 10-2011-0007743 (published on January 25, 2011), a technique capable of correcting user queries determined to be misspelled queries based on statistical data according to the entire query unit or word unit This is disclosed.

A method and system for generating QR candidates for user queries are provided by additionally using the BERT (Bidirectional Encoder Representations from Transformer) model together with the SMT (statistical machine translation) model.

We provide a method and system that can rank QR candidates generated from a machine translation model combined with an SMT model and a BERT model based on search-related features in order to find a suitable QR for searching.

A method for ranking a query executed on a computer device, the computer device comprising at least one processor configured to execute computer readable instructions contained in a memory, the method comprising: by the at least one processor , generating a first query candidate through query reformulation using a statistical machine translation model for the original query; generating, by the at least one processor, a second query candidate through query reconstruction using a BERT (Bidirectional Encoder Representations from Transformer)-based machine translation model with respect to the original query; and ranking, by the at least one processor, the first query candidate and the second query candidate using a search-related feature.

According to one aspect, the generating of the second query candidate may include generating the second query candidate through the BERT-based machine translation model, which is a machine translation model in which an encoder part of a transformer model is replaced with a BERT model. can

According to another aspect, the query ranking method further comprises, by the at least one processor, storing in a cache the query candidates that have been inferred using the BERT-based machine translation model, and the The generating of the second query candidate may include finding and serving the second query candidate corresponding to the original query in the cache.

According to another aspect, in the ranking, the first query candidate and the second query candidate may be ranked through a ranking model that has learned the feature using a gradient boosting regression tree (GBRT) algorithm.

According to another aspect, the feature includes the number of other queries including a query term in relation to the original query, the number of search inflows, the number of clicked documents, the number of impression documents, the click statistics of the search area compared to the surface area, and the length of the query. A first feature including at least one, the number of other queries including a query term in relation to each query candidate corresponding to the first query candidate and the second query candidate, the number of search inflows, the number of clicked documents, and the number of exposure documents The second feature may include at least one of a number, a click statistic of a search region compared to the exposure region, and a query length, and a third feature indicating a difference between the same feature of the original query and the query candidate pair.

According to another aspect, in the ranking step, the search result of the original query, which is at least a part of documents included in the search result of the query candidate, for each query candidate corresponding to the first query candidate and the second query candidate. The method may include calculating a search relevance score of the query candidate by using user feedback information on the included document.

According to another aspect, the ranking includes: classifying a query candidate having a lower search relevance score than the original query among the first query candidate and the second query candidate as a non-relevant query, and wherein the search relevance score is the The method may further include classifying a query candidate larger than the original query as a suitable query.

According to another aspect, the ranking may include acquiring a feature collection generated from a log related to a search using a search engine.

According to another aspect, the ranking may include tuning the ranked query candidates based on word embeddings of terms remaining after removing a common term with the original query.

According to another aspect, the ranking may include tuning the ranked query candidates based on at least one of search-related statistical information, a word length or a number of word terms, and an editing distance in units of grapheme. can

There is provided a computer-readable recording medium in which a program for executing the query ranking method in a computer is recorded.

A computer device comprising: at least one processor configured to execute computer readable instructions contained in a memory, wherein the at least one processor is configured to: a first query candidate through query reconstruction using a statistical machine translation model on an original query generates a second query candidate through query reconstruction using a BERT-based machine translation model for the original query, and ranks the first query candidate and the second query candidate using search-related features It provides a computer device, characterized in that.

According to embodiments of the present invention, QR quality and search quality can be improved by generating a QR candidate for a user query by further using the BERT model together with the SMT model.

According to embodiments of the present invention, a QR suitable for a search can be efficiently found by ranking QR candidates generated in a machine translation model in which an SMT model and a BERT model are combined based on a search-related feature.

1 is a diagram illustrating an example of a network environment according to an embodiment of the present invention.
2 is a block diagram illustrating an example of a computer device according to an embodiment of the present invention.
3 is a flowchart illustrating an example of a method that a computer device may perform according to an embodiment of the present invention.
4 shows an example of a multi-QR generation model for QR generation in an embodiment of the present invention.
5 to 9 show an example of an SMT model for QR generation in an embodiment of the present invention.
10 to 11 show an example of a BERT-based machine translation model for QR generation in an embodiment of the present invention.
12 shows an example of a BERT serving system for a real-time service according to an embodiment of the present invention.
13 shows an example of a cache-based BERT serving system architecture according to an embodiment of the present invention.
14 shows an example of a ranking configuration for QR candidate verification in an embodiment of the present invention.
15 shows examples of non-conforming QR candidates and suitable QR candidates according to an embodiment of the present invention.
16 is a diagram illustrating an example of a process for obtaining features necessary for ranking QR candidates in an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

The query ranking system according to the embodiments of the present invention may be implemented by at least one computer device, and the query ranking method according to the embodiments of the present invention is at least one computer device included in the query ranking system. can be done through In this case, the computer program according to an embodiment of the present invention may be installed and driven in the computer device, and the computer device may perform the query ranking method according to the embodiments of the present invention under the control of the driven computer program. have. The above-described computer program may be stored in a computer-readable recording medium in order to be combined with a computer device to cause the computer to execute the query ranking method.

1 is a diagram illustrating an example of a network environment according to an embodiment of the present invention. The network environment of FIG. 1 shows an example including a plurality of electronic devices 110 , 120 , 130 , 140 , a plurality of servers 150 , 160 , and a network 170 . 1 is an example for explaining the invention, and the number of electronic devices or the number of servers is not limited as in FIG. 1 . In addition, the network environment of FIG. 1 only describes one example of environments applicable to the present embodiments, and the environment applicable to the present embodiments is not limited to the network environment of FIG. 1 .

The plurality of electronic devices 110 , 120 , 130 , and 140 may be a fixed terminal implemented as a computer device or a mobile terminal. Examples of the plurality of electronic devices 110 , 120 , 130 , 140 include a smart phone, a mobile phone, a navigation device, a computer, a notebook computer, a digital broadcasting terminal, a personal digital assistant (PDA), and a portable multimedia player (PMP). ), tablet PCs, etc. As an example, in FIG. 1 , the shape of a smartphone is shown as an example of the electronic device 110 , but in embodiments of the present invention, the electronic device 110 is substantially configured to be different through the network 170 using a wireless or wired communication method. It may refer to one of various physical computer devices capable of communicating with the electronic devices 120 , 130 , 140 and/or the servers 150 and 160 .

The communication method is not limited, and not only a communication method using a communication network (eg, a mobile communication network, a wired Internet, a wireless Internet, a broadcasting network) that the network 170 may include, but also short-range wireless communication between devices may be included. For example, the network 170 may include a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), and a broadband network (BBN). , the Internet, and the like. In addition, the network 170 may include any one or more of a network topology including a bus network, a star network, a ring network, a mesh network, a star-bus network, a tree, or a hierarchical network, etc. not limited

Each of the servers 150 and 160 communicates with the plurality of electronic devices 110 , 120 , 130 , 140 and the network 170 through a computer device or a plurality of computers providing commands, codes, files, contents, services, etc. It can be implemented in devices. For example, the server 150 may be a system that provides a target service (eg, a search service) to the plurality of electronic devices 110 , 120 , 130 , and 140 connected through the network 170 .

2 is a block diagram illustrating an example of a computer device according to an embodiment of the present invention. Each of the plurality of electronic devices 110 , 120 , 130 , 140 or the servers 150 and 160 described above may be implemented by the computer device 200 illustrated in FIG. 2 .

As shown in FIG. 2 , the computer device 200 may include a memory 210 , a processor 220 , a communication interface 230 , and an input/output interface 240 . The memory 210 is a computer-readable recording medium and may include a random access memory (RAM), a read only memory (ROM), and a permanent mass storage device such as a disk drive. Here, a non-volatile mass storage device such as a ROM and a disk drive may be included in the computer device 200 as a separate permanent storage device distinct from the memory 210 . Also, the memory 210 may store an operating system and at least one program code. These software components may be loaded into the memory 210 from a computer-readable recording medium separate from the memory 210 . The separate computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD/CD-ROM drive, and a memory card. In another embodiment, the software components may be loaded into the memory 210 through the communication interface 230 instead of a computer-readable recording medium. For example, the software components may be loaded into the memory 210 of the computer device 200 based on a computer program installed by files received through the network 170 .

The processor 220 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input/output operations. The instructions may be provided to the processor 220 by the memory 210 or the communication interface 230 . For example, the processor 220 may be configured to execute a received instruction according to a program code stored in a recording device such as the memory 210 .

The communication interface 230 may provide a function for the computer device 200 to communicate with other devices (eg, the storage devices described above) through the network 170 . For example, a request, command, data, file, etc. generated by the processor 220 of the computer device 200 according to a program code stored in a recording device such as the memory 210 is transmitted to the network ( 170) to other devices. Conversely, signals, commands, data, files, etc. from other devices may be received by the computer device 200 through the communication interface 230 of the computer device 200 via the network 170 . A signal, command, or data received through the communication interface 230 may be transmitted to the processor 220 or the memory 210 , and the file may be a storage medium (described above) that the computer device 200 may further include. persistent storage).

The input/output interface 240 may be a means for an interface with the input/output device 250 . For example, the input device may include a device such as a microphone, keyboard, or mouse, and the output device may include a device such as a display or a speaker. As another example, the input/output interface 240 may be a means for an interface with a device in which functions for input and output are integrated into one, such as a touch screen. The input/output device 250 may be configured as one device with the computer device 200 .

Also, in other embodiments, the computer device 200 may include fewer or more components than those of FIG. 2 . However, there is no need to clearly show most of the prior art components. For example, the computer device 200 may be implemented to include at least a portion of the above-described input/output device 250 or may further include other components such as a transceiver and a database.

Hereinafter, a specific embodiment of a method and system for ranking reconstructed queries will be described.

The present embodiments relate to a QR technology for application to a query-term matching based search service. The goal is to view QR as a translation problem and transform it into a query suitable for search using a machine translation model. In particular, in this embodiment, a query reconstructed for a user query (hereinafter referred to as a 'QR candidate') is generated through a translation model in which the SMT model and the BERT model are combined, and then information on whether the query is suitable for searching is reflected. For this purpose, ranking of QR candidates may be performed.

3 is a flowchart illustrating an example of a method that a computer device may perform according to an embodiment of the present invention.

The computer device 200 according to the present embodiment may provide a search service to the client through a dedicated application installed on the client or a web/mobile site connection related to the computer device 200 . The computer device 200 may be configured with a computer-implemented query ranking system. For example, the query ranking system may be implemented in the form of a program that operates independently, or may be implemented in the form of an in-app of a specific application to enable operation on the specific application.

The processor 220 of the computer device 200 may include at least one component as a component for performing the query ranking method according to FIG. 3 . Depending on the embodiment, components of the processor 220 may be selectively included or excluded from the processor 220 . Also, according to an embodiment, the components of the processor 220 may be separated or merged to express the functions of the processor 220 .

The processor 220 and components of the processor 220 may control the computer device 200 to perform steps S310 to S350 included in the query ranking method of FIG. 3 . For example, the processor 220 and components of the processor 220 may be implemented to execute instructions according to the code of the operating system included in the memory 210 and the code of at least one program.

Here, the components of the processor 220 may be expressions of different functions performed by the processor 220 according to instructions provided by the program code stored in the computer device 200 .

The processor 220 may read a necessary command from the memory 210 in which the command related to the control of the computer device 200 is loaded. In this case, the read command may include a command for controlling the processor 220 to execute steps S310 to S350 to be described later.

Steps S310 to S350 to be described later may be performed in an order different from the order shown in FIG. 3 , and some of the steps S310 to S350 may be omitted or additional processes may be further included.

Referring to FIG. 3 , in step S310 , the processor 220 may generate a first QR candidate group by reconstructing a user query using an SMT model that is a machine translation model. The processor 220 may generate the first QR candidate group through the SMT model trained with the parallel corpus training data.

In step S320, the processor 220 may generate the second QR candidate group by reconstructing the user query using the BERT-based machine translation model. While the SMT model can generate many QR candidates with a fast translation speed, it has a disadvantage in that it lacks context-aware translation. To compensate for this, the BERT model is applied to the QR. In the case of the BERT model, which is a classification model, unlike SMT, it is possible to understand the meaning of words and to translate them naturally reflecting the context. The BERT model can be trained with a pre-training technique using a masked language model (MLM) and next sentence prediction (NSP) for deeper language understanding.

In step S330, the processor 220 may merge the first QR candidate group generated from the SMT model and the second QR candidate group generated from the BERT-based machine translation model and rank them by utilizing the search-related features. In the QR using the machine translation model, only linguistic fluency is reflected, and information on whether the QR is suitable for searching is not reflected. Many QR candidates are being generated using the SMT model and the BERT-based machine translation model, and even if the linguistic translation is good, the query may not be suitable for search, so QR verification through ranking is necessary. The processor 220 may score the first QR candidate group and the second QR candidate group by using user feedback information in the search service. For example, the processor 220 may rank the first QR candidate group and the second QR candidate group through a learning model using a gradient boosting regression tree (GBRT) algorithm.

In step S340 , the processor 220 is a tuning process for improving the quality of the ranking result, and may remove some QR candidates through tuning using each query feature with respect to the QR candidate group ranked in step S330 . The processor 220 may tune by using the query feature to remove low-quality cases even if the candidate ranks higher in the QR candidate group.

For example, the processor 220 may remove a QR candidate that is not suitable for a search by using the word embedding similarity from which the common term is removed. For example, for the user query 'phone xs size' and the QR candidate 'phone xr size', the word embedding similarity is compared between the remaining terms except for the common term 'phone' and 'size', and the similarity is higher than the threshold. If it is low, the QR candidate 'phone xr size' can be removed.

As another example, the processor 220 may remove QR candidates that are not suitable for search based on statistical information related to the search. In this case, the statistical information may include a query count (QC) rate, an exposure rate, a click rate, and the like. For example, for the user query 'phone xs size' and the QR candidate 'phone xr size', if the QC, exposure, and click of 'phone xr size' are all lower than the reference value, the QR candidate 'phone xr size' can be removed. can

As another example, the processor 220 may remove QR candidates that are not suitable for searching through a combination with the above tuning criteria (eg, word embedding similarity, statistical information) based on the word length or the number of word terms. have. For example, for the user query 'Top' and the QR candidate 'Top', since the ambiguity of one syllable is very high, the two queries are compared by combining the QC ratio or word embedding rule, and then the QR candidate ' tower can be removed.

As another example, the processor 220 may remove QR candidates that are not suitable for search through combination with the above-described tuning criteria (eg, word embedding similarity and statistical information) based on the grapheme unit editing distance. For example, for the user query 'convenience store' and the QR candidate 'convenience sleep', if the grapheme unit editing distance between the queries is 1, the probability of a typo-type query is high. According to the comparison result, the QR candidate 'comfortable sleep' can be removed.

In step S350 , the processor 220 may select at least some of the remaining QR candidates after tuning as a search query to provide a search result corresponding to the user query. For example, the processor 220 may use a certain number of top QR candidates or a certain ratio of QR candidates as an actual search query based on the QR ranking. In other words, the search results corresponding to the user query may include documents matching each QR candidate selected as the search query.

As shown in FIG. 4 , the processor 220 may include an SMT model 410 and a BERT-based machine translation model 420 as a multi-model structure. The processor 220 may generate the first QR candidate group 401 through the reconstruction using the SMT model 410 for the user query 40, and also the first QR candidate group 401 through the reconstruction using the BERT-based machine translation model 420. 2 QR candidate groups 402 may be generated.

The structure of the SMT model 410 is shown in FIG. 5 .

Referring to FIG. 5 , the SMT model 410 provides a translation result with a statistical probability, and includes a translation model 501 and a language model 502 .

The translation model 501 converts a source language into a target language, and requires parallel corpus training data in which a source sentence and a target sentence are paired.

The translation model 501 includes an alignment model and a lexical translation model, and as shown in FIG. 6 , the alignment 601 can be extracted.

The language model 502 predicts the probability of a sentence, and requires learning data for a target language. Referring to FIG. 7 , when consecutive words 'where are we going' are given, the probability that 'are' comes after 'where', the probability that 'we' comes after 'where are', etc. are calculated.

The SMT model 410 may create a statistical translation result by combining the translation model 501 and the language model 502 .

The QR system to which the SMT model 410 is applied is shown in FIG. 8 .

A translation model 501 and a language model 502 composed of an alignment model and a lexical translation model can be learned from the parallel corpus dictionary.

Referring to FIG. 8 , the QR system to which the SMT model 410 is applied goes through a tokenization process (morphological tokenization), a decoding process (SMT decoder), a result generation process (N best QR result), and a heuristic logic. By converting a source query, which is an original query, into a target query, a reconstructed query, that is, the first QR candidate group 401 may be generated.

For example, as shown in FIG. 9 , in the QR system to which the SMT model 410 is applied, when 'Naver free font' is given to the user query 901 that is the original query, 'Naver', 'free', 'face' After tokenization as shown above, 'Naver free font', 'Naver free font', and 'Naver free font' can be generated as a QR candidate group 902 for the user query 901 through the decoding process and result generation process. .

The QR system to which the SMT model 410 is applied has a limitation in that it depends on the language model 502 for the context, so there is a disadvantage in that the translation considering the context compared to deep learning is insufficient.

In this embodiment, a QR system using the BERT-based machine translation model 420 may be added to improve the quality limit of the SMT model 410 .

10 shows the BERT model structure.

Referring to FIG. 10, the BERT model 1000 receives a given sentence as an input, first embeds it in token units through an input embedding layer 1001, and then passes through a transformer layer 1002 to a token. It has a structure that expresses contextual representation with encoding considering the location information of

The transformer layer 1002 consists of a self-attention model instead of a model such as CNN or RNN, and the BERT model 1000 embeds a language using an encoder among the encoders and decoders of the transformer.

By replacing the encoder part of the transformer model with the BERT model 1000, it is possible to build a BERT-based machine translation model 420 for application to the QR system.

The QR system to which the BERT-based machine translation model 420 is applied is shown in FIG. 11 .

The BERT model, a classification model, shows very high performance when applied to various language processing or information retrieval fields. A QR system for generating a QR candidate through a translation model may be composed of an encoder 1110 for understanding an input query as a transformer model and a decoder 1120 for generating a QR candidate as a transformer model, as shown in FIG. The encoder 1110 part may be replaced with the BERT model 1000 .

The encoder 1110 can deeply understand the user query 1101, which is the original query, using the BERT model 1000, and understand the meaning of words to enable natural translation reflecting the context. The decoder 1120 may generate the QR candidate group 1102 for the user query 1101 as a decoder of the transformer model.

For example, as shown in FIG. 11 , the QR system to which the BERT-based machine translation model 420 is applied is deeper using the BERT model 1000 when 'Teenage women's music ranking' is given as the user query 1101. 'Teenage female music ranking', 'teenage female song ranking', 'teenage female music ranking', and 'teenage female music ranking' are generated as a QR candidate group 1102 for the user query 1101 through language understanding. can do.

Accordingly, the processor 220 may generate a QR candidate for providing an optimal search result by using the BERT-based machine translation model 420 together with the SMT model 410 as a multi-QR generation model.

In the case of the BERT-based machine translation model 420, it is difficult to guarantee performance in a real-time service because the model size is large.

Referring to FIG. 12 , the processor 220 may further include a cache-based BERT serving system 1230 to overcome the response speed problem of the BERT-based machine translation model 420 . The BERT serving system 1230 may apply a method of storing the inferred QR result of the BERT-based machine translation model 420 in a cache.

In general, since QR is performed prior to searching for each collection when a query is received in the integrated search, the response speed of the integrated search becomes slower as the response speed of the QR becomes slower. To solve this, a method of storing and using the inferred QR result of the BERT-based machine translation model 420 in the cache server can be applied by adding a cache server.

13 shows an example of a BERT serving system structure in an embodiment of the present invention.

Referring to FIG. 13 , the BERT serving system 1230 may include a cache server 1301 , a Neural Machine Translation (NMT) producer 1302 , a queue 1303 , and an NMT consumer 1304 .

The NMT producer 1302 serves to find a QR candidate corresponding to the user query by referring to the cache server 1301 when a user query is input and to return the result.

Meanwhile, the NMT producer 1302 inserts the user query into the queue 1303 when there is no QR result corresponding to the user query in the cache server 1301 . When the same query is continuously input, a blank value is inserted into the cache server 1301 because duplicate queries may be entered in the queue 1303 . The empty value inserted into the cache server 1301 may be temporarily maintained for a certain period of time (eg, 1 hour).

The queue 1303 serves as a message queue, such as Kafka, and may maintain a query received from the NMT producer 1302 and transmit it to the NMT consumer 1304 .

The NMT consumer 1304 may receive a query list from the queue 1303 and perform inference on the QR candidate using the BERT-based machine translation model 420 . The NMT consumer 1304 inserts the inference result for the QR candidate of the BERT-based machine translation model 420 into the cache server 1301 . The QR result inferred on the cache server 1301 may be updated in units of a predetermined time (eg, 1 day).

The processor 220 may calculate the number of serving equipment required to process the queries accumulated in real time. The processor 220 may determine the number of devices constituting the BERT serving system 1230 to process queries input in real time within a unit time (eg, 1 second).

As an example, the processor 220 may calculate the number of serving devices based on the maximum number of inferences per unit time and the maximum unique query per second (UQPS). It is more advantageous to process n queries in batches than to perform inference on one query n times. For example, to process UQPS 1,500 with a throughput of 1 second (16 queries), approximately 94 (1500/16=93.75) model server containers are required.

Therefore, in this embodiment, a real-time service can be guaranteed by using a cache-based serving platform to apply the BERT model, which has a large model size and slow response speed, to the QR system.

In addition to using a cache-based serving platform, it is also possible to reduce the parameters of the BERT model by three layers and apply a knowledge distillation technique for model compression to optimize performance.

The QR verification process through ranking is as follows.

Referring to FIG. 14 , the processor 220 ranks the QR ranking model 1460 for ranking the first QR candidate group 401 and the second QR candidate group 402 for QR verification, and features related to the search. It may further include a feature server 1470 for forwarding to the model 1460 .

The processor 220 is a first QR candidate group and a second QR candidate group through the QR ranking model 1460 learned using a learning target data set (hereinafter, referred to as 'learning data') 1450 can be ranked in the order suitable for the search.

First, the processor 220 may generate the click (or exposure) graph-based learning data 1450 for learning the QR ranking model 1460 . The QR ranking model 1460 must be trained to rank the QR candidates well in an effective order for actual retrieval. As an example, the processor 220 may generate a click graph for QR candidates generated from the SMT model 410 and the BERT-based machine translation model 420, and then extract and score a QR candidate suitable for a search from the click graph. . The processor 220 may generate the training data 1450 as a click graph and learn the QR ranking model 1460 using the generated training data 1450 .

The processor 220 does not find a QR to be used as a search query based on the translation score through the SMT model 410 and the BERT-based machine translation model 420, but uses both the search-related features and the translation score to perform the search. As a problem of finding a suitable QR, the QR ranking model 1460 can be trained.

The processor 220 merges the first QR candidate group 401 generated from the SMT model 410 and the second QR candidate group 402 generated from the BERT-based machine translation model 420 through the QR ranking model 1460. Ranking may be performed on all QR candidates. At this time, the processor 220 may secure several features that are helpful in the search through the feature server 1470, and based on these features, search through the QR ranking model 1460 trained with the learning data 1450. Distinguish between acceptable and unsuitable QRs.

The processor 220 may use the actual user click information in the search service to score the fit between the original query and the QR candidate and use it as correct answer data for learning the QR ranking model 1460 .

The fitness score of the QR candidate with respect to the original query may be defined by Equation (1).

[Equation 1]

Here, rank(di) means the ranking of the document di included in the search result corresponding to the QR candidate, and clickCount(di) means the number of clicks of the document di included in the search result corresponding to the original query.

In other words, only the scores of documents that appear in common with the search results of the original query are reflected in the suitability score of the QR candidate.

Based on the click graph, the processor 220 classifies a QR candidate having a score lower than the original query among the QR candidates as a non-conforming QR candidate, and classifies a QR candidate having a higher score than the original query as a suitable QR candidate.

15 shows an example of classification of suitable QR candidates and non-conforming QR candidates.

For example, although QR Candidate 1 ranks documents with significant clicks, we cannot guarantee that these documents are related to the original query. As shown in FIG. 15 , assuming that Document _k and Document _m among the documents included in the search result of QR candidate 1 do not appear in the search result of the original query, the corresponding document is not reflected in the suitability score of QR candidate 1. Although Document ₃ , which has 100 clicks as a document that appeared in the search results of the original query, was ranked at the top, if the fitness score of QR Candidate 1 is smaller than the fitness score of the original query, QR Candidate 1 is classified as a non-conforming QR candidate.

As shown in FIG. 15 , assuming that all documents included in the search result of QR Candidate 2 appear in the search result of the original query and are exposed in the order of the number of user clicks, the fitness score of QR Candidate 2 is the original If it is greater than the relevance score of the query, QR candidate 2 is classified as a suitable QR candidate.

The processor 220 may determine whether the QR candidates are suitable through the QR ranking model 1460 and rank them in an order suitable for a search, that is, according to a fitness score. In this case, it is possible to secure features that can improve search quality and learn by using the GBRT algorithm.

Search-related features may be configured as shown in Table 1.

Feature Group Feature original query feature f1: Search QC f2 : the number of other queries that contain the search query term f3 : The number of search inflows, such as keyboard input, real-time spiked search term clicks, topic keyword clicks, etc. f4 : number of documents clicked in integrated search f5 : Unified Search Impression Clicks f6 : Relative click statistics of content search area compared to other search surface area such as blog, web, cafe, etc. f7 : language model score f8 : length of query f9 : Number of query terms in units of morphemes QR Candidate Features f11: Search QC f12 : the number of other queries that contain the search query term f13 : The number of search inflows for each search such as keyboard input, real-time spiked search term clicks, topic keyword clicks, etc. f14 : number of documents clicked in integrated search f15 : Unified Search Impression Clicks f16 : Relative click statistics of content search area compared to other search surface area such as blog, web, cafe, etc. f17 : language model score f18 : length of the query f19 : Number of query terms in units of morphemes Query pair feature f21 : f1 - f11 f22 : f2 - f12 f23 : The number of terms common to the original query and QR candidates f24 : f4 - f14 f25 : f5 - f15 f26 : Cosine similarity between the original query and the QR candidate f27 : f7 - f17 f28 : f8 - f18 f29 : f9 - f19 f30 : Cosine similarity between the original query and QR candidates excluding the common occurrence term f31 : SMT translation score f32 : BERT translation score f33 : SMT translation rank f34 : BERT translation rank

In addition, in this embodiment, the feature server 1470 configured as a search engine may be used to quickly transfer features related to a search to the QR ranking model 1460 for QR candidates.

For example, as shown in FIG. 16 , the feature server 1470 includes a search application server (SAS) 1610 and a search engine server to deliver features necessary for ranking QR candidates to the QR ranking model 1460. engine server) 1620 .

The feature server 1470 may include a refining system 1630, wherein the processing system 1630 processes logs 1601 related to searches such as exposure logs and click logs for a recent period of time to perform QC, click A feature collection 1602 may be created and indexed, such as number, number of exposed documents, number of sm parameters, and the like.

Feature server 1470 may serve features in queries containing QR candidate strings along with indexed feature collection 1602 . In other words, by using the SAS 1610 and the search engine server 1620, it is possible to check the features of other QR candidate character strings including the QR candidate character string.

Then, the feature server 1470 embeds the search-related log 1601 and/or the document 1603 corresponding to each query as a feature necessary for calculating the similarity between the original query and the QR candidate, and embeds it in the QR ranking model 1460. ), and the language model value of the corresponding query (ie, the probability value of the sentence) can also be provided.

Therefore, in this embodiment, in order to select a query more suitable for a search from among numerous QR candidates generated through a multi-model structure including the SMT model 410 and the BERT-based machine translation model 420, search-related features are used. It is possible to verify and rank the corresponding QR candidates.

As such, according to embodiments of the present invention, a QR suitable for a search can be efficiently found by ranking QR candidates generated in a machine translation model in which the SMT model and the BERT model are combined based on the features related to the search.

The device described above may be implemented as a hardware component, a software component, and/or a combination of the hardware component and the software component. For example, the devices and components described in the embodiments may include a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), and a programmable logic unit (PLU). It may be implemented using one or more general purpose or special purpose computers, such as a logic unit, microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that may include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

Software may comprise a computer program, code, instructions, or a combination of one or more of these, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be embodied in any type of machine, component, physical device, computer storage medium or device for interpretation by or providing instructions or data to the processing device. have. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. In this case, the medium may be to continuously store the program executable by the computer, or to temporarily store the program for execution or download. In addition, the medium may be various recording means or storage means in the form of a single or several hardware combined, it is not limited to a medium directly connected to any computer system, and may exist distributed on a network. Examples of the medium include a hard disk, a magnetic medium such as a floppy disk and a magnetic tape, an optical recording medium such as CD-ROM and DVD, a magneto-optical medium such as a floppy disk, and those configured to store program instructions, including ROM, RAM, flash memory, and the like. In addition, examples of other media include recording media or storage media managed by an app store that distributes applications, sites that supply or distribute various other software, and servers.

As described above, although the embodiments have been described with reference to the limited embodiments and drawings, various modifications and variations are possible from the above description by those skilled in the art. For example, the described techniques are performed in an order different from the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (20)

A method for ranking a query executed on a computer device, the method comprising:
the computer device comprises at least one processor configured to execute computer readable instructions contained in a memory;
The query ranking method is,
generating, by the at least one processor, a first query candidate through query reformulation using a statistical machine translation model for the original query;
generating, by the at least one processor, a second query candidate through query reconstruction using a BERT (Bidirectional Encoder Representations from Transformer)-based machine translation model with respect to the original query; and
ranking, by the at least one processor, the first query candidate and the second query candidate using a search-related feature;
A query ranking method comprising According to claim 1,
The step of generating the second query candidate includes:
Generating the second query candidate through the BERT-based machine translation model, which is a machine translation model in which the encoder part of the transformer model is replaced with a BERT model
A query ranking method characterized by According to claim 1,
The query ranking method is,
Storing, by the at least one processor, in a cache the query candidate inferred using the BERT-based machine translation model;
further comprising,
The step of generating the second query candidate includes:
finding and serving the second query candidate corresponding to the original query in the cache;
A query ranking method comprising According to claim 1,
The ranking step is
Ranking the first query candidate and the second query candidate through a ranking model that has learned the feature using a gradient boosting regression tree (GBRT) algorithm
A query ranking method characterized by According to claim 1,
The feature includes at least one of the number of other queries including a query term in relation to the original query, the number of search inflows, the number of clicked documents, the number of exposed documents, the click statistics of the search area compared to the exposed area, and the length of the query 1 feature, the number of other queries including a query term in relation to each query candidate corresponding to the first query candidate and the second query candidate, the number of search inflows, the number of clicked documents, the number of exposed documents, and the search compared to the exposure area including a second feature comprising at least one of a click statistic of a region, a query length, and a third feature representing a difference between the same feature of the original query and the query candidate pair.
A query ranking method characterized by According to claim 1,
The ranking step is
For each query candidate corresponding to the first query candidate and the second query candidate, the query is made using user feedback information on a document included in the search result of the original query, which is at least a part of the documents included in the search result of the query candidate. calculating the search relevance score of the candidate
A query ranking method comprising 7. The method of claim 6,
The ranking step is
classifying a query candidate whose search relevance score is smaller than the original query among the first and second query candidates as an inappropriate query, and classifies a query candidate whose search relevance score is greater than the original query as a suitable query;
A query ranking method further comprising a. According to claim 1,
The ranking step is
Acquiring a feature collection generated from a search-related log using a search engine
A query ranking method comprising According to claim 1,
The ranking step is
Tuning the ranked query candidates based on word embeddings of the remaining terms after removing the common term with the original query
A query ranking method comprising According to claim 1,
The ranking step is
Tuning the ranked query candidates based on at least one of search-related statistical information, a word length or number of word terms, and a grapheme unit editing distance;
A query ranking method comprising A computer-readable recording medium having recorded therein a program for executing the query ranking method of any one of claims 1 to 10 on a computer. In a computer device,
at least one processor configured to execute computer readable instructions contained in memory
including,
the at least one processor,
For the original query, a first query candidate is generated through query reconstruction using a statistical machine translation model,
For the original query, a second query candidate is generated through query reconstruction using a BERT-based machine translation model,
ranking the first query candidate and the second query candidate using a search-related feature;
A computer device characterized by a. 13. The method of claim 12,
the at least one processor,
Generating the second query candidate through the BERT-based machine translation model, which is a machine translation model in which the encoder part of the transformer model is replaced with a BERT model
A computer device characterized by a. 13. The method of claim 12,
the at least one processor,
Storing the query candidates inferred using the BERT-based machine translation model in the cache,
Finding and serving the second query candidate corresponding to the original query in the cache
A computer device characterized by a. 13. The method of claim 12,
the at least one processor,
Ranking the first query candidate and the second query candidate through a ranking model that has learned the feature using the GBRT algorithm
A computer device characterized by a. 13. The method of claim 12,
The feature includes at least one of the number of other queries including a query term in relation to the original query, the number of search inflows, the number of clicked documents, the number of exposed documents, the click statistics of the search area compared to the exposed area, and the length of the query 1 feature, the number of other queries including a query term in relation to each query candidate corresponding to the first query candidate and the second query candidate, the number of search inflows, the number of clicked documents, the number of exposed documents, and the search compared to the exposure area including a second feature comprising at least one of a click statistic of a region, a query length, and a third feature representing a difference between the same feature of the original query and the query candidate pair.
A computer device characterized by a. 13. The method of claim 12,
the at least one processor,
For each query candidate corresponding to the first query candidate and the second query candidate, the query is made using user feedback information on a document included in the search result of the original query, which is at least a part of the documents included in the search result of the query candidate. Calculating a candidate's search relevance score
A computer device characterized by a. 18. The method of claim 17,
the at least one processor,
Classifying a query candidate whose search relevance score is smaller than the original query among the first and second query candidates as an inappropriate query and a query candidate whose search relevance score is greater than the original query as a suitable query.
A computer device characterized by a. 13. The method of claim 12,
the at least one processor,
Acquiring a feature collection generated from a log related to a search using a search engine
A computer device characterized by a. 13. The method of claim 12,
the at least one processor,
Tuning the ranked query candidates based on at least one of word embeddings of the remaining terms after removing the common term with the original query, search-related statistical information, word length or number of word terms, and grapheme unit editing distance
A computer device characterized by a.

Images