KR102541806B1 - Method, system, and computer readable record medium for ranking reformulated query - Google Patents

Abstract

A method, system, and computer readable recording medium for ranking a reconstructed query are disclosed. The query ranking method may include generating, by the at least one processor, a first query candidate through query reformulation using a statistical machine translation model for an original query; generating, by the at least one processor, a second query candidate through query reconstruction using a BERT (Bidirectional Encoder Representations from Transformer) based machine translation model with respect to the original query; and ranking, by the at least one processor, the first query candidate and the second query candidate using a feature related to a search.

Description

Method, system, and computer readable recording medium for ranking restructured queries

The description below relates to a query reformulation (referred to as 'QR') technique for providing optimal search results.

A user may perform a search to obtain desired information through a site such as a search engine. A user may obtain desired information by inputting a query into a query input window of a search engine through a user terminal and checking a search result output for the query.

In a search system based on query-term matching, a query reconstruction technique capable of providing optimal search results by adding reformulated queries having similar meanings to a user's query is used.

For example, Korean Patent Publication No. 10-2011-0007743 (published on January 25, 2011) describes a technique for correcting a user query determined as a misspelled query based on statistical data according to the entire query unit or word unit. This is disclosed.

We provide a method and system capable of generating a QR candidate for a user query by additionally using a BERT (Bidirectional Encoder Representations from Transformer) model along with a statistical machine translation (SMT) model.

In order to find a QR suitable for search, we provide a method and system that can rank QR candidates generated from a machine translation model in which the SMT model and the BERT model are combined based on features related to search.

A query ranking method executed on a computer device, the computer device comprising at least one processor configured to execute computer readable instructions contained in a memory, the query ranking method comprising: by the at least one processor , generating a first query candidate through query reformulation using a statistical machine translation model for the original query; generating, by the at least one processor, a second query candidate through query reconstruction using a BERT (Bidirectional Encoder Representations from Transformer) based machine translation model with respect to the original query; and ranking, by the at least one processor, the first query candidate and the second query candidate using a feature related to search.

According to one aspect, the generating of the second query candidate may include generating the second query candidate through the BERT-based machine translation model, which is a machine translation model obtained by replacing an encoder part of a transformer model with a BERT model. can

According to another aspect, the query ranking method further comprises storing, by the at least one processor, a query candidate referenced using the BERT-based machine translation model in a cache, wherein the The generating of the second query candidate may include finding and serving the second query candidate corresponding to the original query in the cache.

According to another aspect, the ranking may include ranking the first query candidate and the second query candidate through a ranking model obtained by learning the feature using a gradient boosting regression tree (GBRT) algorithm.

According to another aspect, the feature may include the number of other queries including the query term in relation to the original query, the number of search inflows, the number of clicked documents, the number of exposed documents, the click statistic of the search area versus the surface area, and the query length. The first feature including at least one, the number of other queries including the query term in relation to each query candidate corresponding to the first query candidate and the second query candidate, the number of search inflows, the number of clicked documents, and the exposed documents A second feature including at least one of a number of clicks, a click statistic of a search region versus an exposure region, and a query length, and a third feature representing a difference between identical features of the original query and the query candidate pair.

According to another aspect, the step of ranking may include, for each query candidate corresponding to the first query candidate and the second query candidate, the search result of the original query, which is at least a part of documents included in the search result of the query candidate. The method may include calculating a search relevance score of the query candidate by using user feedback information on the included document.

According to another aspect, the ranking may include classifying a query candidate having a lower search relevance score than the original query among the first query candidate and the second query candidate as an unsuitable query, and The method may further include classifying query candidates larger than the original query as suitable queries.

According to another aspect, the ranking may include acquiring a feature collection generated from a log related to a search by using a search engine.

According to another aspect, the ranking may include tuning the ranked query candidates based on word embeddings of terms remaining after removing common terms with the original query.

According to another aspect, the ranking may include the ranking based on at least one of (1) statistical information related to search, (2) word length or number of word terms, and (3) grapheme unit editing distance. It may include tuning the query candidates.

A computer-readable recording medium having a program recorded thereon for executing the query ranking method on a computer is provided.

A computer device comprising at least one processor configured to execute computer readable instructions included in a memory, wherein the at least one processor performs query reconstruction using a statistical machine translation model on an original query as a first query candidate. generating a second query candidate through query reconstruction using a BERT-based machine translation model for the original query, and ranking the first query candidate and the second query candidate using features related to search. It provides a computer device characterized in that.

According to embodiments of the present invention, QR quality and search quality can be improved by generating a QR candidate for a user query by additionally using the BERT model together with the SMT model.

According to embodiments of the present invention, a QR suitable for a search can be efficiently found by ranking the QR candidates generated from the machine translation model in which the SMT model and the BERT model are combined based on features related to the search.

1 is a diagram illustrating an example of a network environment according to an embodiment of the present invention.
2 is a block diagram illustrating an example of a computer device according to one embodiment of the present invention.
3 is a flowchart illustrating an example of a method that can be performed by a computer device according to an embodiment of the present invention.
4 illustrates an example of a multi-QR generation model for QR generation in one embodiment of the present invention.
5 to 9 show examples of SMT models for QR generation in one embodiment of the present invention.
10 to 11 show examples of BERT-based machine translation models for QR generation in one embodiment of the present invention.
12 shows an example of a BERT serving system for real-time service in one embodiment of the present invention.
13 illustrates an example of a structure of a cache-based BERT serving system according to an embodiment of the present invention.
14 illustrates an example of a ranking configuration for QR candidate verification in one embodiment of the present invention.
15 illustrates an example of an unsuitable QR candidate and a suitable QR candidate according to an embodiment of the present invention.
16 illustrates an example of a process of obtaining features necessary for ranking QR candidates according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

A query ranking system according to embodiments of the present invention may be implemented by at least one computer device, and a query ranking method according to embodiments of the present invention may be implemented by at least one computer device included in the query ranking system. can be performed through At this time, a computer program according to an embodiment of the present invention may be installed and driven in the computer device, and the computer device may perform the query ranking method according to the embodiments of the present invention under the control of the driven computer program. there is. The above-described computer program may be combined with a computer device and stored in a computer readable recording medium to execute a query ranking method on a computer.

1 is a diagram illustrating an example of a network environment according to an embodiment of the present invention. The network environment of FIG. 1 shows an example including a plurality of electronic devices 110 , 120 , 130 , and 140 , a plurality of servers 150 and 160 , and a network 170 . 1 is an example for explanation of the invention, and the number of electronic devices or servers is not limited as shown in FIG. 1 . In addition, the network environment of FIG. 1 only describes one example of environments applicable to the present embodiments, and the environment applicable to the present embodiments is not limited to the network environment of FIG. 1 .

The plurality of electronic devices 110, 120, 130, and 140 may be fixed terminals implemented as computer devices or mobile terminals. Examples of the plurality of electronic devices 110, 120, 130, and 140 include a smart phone, a mobile phone, a navigation device, a computer, a laptop computer, a digital broadcast terminal, a personal digital assistant (PDA), and a portable multimedia player (PMP). ), and tablet PCs. As an example, FIG. 1 shows the shape of a smartphone as an example of the electronic device 110, but in the embodiments of the present invention, the electronic device 110 substantially uses a wireless or wired communication method to transmit other information via the network 170. It may refer to one of various physical computer devices capable of communicating with the electronic devices 120 , 130 , and 140 and/or the servers 150 and 160 .

The communication method is not limited, and short-distance wireless communication between devices as well as a communication method utilizing a communication network (eg, a mobile communication network, a wired Internet, a wireless Internet, and a broadcasting network) that the network 170 may include may also be included. For example, the network 170 may include a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), and a broadband network (BBN). , one or more arbitrary networks such as the Internet. In addition, the network 170 may include any one or more of network topologies including a bus network, a star network, a ring network, a mesh network, a star-bus network, a tree or a hierarchical network, and the like. Not limited.

Each of the servers 150 and 160 communicates with the plurality of electronic devices 110, 120, 130, and 140 through the network 170 to provide commands, codes, files, contents, services, and the like, or a computer device or a plurality of computers. It can be implemented in devices. For example, the server 150 may be a system that provides a desired service (eg, a search service) to the plurality of electronic devices 110, 120, 130, and 140 accessed through the network 170.

2 is a block diagram illustrating an example of a computer device according to one embodiment of the present invention. Each of the plurality of electronic devices 110 , 120 , 130 , and 140 or each of the servers 150 and 160 described above may be implemented by the computer device 200 shown in FIG. 2 .

As shown in FIG. 2 , the computer device 200 may include a memory 210, a processor 220, a communication interface 230, and an input/output interface 240. The memory 210 is a computer-readable recording medium and may include a random access memory (RAM), a read only memory (ROM), and a permanent mass storage device such as a disk drive. Here, a non-perishable mass storage device such as a ROM and a disk drive may be included in the computer device 200 as a separate permanent storage device distinct from the memory 210 . Also, an operating system and at least one program code may be stored in the memory 210 . These software components may be loaded into the memory 210 from a computer-readable recording medium separate from the memory 210 . The separate computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD/CD-ROM drive, and a memory card. In another embodiment, software components may be loaded into the memory 210 through the communication interface 230 rather than a computer-readable recording medium. For example, software components may be loaded into memory 210 of computer device 200 based on a computer program installed by files received over network 170 .

The processor 220 may be configured to process commands of a computer program by performing basic arithmetic, logic, and input/output operations. Instructions may be provided to processor 220 by memory 210 or communication interface 230 . For example, processor 220 may be configured to execute received instructions according to program codes stored in a recording device such as memory 210 .

The communication interface 230 may provide a function for the computer device 200 to communicate with other devices (eg, storage devices described above) through the network 170 . For example, a request, command, data, file, etc. generated according to a program code stored in a recording device such as the memory 210 by the processor 220 of the computer device 200 is controlled by the communication interface 230 to the network ( 170) to other devices. Conversely, signals, commands, data, files, etc. from other devices may be received by the computer device 200 through the communication interface 230 of the computer device 200 via the network 170 . Signals, commands, data, etc. received through the communication interface 230 may be transferred to the processor 220 or the memory 210, and files, etc. may be stored as storage media that the computer device 200 may further include (described above). permanent storage).

The input/output interface 240 may be a means for interface with the input/output device 250 . For example, the input device may include a device such as a microphone, keyboard, or mouse, and the output device may include a device such as a display or speaker. As another example, the input/output interface 240 may be a means for interface with a device in which functions for input and output are integrated into one, such as a touch screen. The input/output device 250 and the computer device 200 may be configured as one device.

Also, in other embodiments, computer device 200 may include fewer or more elements than those of FIG. 2 . However, there is no need to clearly show most of the prior art components. For example, the computer device 200 may be implemented to include at least some of the aforementioned input/output devices 250 or may further include other components such as a transceiver and a database.

Hereinafter, specific embodiments of a method and system for ranking reconstructed queries will be described.

The present embodiments relate to QR technology for application to a search service based on query-term matching. It is to view QR as a translation problem and convert it into a query suitable for search using a machine translation model. In particular, in this embodiment, after generating a reconstructed query (hereinafter referred to as a 'QR candidate') for a user query through a translation model in which an SMT model and a BERT model are combined, information on whether the query is suitable for search is reflected. For this purpose, ranking of QR candidates may be performed.

3 is a flowchart illustrating an example of a method that a computer device may perform according to an embodiment of the present invention.

The computer device 200 according to the present embodiment may provide a search service to a client through a dedicated application installed on the client or access to a web/mobile site related to the computer device 200 . A computer-implemented query ranking system may be configured in the computer device 200 . For example, the query ranking system may be implemented in the form of a program that operates independently, or may be implemented in the form of an in-app of a specific application to be operated on the specific application.

The processor 220 of the computer device 200 may include at least one or more components as components for performing the query ranking method according to FIG. 3 . Depending on embodiments, components of the processor 220 may be selectively included in or excluded from the processor 220 . Also, components of the processor 220 may be separated or merged to express functions of the processor 220 according to embodiments.

The processor 220 and components of the processor 220 may control the computer device 200 to perform steps S310 to S350 included in the query ranking method of FIG. 3 . For example, the processor 220 and components of the processor 220 may be implemented to execute instructions according to an operating system code and at least one program code included in the memory 210 .

Here, elements of the processor 220 may be expressions of different functions performed by the processor 220 according to instructions provided by program codes stored in the computer device 200 .

The processor 220 may read necessary commands from the memory 210 in which commands related to controlling the computer device 200 are loaded. In this case, the read command may include a command for controlling the processor 220 to execute steps S310 to S350 to be described later.

Steps (S310 to S350) to be described later may be performed in an order different from the order shown in FIG. 3, and some of the steps (S310 to S350) may be omitted or additional processes may be further included.

Referring to FIG. 3 , in step S310, the processor 220 may generate a first QR candidate group by reconstructing a user query using an SMT model, which is a machine translation model. The processor 220 may generate a first QR candidate group through the SMT model learned with the parallel corpus learning data.

In step S320, the processor 220 may generate a second QR candidate group by reconstructing a user query using a BERT-based machine translation model. In the case of the SMT model, while it can generate many QR candidates at a fast translation speed, it has a disadvantage in that it lacks translation considering the context. To compensate for this, the BERT model is applied to QR. Unlike SMT, the BERT model, which is a classification model, understands the meaning of words and enables natural translation reflecting the context. The BERT model can be learned with a pre-training technique using a masked language model (MLM) and next sentence prediction (NSP) for deeper language understanding.

In step S330, the processor 220 may merge the first QR candidate group generated from the SMT model and the second QR candidate group generated from the BERT-based machine translation model, and rank them using features related to search. The QR using the machine translation model only reflects the linguistic fluency, and information on whether the QR is suitable for search is not reflected. Since many QR candidates are being generated using the SMT model and the BERT-based machine translation model, and even if the translation is linguistically well, it may not be a query suitable for search, so QR verification through ranking is necessary. The processor 220 may score the first QR candidate group and the second QR candidate group by utilizing user feedback information from the search service. For example, the processor 220 may rank the first QR candidate group and the second QR candidate group through a learning model using a gradient boosting regression tree (GBRT) algorithm.

In step S340, the processor 220 is a tuning process for improving the quality of the ranking results. In step S330, the processor 220 may remove some QR candidates through tuning using each query feature for the QR candidate group ranked in step S330. The processor 220 may perform tuning using a query feature in order to remove low-quality cases even in a candidate ranked high in the QR candidate group.

For example, the processor 220 may remove QR candidates that are not suitable for search by using word embedding similarity from which common terms are removed. For example, for the user query 'phone xs size' and the QR candidate 'phone xr size', the word embedding similarity is compared between the terms other than the common terms 'phone' and 'size', and the similarity is higher than the reference value (threshold). If it is low, the QR candidate 'phone xr size' can be removed.

As another example, the processor 220 may remove QR candidates unsuitable for search based on statistical information related to search. In this case, the statistical information may include a query count (QC) rate, an exposure rate, a click rate, and the like. For example, for the user query 'phone xs size' and the QR candidate 'phone xr size', if the QC, exposure, and clicks of 'phone xr size' are all lower than the reference value, the QR candidate 'phone xr size' can be removed. can

As another example, the processor 220 may remove QR candidates that are not suitable for search through a combination with the above tuning criterion (eg, word embedding similarity, statistical information) based on the word length or the number of word terms. there is. For example, for the user query 'Top' and the QR candidate 'Top', since the ambiguity is very high in the case of 1 syllable, the two queries are compared by combining the QC ratio or the word embedding rule, and the QR candidate 'QR candidate' according to the comparison result. The top can be removed.

As another example, the processor 220 may remove QR candidates unsuitable for search through a combination with the above-mentioned tuning criterion (eg, word embedding similarity, statistical information) based on the grapheme unit editing distance. For example, for the user query 'convenience store' and the QR candidate 'convenience sleep', if the grapheme unit editing distance between queries is 1, it is highly likely that the query is a typo. According to the comparison result, the QR candidate 'convenience sleep' can be removed.

In step S350, the processor 220 may select at least some of the QR candidates remaining after tuning as a search query in order to provide a search result corresponding to the user query. For example, the processor 220 may use QR candidates of a certain number or a certain ratio of upper ranks based on the QR ranking as an actual search query. In other words, a search result corresponding to a user query may include documents matching each QR candidate selected as a search query.

As shown in FIG. 4 , the processor 220 may include an SMT model 410 and a BERT-based machine translation model 420 as a multi-model structure. The processor 220 may generate the first QR candidate group 401 through reconstruction using the SMT model 410 for the user query 40, and also generate the first QR candidate group 401 through reconstruction using the BERT-based machine translation model 420. 2 QR candidates 402 can be generated.

The structure of the SMT model 410 is shown in FIG. 5 .

Referring to FIG. 5 , the SMT model 410 provides a translation result with a statistical probability, and includes a translation model 501 and a language model 502.

The translation model 501 converts a source language into a target language, and requires parallel corpus learning data consisting of pairs of source sentences and target sentences.

The translation model 501 is composed of an alignment model and a lexical translation model, and the alignment 601 can be extracted as shown in FIG. 6 .

The language model 502 predicts the probability of a sentence, and requires training data for a target language. Referring to FIG. 7 , when consecutive words 'where are we going' are given, the probability that 'are' comes after 'where' and the probability that 'we' comes after 'where are' is calculated.

The SMT model 410 can create statistical translation results by combining the translation model 501 and the language model 502.

The QR system to which the above-described SMT model 410 is applied is shown in FIG. 8 .

A translation model 501 composed of an alignment model and a vocabulary translation model and a language model 502 may be learned from the parallel corpus dictionary.

Referring to FIG. 8, the QR system to which the SMT model 410 is applied goes through a tokenization process (morphological tokenization), a decoding process (SMT decoder), a result generation process (N best QR result), and a heuristic logic. A reconstructed query, that is, the first QR candidate group 401 may be generated by converting a source query, which is an original query, into a target query.

For example, as shown in FIG. 9, in the QR system to which the SMT model 410 is applied, when 'Naver free font' is given as the user query 901, which is the original query, 'Naver', 'free', 'font' After tokenization as in, 'Naver free fonts', 'Naver free fonts', and 'Naver free fonts' can be generated as the QR candidates 902 for the user query 901 through a decoding process and a result generation process. .

The QR system to which the SMT model 410 is applied has a limitation in depending on the language model 502 for context, and thus lacks translation considering context compared to deep learning.

In this embodiment, a QR system using a BERT-based machine translation model 420 may be added to improve the quality limit of the SMT model 410.

10 shows the BERT model structure.

Referring to FIG. 10, the BERT model 1000 receives a given sentence as an input, first embeds it in token units through an input embedding layer 1001, and then converts the token through a transformer layer 1002. It is composed of a contextual representation structure by encoding considering the location information of .

The transformer layer 1002 is composed of a self-attention model instead of a model such as CNN or RNN, and the BERT model 1000 embeds a language using an encoder among encoders and decoders of a transformer.

By replacing the encoder part of the transformer model with the BERT model 1000, a BERT-based machine translation model 420 for application to the QR system can be constructed.

The QR system to which the BERT-based machine translation model 420 is applied is shown in FIG. 11.

The BERT model, a classification model, shows very high performance when applied to various language processing or information retrieval fields. A QR system for generating a QR candidate through a translation model, as shown in FIG. 11, may consist of an encoder 1110 for understanding an input query as a transformer model and a decoder 1120 for generating a QR candidate. The encoder 1110 part can be replaced with the BERT model 1000.

The encoder 1110 can deeply understand the user query 1101, which is the original query, by using the BERT model 1000, and can understand the meaning of words and perform natural translation reflecting the context. The decoder 1120 may generate a QR candidate group 1102 for a user query 1101 as a decoder of a transformer model.

For example, as shown in FIG. 11, the QR system to which the BERT-based machine translation model 420 is applied is given a 'teenage female music ranking' as a user query 1101, a deeper search using the BERT model 1000. Through language understanding, 'teenage female music ranking', 'teenage female song ranking', 'teenage female music ranking', and 'teenage female music ranking' are generated as QR candidates 1102 for the user query 1101. can do.

Accordingly, the processor 220 may generate a QR candidate for providing an optimal search result by using the BERT-based machine translation model 420 together with the SMT model 410 as a multi-QR generation model.

In the case of the BERT-based machine translation model 420, it is difficult to guarantee performance in a real-time service because the model size is large.

Referring to FIG. 12 , the processor 220 may further include a cache-based BERT serving system 1230 to overcome a response speed problem of the BERT-based machine translation model 420 . The BERT serving system 1230 may apply a method of storing the inferred QR result of the BERT-based machine translation model 420 in a cache.

In general, since QR is performed prior to searching for each collection when a query is received in integrated search, the response speed of integrated search slows down as much as the response speed of QR slows down. To solve this problem, a cache server can be added to store and use the referenced QR result of the BERT-based machine translation model 420 in the cache server.

13 shows an example of a BERT serving system structure in one embodiment of the present invention.

Referring to FIG. 13 , a BERT serving system 1230 may include a cache server 1301, a Neural Machine Translation (NMT) producer 1302, a queue 1303, and an NMT consumer 1304.

When a user query is input, the NMT producer 1302 refers to the cache server 1301 to find a QR candidate corresponding to the user query and returns the result.

Meanwhile, the NMT producer 1302 inserts the user query into the queue 1303 when there is no QR result corresponding to the user query in the cache server 1301. When the same query is continuously input, an empty value is inserted into the cache server 1301 because duplicate queries may enter the queue 1303 . The empty value inserted into the cache server 1301 may be temporarily maintained for a certain period of time (eg, 1 hour).

The queue 1303 serves as a message queue like Kafka, and can maintain queries received from the NMT producer 1302 and deliver them to the NMT consumer 1304.

The NMT consumer 1304 may receive a query list from the queue 1303 and perform an inference on a QR candidate using a BERT-based machine translation model 420. The NMT consumer 1304 inserts the inference result for the QR candidate of the BERT-based machine translation model 420 into the cache server 1301. QR results inferred on the cache server 1301 may be updated in units of a certain time (eg, one day).

The processor 220 may calculate the number of serving equipment required to process the accumulated queries in real time. The processor 220 may determine the number of devices constituting the BERT serving system 1230 so that queries entered in real time can be processed within a unit of time (eg, 1 second).

For example, the processor 220 may calculate the number of serving devices based on the maximum number of inferences per unit time and the maximum unique query per second (UQPS). It is more advantageous to process n queries in a batch than to perform n inferences for one query. For example, to process UQPS 1,500 with a throughput of 1 second (16 queries), approximately 94 (1500/16 = 93.75) model server containers are required.

Therefore, in this embodiment, real-time service can be guaranteed by using a cache-based serving platform to apply the BERT model, which has a large model size and slow response speed, to the QR system.

In addition to using a cache-based serving platform, it is also possible to optimize performance by reducing the parameters of the BERT model to three layers and applying a knowledge distillation technique for model compression.

The QR verification process through ranking is as follows.

Referring to FIG. 14 , a processor 220 uses a QR ranking model 1460 for ranking a first QR candidate group 401 and a second QR candidate group 402 for QR verification and a feature related to a search by QR ranking. It may further include a feature server 1470 for delivery to model 1460.

The processor 220 determines the first QR candidate group and the second QR candidate group through the QR ranking model 1460 learned using the learning target data set (hereinafter referred to as 'learning data') 1450. can be ranked in order suitable for search.

First, the processor 220 may generate click (or exposure) graph-based learning data 1450 for learning the QR ranking model 1460 . The QR ranking model 1460 should be trained to rank QR candidates in an effective order for actual search. For example, the processor 220 may generate a click graph for QR candidates generated by the SMT model 410 and the BERT-based machine translation model 420, and then extract and score QR candidates suitable for search from the click graph. . The processor 220 may generate training data 1450 as a click graph and learn the QR ranking model 1460 with the generated training data 1450 .

The processor 220 does not find a QR to be used as a search query based on the translation score through the SMT model 410 and the BERT-based machine translation model 420, but searches using both features related to the search and the translation score. The QR ranking model 1460 can be learned as a problem of finding a suitable QR.

The processor 220 merges the first QR candidate group 401 generated from the SMT model 410 and the second QR candidate group 402 generated from the BERT-based machine translation model 420 to obtain a QR ranking model 1460. Ranking may be performed on all QR candidates. At this time, the processor 220 may secure several features that are helpful in the search through the feature server 1470, and based on these features, the search is performed through the QR ranking model 1460 learned with the learning data 1450. Distinguish between suitable QRs and non-conforming QRs.

The processor 220 may score the suitability between the original query and the QR candidate using actual user click information in the search service and use it as correct answer data for learning the QR ranking model 1460 .

The fitness score of the QR candidate for the original query may be defined as Equation 1.

[Equation 1]

Here, rank(di) means the ranking of document di included in the search result corresponding to the QR candidate, and clickCount(di) means the number of clicks on document di included in the search result corresponding to the original query.

In other words, only scores of documents appearing in common with the search result of the original query are reflected in the relevance score of the QR candidate.

Based on the click graph, the processor 220 classifies a QR candidate having a lower score than the original query as a non-conforming QR candidate, and classifies a QR candidate having a higher score than the original query as a suitable QR candidate, based on the click graph.

15 illustrates an example of classification of a suitable QR candidate and an unsuitable QR candidate.

For example, although QR candidate 1 ranked documents with a significant number of clicks, it cannot be guaranteed that these documents are related to the original query. As shown in FIG. 15 , assuming that among the documents included in the search result of QR candidate 1, Document _k and Document _m do not appear in the search result of the original query, the corresponding document is not reflected in the relevance score of QR candidate 1. Although Document ₃ with 100 clicks was ranked as the document that appeared in the search results of the original query, if the relevance score of QR candidate 1 is lower than that of the original query, QR candidate 1 is classified as a non-conforming QR candidate.

As shown in FIG. 15, assuming that all documents included in the search results of QR candidate 2 appear in the search results of the original query and are exposed in the order of the number of user clicks, the relevance score of QR candidate 2 is the original If it is greater than the relevance score of the query, QR candidate 2 is classified as a suitable QR candidate.

The processor 220 determines whether the QR candidates are suitable through the QR ranking model 1460 and ranks them in an order suitable for search, that is, according to the suitability score. In this case, features capable of improving search quality may be acquired and learned using the GBRT algorithm.

Features related to search may be configured as shown in Table 1.

Feature Group Feature original query feature f1: search QC f2 : the number of other queries that contain the search query term f3: The number of search inflows, such as keyboard input, real-time rapidly rising search word clicks, topic keyword clicks, etc. f4 : number of documents clicked on integrated search f5 : Number of clicks on integrated search impressions f6: Relative click statistics in the content search area compared to other search surface areas such as blogs, web, and cafes f7: language model score f8 : query length f9: the number of query terms in morpheme units QR candidate features f11: Search QC f12: number of other queries that contain the search query term f13: The number of search inflows, such as keyboard input, real-time rapidly rising search word clicks, and topic keyword clicks f14: number of documents clicked on integrated search f15: number of clicks on integrated search f16: Relative click statistics in the content search area compared to other search surface areas such as blogs, web, and cafes f17: language model score f18: query length f19: number of query terms in morpheme units Query pair features f21 : f1 - f11 f22 : f2 - f12 f23: Number of common terms between original query and QR candidate f24 : f4 - f14 f25 : f5 - f15 f26: Cosine similarity between original query and QR candidate f27 : f7 - f17 f28 : f8 - f18 f29 : f9 - f 19 f30: Cosine similarity between original query and QR candidate excluding common occurrence terms f31: SMT translation score f32: BERT translation score f33: SMT translation ranking f34: BERT translation ranking

In addition, in this embodiment, a feature server 1470 configured as a search engine may be used to quickly transfer a feature related to a search for QR candidates to the QR ranking model 1460.

For example, as shown in FIG. 16, a feature server 1470 connects a search application server (SAS) 1610 and a search engine server 1610 to transfer features necessary for ranking QR candidates to a QR ranking model 1460. engine server) (1620).

The feature server 1470 may include a refining system 1630. At this time, the refining system 1630 processes the log 1601 related to search, such as an impression log and a click log for a recent period, to perform QC and click operations. You can create and index feature collections 1602 such as count, number of exposed documents, number of sm parameters, etc.

Feature server 1470 can provide features from queries that include QR candidate strings along with indexed feature collection 1602 . In other words, by using the SAS 1610 and the search engine server 1620, it is possible to check features of other QR candidate strings including the QR candidate string.

In addition, the feature server 1470 word-embeds the log 1601 related to the search and/or the document 1603 corresponding to each query as a feature necessary to calculate the similarity between the original query and the QR candidate, and uses the QR ranking model 1460 ), and the language model value of the query (ie, the probability value of the sentence) can also be provided together.

Therefore, in this embodiment, in order to select a query more suitable for search among numerous QR candidates generated through a multi-model structure including the SMT model 410 and the BERT-based machine translation model 420, features related to search are used. Corresponding QR candidates may be verified and ranked.

As described above, according to embodiments of the present invention, a QR suitable for a search can be efficiently found by ranking the QR candidates generated from the machine translation model in which the SMT model and the BERT model are combined based on features related to the search.

The devices described above may be implemented as hardware components, software components, and/or a combination of hardware components and software components. For example, devices and components described in the embodiments include a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), and a programmable PLU (programmable logic unit). logic unit), microprocessor, or any other device capable of executing and responding to instructions. The processing device may run an operating system (OS) and one or more software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of software. For convenience of understanding, there are cases in which one processing device is used, but those skilled in the art will understand that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it can include. For example, a processing device may include a plurality of processors or a processor and a controller. Other processing configurations are also possible, such as parallel processors.

Software may include a computer program, code, instructions, or a combination of one or more of the foregoing, which configures a processing device to operate as desired or processes independently or collectively. The device can be commanded. The software and/or data may be embodied in any tangible machine, component, physical device, computer storage medium or device to be interpreted by or to provide instructions or data to a processing device. there is. Software may be distributed on networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer readable media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. In this case, the medium may continuously store a program executable by a computer or temporarily store the program for execution or download. In addition, the medium may be various recording means or storage means in the form of a single or combined hardware, but is not limited to a medium directly connected to a certain computer system, and may be distributed on a network. Examples of the medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROM and DVD, magneto-optical media such as floptical disks, and ROM, RAM, flash memory, etc. configured to store program instructions. In addition, examples of other media include recording media or storage media managed by an app store that distributes applications, a site that supplies or distributes various other software, and a server.

As described above, although the embodiments have been described with limited examples and drawings, those skilled in the art can make various modifications and variations from the above description. For example, the described techniques may be performed in an order different from the method described, and/or components of the described system, structure, device, circuit, etc. may be combined or combined in a different form than the method described, or other components may be used. Or even if it is replaced or substituted by equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents of the claims are within the scope of the following claims.

Claims (20)

A query ranking method executed on a computer device,
The computer device includes at least one processor configured to execute computer readable instructions contained in a memory;
The query ranking method,
generating, by the at least one processor, a first query candidate through query reformulation using a statistical machine translation model for an original query;
generating, by the at least one processor, a second query candidate through query reconstruction using a BERT (Bidirectional Encoder Representations from Transformer) based machine translation model with respect to the original query; and
Ranking, by the at least one processor, the first query candidate and the second query candidate using a feature related to a search.
including,
The features include a first feature related to searching the original query, a second feature related to searching each query candidate corresponding to at least one of the first query candidate and the second query candidate, and the original query and the query candidate. Including a third feature that represents a difference between identical features in a pair
A query ranking method characterized by. According to claim 1,
Generating the second query candidate,
Generating the second query candidate through the BERT-based machine translation model, which is a machine translation model in which an encoder part of a transformer model is replaced with a BERT model
Characterized by a query ranking method. According to claim 1,
The query ranking method,
Storing, by the at least one processor, a query candidate referenced using the BERT-based machine translation model in a cache.
Including more,
Generating the second query candidate,
Finding and serving the second query candidate corresponding to the original query in the cache
A query ranking method comprising a. According to claim 1,
The ranking step is
Ranking the first query candidate and the second query candidate through a ranking model obtained by learning the feature using a gradient boosting regression tree (GBRT) algorithm.
Characterized by a query ranking method. According to claim 1,
The first feature includes at least one of the number of other queries including the query term in relation to the original query, the number of search inflows, the number of clicked documents, the number of exposed documents, the click statistic of the search area versus the exposed area, and the query length. do,
The second feature includes at least one of the number of other queries including the query term in relation to the query candidate, the number of search inflows, the number of clicked documents, the number of exposed documents, the click statistic of the search area versus the exposed area, and the query length. to do
Characterized by a query ranking method. According to claim 1,
The ranking step is
For each query candidate corresponding to the first query candidate and the second query candidate, user feedback information about a document included in the search result of the original query, which is at least a part of documents included in the search result of the query candidate, is used to perform the query. Calculating the candidate's search relevance score
A query ranking method comprising a. According to claim 6,
The ranking step is
Classifying a query candidate having a lower search relevance score than the original query among the first query candidate and the second query candidate as an unsuitable query, and classifying a query candidate having a higher search relevance score than the original query as a suitable query.
A query ranking method further comprising a. According to claim 1,
The ranking step is
Obtaining a feature collection generated from a log related to a search using a search engine
A query ranking method comprising a. According to claim 1,
The ranking step is
Tuning the ranked query candidates based on word embeddings of terms remaining after removing common terms with the original query.
A query ranking method comprising a. According to claim 1,
The ranking step is
Tuning the ranked query candidates based on at least one of (1) statistical information related to search, (2) word length or number of word terms, and (3) grapheme unit editing distance.
A query ranking method comprising a. A computer-readable recording medium in which a program for executing the query ranking method according to any one of claims 1 to 10 in a computer is recorded. In a computer device,
at least one processor configured to execute computer readable instructions contained in memory;
including,
The at least one processor,
Generating a first query candidate through query reconstruction using a statistical machine translation model for the original query;
For the original query, a second query candidate is generated through query reconstruction using a BERT-based machine translation model;
ranking the first query candidate and the second query candidate using a feature related to a search;
The features include a first feature related to searching the original query, a second feature related to searching each query candidate corresponding to at least one of the first query candidate and the second query candidate, and the original query and the query candidate. Including a third feature that represents a difference between identical features in a pair
Characterized by a computer device. According to claim 12,
The at least one processor,
Generating the second query candidate through the BERT-based machine translation model, which is a machine translation model in which the encoder part of the transformer model is replaced with the BERT model
Characterized by a computer device. According to claim 12,
The at least one processor,
Store a query candidate referenced using the BERT-based machine translation model in a cache;
Finding and serving the second query candidate corresponding to the original query in the cache
Characterized by a computer device. According to claim 12,
The at least one processor,
Ranking the first query candidate and the second query candidate through a ranking model learned from the feature using a GBRT algorithm
Characterized by a computer device. According to claim 12,
The first feature includes at least one of the number of other queries including the query term in relation to the original query, the number of search inflows, the number of clicked documents, the number of exposed documents, the click statistic of the search area versus the exposed area, and the query length. do,
The second feature includes at least one of the number of other queries including the query term in relation to the query candidate, the number of search inflows, the number of clicked documents, the number of exposed documents, the click statistic of the search area versus the exposed area, and the query length. to do
Characterized by a computer device. According to claim 12,
The at least one processor,
For each query candidate corresponding to the first query candidate and the second query candidate, user feedback information about a document included in the search result of the original query, which is at least a part of documents included in the search result of the query candidate, is used to perform the query. Calculating a search relevance score of a candidate
Characterized by a computer device. According to claim 17,
The at least one processor,
Classifying a query candidate having a lower search relevance score than the original query among the first query candidate and the second query candidate as an unsuitable query, and classifying a query candidate having a higher search relevance score than the original query as a suitable query.
Characterized by a computer device. According to claim 12,
The at least one processor,
Obtaining, using a search engine, a collection of features created from logs related to searches.
Characterized by a computer device. According to claim 12,
The at least one processor,
Based on at least one of (1) word embedding of terms remaining after removing common terms with the original query, (2) statistical information related to search, (3) word length or number of word terms, and (4) grapheme unit editing distance tuning the ranked query candidates by
Characterized by a computer device.

Images