WO2017198022A1 - 基于Lucence自定义词库的全文检索匹配方法及系统 - Google Patents

基于Lucence自定义词库的全文检索匹配方法及系统 Download PDF

Info

Publication number
WO2017198022A1
WO2017198022A1 PCT/CN2017/080784 CN2017080784W WO2017198022A1 WO 2017198022 A1 WO2017198022 A1 WO 2017198022A1 CN 2017080784 W CN2017080784 W CN 2017080784W WO 2017198022 A1 WO2017198022 A1 WO 2017198022A1
Authority
WO
WIPO (PCT)
Prior art keywords
search
lucence
custom
field
result
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2017/080784
Other languages
English (en)
French (fr)
Inventor
白凡
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Wuhan Douyu Network Technology Co Ltd
Original Assignee
Wuhan Douyu Network Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Wuhan Douyu Network Technology Co Ltd filed Critical Wuhan Douyu Network Technology Co Ltd
Priority to US15/771,057 priority Critical patent/US20180260473A1/en
Publication of WO2017198022A1 publication Critical patent/WO2017198022A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • G06F16/33—Querying
    • G06F16/3331—Query processing
    • G06F16/3332—Query translation
    • G06F16/3334—Selection or weighting of terms from queries, including natural language queries
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • G06F16/33—Querying
    • G06F16/332—Query formulation
    • G06F16/3325—Reformulation based on results of preceding query
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • G06F16/33—Querying
    • G06F16/332—Query formulation
    • G06F16/3325—Reformulation based on results of preceding query
    • G06F16/3326—Reformulation based on results of preceding query using relevance feedback from the user, e.g. relevance feedback on documents, documents sets, document terms or passages
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • G06F16/33—Querying
    • G06F16/3331—Query processing
    • G06F16/3332—Query translation
    • G06F16/3335—Syntactic pre-processing, e.g. stopword elimination, stemming
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • G06F16/33—Querying
    • G06F16/3331—Query processing
    • G06F16/334—Query execution
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • G06F16/35—Clustering; Classification
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • G06F16/36—Creation of semantic tools, e.g. ontology or thesauri
    • G06F16/374—Thesaurus

Definitions

  • the invention relates to the field of big data search, in particular to a full-text search matching method and system based on a Lucence custom vocabulary.
  • Apache Lucence is an open source full-text search engine toolkit, but it is not a complete full-text search engine, but a full-text search engine architecture that provides a complete query engine and indexing engine, a partial text analysis engine.
  • Apache Lucence an open source full-text search project under Apache
  • Full-text search Different from the traditional fuzzy matching, the search words are firstly segmented according to certain rules, then the word segmentation is matched with the source data, and the data is scored according to the number of occurrences of the word segmentation, the distance of the word segmentation, the weight and the like, and the retrieval result is obtained;
  • Participle Full-text search index, for example: I am Chinese, and the participles can be: I, Yes, China, People, Chinese, etc.;
  • Public Thesaurus A thesaurus that stores public participle rules, for example, commonly used: hello, China, etc.;
  • Custom vocabulary a dictionary vocabulary for storing the word segmentation rules you need according to your needs
  • Search feedback search effect feedback, that is, after the user enters the search word into the search page, whether there is a click on the page link or click the link after repeatedly turning the page;
  • Search volume The total search volume of a search term during a certain period of time
  • Field The field to be searched, for example: game name, anchor name, room name, etc.
  • the object of the present invention is to overcome the deficiencies of the above background art, and to provide a full-text search matching method and system based on a Lucence custom vocabulary, which can quickly and efficiently establish a dedicated Lucence custom vocabulary according to a search term input by a user. .
  • the invention provides a full-text search matching method based on a Lucence custom vocabulary, comprising the following steps:
  • the search words input by the user are acquired in real time, and whether the search result is detected is detected. If the search result is not found, the search words without the search result are removed from the special characters and stored in the Lucence custom word.
  • the search term of the search result is processed by word segmentation, and some phrases after the word segmentation are obtained; and some phrases after the word segmentation are continuously searched to detect whether the result is searched, and if the result is not found, The phrase after the word segmentation that cannot find the result is removed from the special character and stored in the Lucence custom vocabulary; if the result is searched, the time of the search, the search term after the word segmentation, the search feedback information, and finally the Lucence supporting Lucence full-text search is established. Custom thesaurus.
  • the method further includes the following steps: based on the Lucence custom vocabulary supporting Lucence full-text search, according to the search amount of the field, Search counter The feed information and the custom weight variable are linearly superimposed, and the weight value of the field is calculated according to the field weight dynamic allocation formula. Then, the calculated weight value of the field is dynamically assigned to the field through the weight setting interface of the Lucence full-text search engine.
  • Boost is the weight value of a field
  • n is the amount of retrieval of the field in a certain period of time
  • m is the total amount of complete search feedback after the field is retrieved in a certain period of time
  • t is the incomplete search feedback after the field is retrieved in a certain period of time.
  • the total amount r is the custom weight variable
  • ⁇ is the coefficient factor of the retrieval quantity
  • ⁇ is the coefficient factor of the complete search feedback
  • ⁇ is the coefficient factor of the incomplete search feedback
  • ⁇ is the global coordination coefficient factor.
  • the custom weight variable is a primary broadcast name, an anchor room name, or a room type.
  • the custom weight variable changes after the system changes or the user search preferences change.
  • the invention also provides a full-text search matching system based on Lucence custom vocabulary, the system comprises a Lucence custom vocabulary building unit, and the Lucence custom vocabulary building unit is used to establish a Lucence custom word supporting Lucence full-text search.
  • Library In the search environment based on Lucence full-text search engine, the search words input by the user are obtained in real time, and whether the search result is detected is detected. If the search result is not found, the search words without the search result are removed from the special characters and stored in Lucence self.
  • the search terms of the searched results are subjected to word segmentation to obtain a plurality of phrases after the word segmentation; and the plurality of phrases after the segmentation are further searched to detect whether the results are searched, and if the results are not found, Then, the phrase after the word segmentation of the result cannot be searched for, and the special character is removed, and the result is stored in the Lucence custom vocabulary; if the result is searched, the search time, the search term after the word segmentation, and the search feedback information are recorded.
  • the system further includes a field weight dynamic allocation unit, and the field weight dynamic allocation unit is configured to dynamically allocate field weights: based on the Lucence custom vocabulary, according to the search amount of the field and the search feedback.
  • the information and the custom weight variable are linearly superimposed, and the weight value of the field is calculated according to the field weight dynamic allocation formula. Then, the calculated weight value of the field is dynamically assigned to the field through the weight setting interface of the Lucence full-text search engine.
  • Boost is the weight value of a field
  • n is the amount of retrieval of the field in a certain period of time
  • m is the total amount of complete search feedback after the field is retrieved in a certain period of time
  • t is the incomplete search feedback after the field is retrieved in a certain period of time.
  • the total amount, r is a custom weight variable, for example: anchor name, anchor room name, room type
  • ⁇ is the coefficient factor of the retrieval amount
  • ⁇ is the coefficient factor of the complete search feedback
  • ⁇ is the coefficient factor of the incomplete search feedback
  • ⁇ A global coordination factor factor.
  • the custom weight variable is a primary broadcast name, an anchor room name, or a room type.
  • the custom weight variable changes after the system changes or the user search preferences change.
  • the present invention establishes a Lucence custom vocabulary for performing Lucence full-text search in a search environment based on the Lucence full-text search engine, and acquires search words input by the user in real time, and detects whether the search result is found.
  • the search term without the search result is removed from the special character and stored in the Lucence custom vocabulary; if the result is searched, the search term of the search result is processed by word segmentation, and some phrases after the word segmentation are obtained; The phrase continues to search to detect if the result is searched, such as If the result is not searched, the phrase after the word segmentation of the result cannot be searched for, and the special character is removed, and the result is stored in the Lucence custom vocabulary; if the result is searched, the search time, the search term after the word segmentation, and the search feedback information are recorded.
  • the invention can quickly and effectively establish a dedicated Lucence custom vocabulary according to the search words input by the user, and form a Lucence custom vocabulary that conforms to the current search environment, and is used for Lucence full-text search, which can achieve better search effect.
  • a dedicated Lucence custom vocabulary according to the search words input by the user
  • a Lucence custom vocabulary that conforms to the current search environment, and is used for Lucence full-text search, which can achieve better search effect.
  • conventional thesaurus may not be able to meet such needs.
  • the optimal result may not be obtained during the first search, but as the Lucence custom thesaurus is continuously iteratively updated, the search result is gradually optimized as the user's search volume increases.
  • the present invention dynamically assigns field weights: according to the search amount of the field, the search feedback information and the linear superposition of the custom weight variables, the field weights are dynamically calculated according to the field weights, and then the field weight values are calculated.
  • the weight setting interface (setboost) of the Lucence full-text search engine the calculated field weight value is dynamically assigned to the field, and the weight of each field can be dynamically and stably allocated.
  • the custom weight variable changes as the system transitions or the user's search preferences change.
  • the search system has the following fields: anchor name, anchor room name, room type. At the beginning of the system, the search needs to search for the name of the anchor name more closely, then only need to increase the custom weight, that is, the custom weight variable in the field weight dynamic allocation formula.
  • FIG. 1 is a flowchart of a full-text search matching method based on a Lucence custom vocabulary in an embodiment of the present invention.
  • an embodiment of the present invention provides a full-text search matching method based on a Lucence custom vocabulary, including the following steps:
  • the phrase after the word segmentation of the result cannot be searched for, and the special character is removed, and the result is stored in the Lucence custom vocabulary; if the result is searched, the search after the search time and the word segmentation is recorded. Words, search feedback information, and finally establish a Lucence custom vocabulary that supports Lucence full-text search;
  • S2 dynamically assigning field weights: based on the Lucence custom vocabulary supporting Lucence full-text search, according to the field search amount, search feedback information and custom weight variables linear superposition, according to the field weight dynamic allocation formula timing calculation field weight Value, and then through the weight setting interface (setboost) of the Lucence full-text search engine, dynamically assigning the calculated field weight value to the field;
  • Boost is the weight value of a field
  • n is the amount of retrieval of the field in a certain period of time
  • m is the total amount of complete search feedback after the field is retrieved in a certain period of time
  • t is the incomplete search feedback after the field is retrieved in a certain period of time.
  • the total amount, r is a custom weight variable, for example: anchor name, anchor room name, room type
  • ⁇ is the coefficient factor of the retrieval amount
  • ⁇ is the coefficient factor of the complete search feedback
  • ⁇ is the coefficient factor of the incomplete search feedback
  • ⁇ A global coordination factor factor.
  • the custom weight variable can be the primary broadcast name, the anchor room name, or the room type.
  • the custom weight variable changes.
  • the embodiment of the invention further provides a full-text search matching system based on a Lucence custom vocabulary, the system comprising a Lucence custom vocabulary building unit and a field weight dynamic allocating unit, wherein:
  • the Lucence custom vocabulary building unit is used to establish a Lucence custom vocabulary that supports Lucence full-text search: in the search environment based on the Lucence full-text search engine, the user inputs the search words in real time, and detects whether the search result is found. As a result, the search term without the search result is removed from the special character and stored in the Lucence custom vocabulary; if the result is searched, the search term of the search result is processed by word segmentation, and some phrases after the word segmentation are obtained; Several phrases continue to search to detect whether the result is searched. If the result is not found, the phrase after the word segmentation of the result is not removed, and the special character is deleted and stored in the Lucence custom dictionary. If the result is found, the search is recorded. Time, search terms after word segmentation, search feedback information;
  • the field weight dynamic allocation unit is used to dynamically allocate field weights: based on the Lucence custom vocabulary supporting Lucence full-text search, according to the field search amount, search feedback information and custom weight variables, linear superposition, dynamic allocation according to field weights
  • the formula calculates the field weight value periodically, and then dynamically assigns the calculated field weight value to the field through the weight setting interface (setboost) of the Lucence full-text search engine;
  • Boost is the weight value of a field
  • n is the amount of retrieval of the field in a certain period of time
  • m is the total amount of complete search feedback after the field is retrieved in a certain period of time
  • t is the incomplete search feedback after the field is retrieved in a certain period of time.
  • Total r is a custom weight variable, for example: anchor name, anchor room name, room type
  • ⁇ is the coefficient factor of the retrieval amount
  • ⁇ is complete
  • ⁇ is the coefficient factor of the incomplete search feedback
  • ⁇ is the global coordination coefficient factor.
  • the custom weight variable can be the primary broadcast name, the anchor room name, or the room type.
  • the custom weight variable changes.
  • the embodiment of the invention can quickly and effectively establish a Lucence custom vocabulary dedicated to the user, and form a Lucence custom vocabulary that conforms to the current search environment, and is used for Lucence full-text search, which can achieve better search results.
  • the optimal result may not be obtained during the first search, but as the Lucence custom thesaurus is continuously iteratively updated, the search result is gradually optimized as the user's search volume increases.
  • the weight is often given as a constant, and this setting may result in good search results at a certain time.
  • this setting will be difficult to obtain accurate results.
  • how to dynamically assign the weights of each field according to the search feedback effect, the search amount and other factors to achieve the optimal matching result is also a key consideration for those skilled in the art.
  • the search volume of the anchor name in the system will increase, and the search feedback effect will be the best, and the weight will be the same. Will dynamically bias this field; however, as users gradually understand the system, they pay more attention to the content of the room, the corresponding search volume will increase, and the feedback effect is better, naturally, the weight is also Will be biased to the corresponding room name and room type.
  • the search system has the following fields: anchor name, anchor room name, room type.
  • the search needs to search for the name of the anchor name more closely, then only need to increase the custom weight, that is, the custom weight variable in the field weight dynamic allocation formula.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computational Linguistics (AREA)
  • Mathematical Physics (AREA)
  • Artificial Intelligence (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

一种基于Lucene自定义词库的全文检索匹配方法及系统，涉及大数据搜索领域。该方法包括以下步骤：在Lucene搜索环境中，实时获取用户输入的搜索词，检测是否搜索到结果，搜索不到，则将搜索词去掉特殊字符，存入Lucene自定义词库；搜索到，则对搜索词进行分词，对分词后的若干词组继续进行搜索，检测是否搜索到结果，搜索不到，则将搜索不到结果的分词后的词组去掉特殊字符，存入Lucene自定义词库；搜索到，则记录搜索的时间、分词后的搜索词、搜索反馈信息，最终建立支持Lucene全文检索的Lucene自定义词库。该方法根据用户输入的搜索词，能够快速有效地建立自己专用的Lucene自定义词库。

Description

基于Lucence自定义词库的全文检索匹配方法及系统 技术领域
本发明涉及大数据搜索领域，具体是涉及一种基于Lucence自定义词库的全文检索匹配方法及系统。
背景技术
Apache Lucence是一个开放源代码的全文检索引擎工具包，但它不是一个完整的全文检索引擎，而是一个全文检索引擎的架构，提供了完整的查询引擎和索引引擎，部分文本分析引擎。
为了便于读者理解，下面先对相关术语进行简单说明：
Apache Lucence：Apache旗下一款开源全文检索项目；
全文检索：不同于传统的模糊匹配，先对搜索词按一定规则分词后，再将分词与源数据进行匹配，根据出现分词次数，分词邻近距离，权重等数据进行打分，获得检索结果；
分词：全文检索索引，例如：我是中国人，其分词可为：我、是、中国、人、中国人等；
公共词库：存放公共分词规则的词库，例如，常用的：你好、中国等；
自定义词库：根据自己需求，用于存放自己需要的分词规则的字典词库；
搜索反馈：搜索效果反馈，即用户输入搜索词进入搜索页面后，是否有点击页面链接亦或在多次翻页之后点击链接；
搜索量：某一搜索词某时段全站搜索量；
字段：需要搜索的字段，例如：游戏名、主播名、房间名等。
在Apache Lucence全文检索中，需要对源数据进行分词索引。若没有针对特定词组进行分词处理，则该词组无法被检索到。例如，针对游戏直播领域的搜索，“英雄联盟”、“Dota2”、“炉石传说”等在公共词库中基本不会出现的词汇，将很难被检索到。因此，如何获取用户最需要检索的词汇，并生成自定义词库是在全文检索领域一个重要的难点。
发明内容
本发明的目的是为了克服上述背景技术的不足，提供一种基于Lucence自定义词库的全文检索匹配方法及系统，根据用户输入的搜索词，能够快速有效地建立自己专用的Lucence自定义词库。
本发明提供一种基于Lucence自定义词库的全文检索匹配方法，包括以下步骤：
在基于Lucence全文检索引擎的搜索环境中，实时获取用户输入的搜索词，检测是否搜索到结果，如果搜索不到结果，则将搜索不到结果的搜索词去掉特殊字符，存入Lucence自定义词库；如果搜索到结果，则对搜索到结果的搜索词进行分词处理，得到分词后的若干词组；对分词后的若干词组继续进行搜索，检测是否搜索到结果，如果搜索不到结果，则将搜索不到结果的分词后的词组去掉特殊字符，存入Lucence自定义词库；如果搜索到结果，则记录搜索的时间、分词后的搜索词、搜索反馈信息，最终建立支持Lucence全文检索的Lucence自定义词库。
在上述技术方案的基础上，所述建立支持Lucence全文检索的Lucence自定义词库之后，还包括以下步骤：在建立支持Lucence全文检索的Lucence自定义词库的基础上，根据字段的搜索量、搜索反 馈信息及自定义权重变量线性叠加，按照字段权重动态分配公式定时计算字段权重值，再通过Lucence全文检索引擎的权重设置接口，将计算得到的字段权重值动态赋值给字段。
在上述技术方案的基础上，所述字段权重动态分配公式为：
boost＝(α*n+β*m+δ*In(t)+r)*ρ，
其中，boost为某字段权重值，n为某时段该字段被检索量，m为某时段该字段被检索后具有完整搜索反馈的总量，t为某时段该字段被检索后不完整搜索反馈的总量，r为自定义权重变量，α为检索量的系数因子，β为完整搜索反馈的系数因子，δ为不完整搜索反馈的系数因子，ρ为全局协调系数因子。
在上述技术方案的基础上，所述自定义权重变量为主播名、主播房间名或房间类型。
在上述技术方案的基础上，当系统出现转型或用户搜索喜好发生变化之后，所述自定义权重变量随之改变。
本发明还提供一种基于Lucence自定义词库的全文检索匹配系统，该系统包括Lucence自定义词库建立单元，所述Lucence自定义词库建立单元用于建立支持Lucence全文检索的Lucence自定义词库：在基于Lucence全文检索引擎的搜索环境中，实时获取用户输入的搜索词，检测是否搜索到结果，如果搜索不到结果，则将搜索不到结果的搜索词去掉特殊字符，存入Lucence自定义词库；如果搜索到结果，则对搜索到结果的搜索词进行分词处理，得到分词后的若干词组；对分词后的若干词组继续进行搜索，检测是否搜索到结果，如果搜索不到结果，则将搜索不到结果的分词后的词组去掉特殊字符，存入Lucence自定义词库；如果搜索到结果，则记录搜索的时间、分词后的搜索词、搜索反馈信息。
在上述技术方案的基础上，该系统还包括字段权重动态分配单元，所述字段权重动态分配单元用于动态分配字段权重：在Lucence自定义词库的基础上，根据字段的搜索量、搜索反馈信息及自定义权重变量线性叠加，按照字段权重动态分配公式定时计算字段权重值，再通过Lucence全文检索引擎的权重设置接口，将计算得到的字段权重值动态赋值给字段。
在上述技术方案的基础上，所述字段权重动态分配公式为：
boost＝(α*n+β*m+δ*In(t)+r)*ρ，
其中，boost为某字段权重值，n为某时段该字段被检索量，m为某时段该字段被检索后具有完整搜索反馈的总量，t为某时段该字段被检索后不完整搜索反馈的总量，r为自定义权重变量，例如：主播名、主播房间名、房间类型；α为检索量的系数因子，β为完整搜索反馈的系数因子，δ为不完整搜索反馈的系数因子，ρ为全局协调系数因子。
在上述技术方案的基础上，所述自定义权重变量为主播名、主播房间名或房间类型。
在上述技术方案的基础上，当系统出现转型或用户搜索喜好发生变化之后，所述自定义权重变量随之改变。
与现有技术相比，本发明的优点如下：
(1)本发明在基于Lucence全文检索引擎的搜索环境中，建立用于进行Lucence全文检索的Lucence自定义词库，实时获取用户输入的搜索词，检测是否搜索到结果，如果搜索不到结果，则将搜索不到结果的搜索词去掉特殊字符，存入Lucence自定义词库；如果搜索到结果，则对搜索到结果的搜索词进行分词处理，得到分词后的若干词组；对分词后的若干词组继续进行检索，检测是否搜索到结果，如 果搜索不到结果，则将搜索不到结果的分词后的词组去掉特殊字符，存入Lucence自定义词库；如果搜索到结果，则记录搜索的时间、分词后的搜索词、搜索反馈信息。本发明根据用户输入的搜索词，能够快速有效地建立自己专用的Lucence自定义词库，形成符合当前搜索环境的Lucence自定义词库，用于Lucence全文检索，可达到更佳的搜索效果。例如，对于游戏直播，可能用户更期望搜索关于“YYF”、“55开”、“安德罗妮”等信息，采用常规词库可能无法满足这类需求。采用本发明实施例的方法，在首次搜索时，可能也不会得到最优结果，但是，随着Lucence自定义词库不断迭代更新，搜索结果会随着用户的搜索量上涨逐渐优化。
(2)本发明在Lucence自定义词库的基础上，动态分配字段权重：根据字段的搜索量、搜索反馈信息及自定义权重变量线性叠加，按照字段权重动态分配公式定时计算字段权重值，再通过Lucence全文检索引擎的权重设置接口(setboost)，将计算得到的字段权重值动态赋值给字段，能够稳定有效地动态分配各字段权重。当系统出现转型或用户搜索喜好发生变化之后，自定义权重变量随之改变。例如，搜索系统有以下几个字段：主播名、主播房间名、房间类型。系统一开始需要搜索对主播名的搜索更侧重，则只需要增大自定义权重，即字段权重动态分配公式中的自定义权重变量。
附图说明
图1是本发明实施例中基于Lucence自定义词库的全文检索匹配方法的流程图。
具体实施方式
下面结合附图及具体实施例对本发明作进一步的详细描述。
参见图1所示，本发明实施例提供一种基于Lucence自定义词库的全文检索匹配方法，包括以下步骤：
S1、建立支持Lucence全文检索的Lucence自定义词库：在基于Lucence全文检索引擎的搜索环境中，实时获取用户输入的搜索词，检测是否搜索到结果，如果搜索不到结果，则将搜索不到结果的搜索词去掉特殊字符，存入Lucence自定义词库；如果搜索到结果，则对搜索到结果的搜索词进行分词处理，得到分词后的若干词组；对分词后的若干词组继续进行搜索，检测是否搜索到结果，如果搜索不到结果，则将搜索不到结果的分词后的词组去掉特殊字符，存入Lucence自定义词库；如果搜索到结果，则记录搜索的时间、分词后的搜索词、搜索反馈信息，最终建立支持Lucence全文检索的Lucence自定义词库；
S2、动态分配字段权重：在建立支持Lucence全文检索的Lucence自定义词库的基础上，根据字段的搜索量、搜索反馈信息及自定义权重变量线性叠加，按照字段权重动态分配公式定时计算字段权重值，再通过Lucence全文检索引擎的权重设置接口(setboost)，将计算得到的字段权重值动态赋值给字段；
字段权重动态分配公式为：
boost＝(α*n+β*m+δ*In(t)+r)*ρ，
其中，boost为某字段权重值，n为某时段该字段被检索量，m为某时段该字段被检索后具有完整搜索反馈的总量，t为某时段该字段被检索后不完整搜索反馈的总量，r为自定义权重变量，例如：主播名、主播房间名、房间类型；α为检索量的系数因子，β为完整搜索反馈的系数因子，δ为不完整搜索反馈的系数因子，ρ为全局协调系数因子。
自定义权重变量可以为主播名、主播房间名或房间类型，当系统出现转型或用户搜索喜好发生变化之后，自定义权重变量随之改变。
本发明实施例还提供一种基于Lucence自定义词库的全文检索匹配系统，该系统包括Lucence自定义词库建立单元和字段权重动态分配单元，其中：
Lucence自定义词库建立单元用于建立支持Lucence全文检索的Lucence自定义词库：在基于Lucence全文检索引擎的搜索环境中，实时获取用户输入的搜索词，检测是否搜索到结果，如果搜索不到结果，则将搜索不到结果的搜索词去掉特殊字符，存入Lucence自定义词库；如果搜索到结果，则对搜索到结果的搜索词进行分词处理，得到分词后的若干词组；对分词后的若干词组继续进行搜索，检测是否搜索到结果，如果搜索不到结果，则将搜索不到结果的分词后的词组去掉特殊字符，存入Lucence自定义词库；如果搜索到结果，则记录搜索的时间、分词后的搜索词、搜索反馈信息；
字段权重动态分配单元用于动态分配字段权重：在建立支持Lucence全文检索的Lucence自定义词库的基础上，根据字段的搜索量、搜索反馈信息及自定义权重变量线性叠加，按照字段权重动态分配公式定时计算字段权重值，再通过Lucence全文检索引擎的权重设置接口(setboost)，将计算得到的字段权重值动态赋值给字段；
字段权重动态分配公式为：
boost＝(α*n+β*m+δ*In(t)+r)*ρ，
其中，boost为某字段权重值，n为某时段该字段被检索量，m为某时段该字段被检索后具有完整搜索反馈的总量，t为某时段该字段被检索后不完整搜索反馈的总量，r为自定义权重变量，例如：主播名、主播房间名、房间类型；α为检索量的系数因子，β为完整 搜索反馈的系数因子，δ为不完整搜索反馈的系数因子，ρ为全局协调系数因子。
自定义权重变量可以为主播名、主播房间名或房间类型，当系统出现转型或用户搜索喜好发生变化之后，自定义权重变量随之改变。
本发明实施例根据用户输入条件，能够快速有效地建立自己专用的Lucence自定义词库，并且形成符合当前搜索环境的Lucence自定义词库，用于Lucence全文检索，可达到更佳的搜索效果。
例如，对于游戏直播，可能用户更期望搜索关于“YYF”、“55开”、“安德罗妮”等信息，采用常规词库可能无法满足这类需求。采用本发明实施例的方法，在首次搜索时，可能也不会得到最优结果，但是，随着Lucence自定义词库不断迭代更新，搜索结果会随着用户的搜索量上涨逐渐优化。
此外，在搜索系统中，权重往往被赋予为某一常量，这种设定在某一时期可能会得到良好的搜素结果。然而，随着系统的转型，用户人群口味变更或者源数据的变化等因素，这一设定便将难以获得准确的结果。在多字段检索中，如何根据搜索反馈效果，搜索量等因素动态分配各字段权重以达到最优匹配结果，也是本领域技术人员需要重点考虑的。
例如，搜索系统中的用户一开始只是对某几个主播感兴趣，则他们对主播名的搜索结果更加看重，系统中主播名的搜索量变化加大，搜索反馈效果也会最为良好，权重也会动态偏向这一字段；然而，随着用户对系统的逐渐了解，他们更对房间内容关注更多，则相应的其搜索量便会增大，且反馈效果也更为良好，自然，权重也会偏向到相应的房间名及房间类型。
当系统出现转型或用户搜索喜好发生变化之后，自定义权重变量 随之改变。例如，搜索系统有以下几个字段：主播名、主播房间名、房间类型。系统一开始需要搜索对主播名的搜索更侧重，则只需要增大自定义权重，即字段权重动态分配公式中的自定义权重变量。
本领域的技术人员可以对本发明实施例进行各种修改和变型，倘若这些修改和变型在本发明权利要求及其等同技术的范围之内，则这些修改和变型也在本发明的保护范围之内。
说明书中未详细描述的内容为本领域技术人员公知的现有技术。

Claims (10)

  1. 一种基于Lucence自定义词库的全文检索匹配方法，其特征在于，包括以下步骤：
    在基于Lucence全文检索引擎的搜索环境中，实时获取用户输入的搜索词，检测是否搜索到结果，如果搜索不到结果，则将搜索不到结果的搜索词去掉特殊字符，存入Lucence自定义词库；如果搜索到结果，则对搜索到结果的搜索词进行分词处理，得到分词后的若干词组；对分词后的若干词组继续进行搜索，检测是否搜索到结果，如果搜索不到结果，则将搜索不到结果的分词后的词组去掉特殊字符，存入Lucence自定义词库；如果搜索到结果，则记录搜索的时间、分词后的搜索词、搜索反馈信息，最终建立支持Lucence全文检索的Lucence自定义词库。
  2. 如权利要求1所述的基于Lucence自定义词库的全文检索匹配方法，其特征在于：所述建立支持Lucence全文检索的Lucence自定义词库之后，还包括以下步骤：在建立支持Lucence全文检索的Lucence自定义词库的基础上，根据字段的搜索量、搜索反馈信息及自定义权重变量线性叠加，按照字段权重动态分配公式定时计算字段权重值，再通过Lucence全文检索引擎的权重设置接口，将计算得到的字段权重值动态赋值给字段。
  3. 如权利要求2所述的基于Lucence自定义词库的全文检索匹配方法，其特征在于：所述字段权重动态分配公式为：
    boost＝(α*n+β*m+δ*In(t)+r)*ρ，
    其中，boost为某字段权重值，n为某时段该字段被检索量，m为某时段该字段被检索后具有完整搜索反馈的总量，t为某时段该字段被检索后不完整搜索反馈的总量，r为自定义权重变量，α为检索 量的系数因子，β为完整搜索反馈的系数因子，δ为不完整搜索反馈的系数因子，ρ为全局协调系数因子。
  4. 如权利要求3所述的基于Lucence自定义词库的全文检索匹配方法，其特征在于：所述自定义权重变量为主播名、主播房间名或房间类型。
  5. 如权利要求4所述的基于Lucence自定义词库的全文检索匹配方法，其特征在于：当系统出现转型或用户搜索喜好发生变化之后，所述自定义权重变量随之改变。
  6. 一种基于Lucence自定义词库的全文检索匹配系统，其特征在于：该系统包括Lucence自定义词库建立单元，所述Lucence自定义词库建立单元用于建立支持Lucence全文检索的Lucence自定义词库：在基于Lucence全文检索引擎的搜索环境中，实时获取用户输入的搜索词，检测是否搜索到结果，如果搜索不到结果，则将搜索不到结果的搜索词去掉特殊字符，存入Lucence自定义词库；如果搜索到结果，则对搜索到结果的搜索词进行分词处理，得到分词后的若干词组；对分词后的若干词组继续进行搜索，检测是否搜索到结果，如果搜索不到结果，则将搜索不到结果的分词后的词组去掉特殊字符，存入Lucence自定义词库；如果搜索到结果，则记录搜索的时间、分词后的搜索词、搜索反馈信息。
  7. 如权利要求6所述的基于Lucence自定义词库的全文检索匹配系统，其特征在于：该系统还包括字段权重动态分配单元，所述字段权重动态分配单元用于动态分配字段权重：在Lucence自定义词库的基础上，根据字段的搜索量、搜索反馈信息及自定义权重变量线性叠加，按照字段权重动态分配公式定时计算字段权重值，再通过Lucence全文检索引擎的权重设置接口，将计算得到的字段权重值动 态赋值给字段。
  8. 如权利要求7所述的基于Lucence自定义词库的全文检索匹配系统，其特征在于：所述字段权重动态分配公式为：
    boost＝(α*n+β*m+δ*In(t)+r)*ρ，
    其中，boost为某字段权重值，n为某时段该字段被检索量，m为某时段该字段被检索后具有完整搜索反馈的总量，t为某时段该字段被检索后不完整搜索反馈的总量，r为自定义权重变量，例如：主播名、主播房间名、房间类型；α为检索量的系数因子，β为完整搜索反馈的系数因子，δ为不完整搜索反馈的系数因子，ρ为全局协调系数因子。
  9. 如权利要求8所述的基于Lucence自定义词库的全文检索匹配系统，其特征在于：所述自定义权重变量为主播名、主播房间名或房间类型。
  10. 如权利要求9所述的基于Lucence自定义词库的全文检索匹配系统，其特征在于：当系统出现转型或用户搜索喜好发生变化之后，所述自定义权重变量随之改变。
PCT/CN2017/080784 2016-05-16 2017-04-17 基于Lucence自定义词库的全文检索匹配方法及系统 Ceased WO2017198022A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US15/771,057 US20180260473A1 (en) 2016-05-16 2017-04-17 Full text retrieving and matching method and system based on lucene custom lexicon

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201610321306.6A CN106021430B (zh) 2016-05-16 2016-05-16 基于Lucence自定义词库的全文检索匹配方法及系统
CN201610321306.6 2016-05-16

Publications (1)

Publication Number Publication Date
WO2017198022A1 true WO2017198022A1 (zh) 2017-11-23

Family

ID=57097899

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2017/080784 Ceased WO2017198022A1 (zh) 2016-05-16 2017-04-17 基于Lucence自定义词库的全文检索匹配方法及系统

Country Status (3)

Country Link
US (1) US20180260473A1 (zh)
CN (1) CN106021430B (zh)
WO (1) WO2017198022A1 (zh)

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106021430B (zh) * 2016-05-16 2018-01-19 武汉斗鱼网络科技有限公司 基于Lucence自定义词库的全文检索匹配方法及系统
CN107169115A (zh) * 2017-05-22 2017-09-15 深圳市连用科技有限公司 添加自定义分词的方法及装置
CN110069536A (zh) * 2018-08-14 2019-07-30 深圳开黑科技有限公司 基于游戏陪玩平台的车队搜索方法、设备及系统
CN111061836B (zh) * 2019-12-18 2022-07-22 焦点科技股份有限公司 一种适用于Lucene全文检索引擎的自定义打分方法
CN115064237A (zh) * 2022-06-09 2022-09-16 山东浪潮智慧医疗科技有限公司 一种实现医院体检总结数据标准化的方法
CN115982334A (zh) * 2023-01-16 2023-04-18 长城汽车股份有限公司 搜索方法、装置、电子设备及存储介质

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102929902A (zh) * 2012-07-05 2013-02-13 江苏新瑞峰信息科技有限公司 一种基于中文检索的分词方法及装置
CN103729463A (zh) * 2014-01-14 2014-04-16 赛特斯信息科技股份有限公司 基于Lucene和Solr实现全文检索的方法
US20140280166A1 (en) * 2013-03-15 2014-09-18 Maritz Holdings Inc. Systems and methods for classifying electronic documents
CN104469832A (zh) * 2014-12-19 2015-03-25 武汉虹信通信技术有限责任公司 移动通信网络故障分析定位辅助系统
CN105404661A (zh) * 2015-11-05 2016-03-16 浪潮(北京)电子信息产业有限公司 一种更新索引文件的方法及系统
CN106021430A (zh) * 2016-05-16 2016-10-12 武汉斗鱼网络科技有限公司 基于Lucence自定义词库的全文检索匹配方法及系统

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7424421B2 (en) * 2004-03-03 2008-09-09 Microsoft Corporation Word collection method and system for use in word-breaking
CN100557612C (zh) * 2007-11-15 2009-11-04 深圳市迅雷网络技术有限公司 一种基于搜索引擎的搜索结果排序方法及装置
CN103455619B (zh) * 2013-09-12 2016-09-07 焦点科技股份有限公司 一种基于Lucene分片结构的打分处理方法及系统
CN104133854A (zh) * 2014-07-09 2014-11-05 新乡学院 一种MySQL多语言混合文本全文检索的实现方法

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102929902A (zh) * 2012-07-05 2013-02-13 江苏新瑞峰信息科技有限公司 一种基于中文检索的分词方法及装置
US20140280166A1 (en) * 2013-03-15 2014-09-18 Maritz Holdings Inc. Systems and methods for classifying electronic documents
CN103729463A (zh) * 2014-01-14 2014-04-16 赛特斯信息科技股份有限公司 基于Lucene和Solr实现全文检索的方法
CN104469832A (zh) * 2014-12-19 2015-03-25 武汉虹信通信技术有限责任公司 移动通信网络故障分析定位辅助系统
CN105404661A (zh) * 2015-11-05 2016-03-16 浪潮(北京)电子信息产业有限公司 一种更新索引文件的方法及系统
CN106021430A (zh) * 2016-05-16 2016-10-12 武汉斗鱼网络科技有限公司 基于Lucence自定义词库的全文检索匹配方法及系统

Also Published As

Publication number Publication date
US20180260473A1 (en) 2018-09-13
CN106021430B (zh) 2018-01-19
CN106021430A (zh) 2016-10-12

Similar Documents

Publication Publication Date Title
US9836511B2 (en) Computer-generated sentiment-based knowledge base
Veale et al. Comprehending and generating apt metaphors: a web-driven, case-based approach to figurative language
CN105975558B (zh) 建立语句编辑模型的方法、语句自动编辑方法及对应装置
JP6788637B2 (ja) 情報検索装置及び情報検索システム
CN106021430B (zh) 基于Lucence自定义词库的全文检索匹配方法及系统
US8843949B2 (en) Apparatus, method, and program for processing information
CN110059177B (zh) 一种基于用户画像的活动推荐方法及装置
CN105740229B (zh) 关键词提取的方法及装置
JP2019504410A (ja) 旅行ガイド生成方法及びシステム
CN105956053B (zh) 一种基于网络信息的搜索方法及装置
CN104008180B (zh) 结构化数据与图片的关联方法与关联装置
US20160357738A1 (en) Apparatus, system and method for string disambiguation and entity ranking
CN105608113B (zh) 判断文本中poi数据的方法及装置
CN107533558A (zh) 脉络化知识面板
CN103744887B (zh) 一种用于人物搜索的方法、装置和计算机设备
US11693900B2 (en) Method and system for providing resegmented audio content
CN107679226A (zh) 基于主题的旅游本体构建方法
Qu et al. A novel approach based on multi-view content analysis and semi-supervised enrichment for movie recommendation
JP5951300B2 (ja) サービス制御装置、サービス制御方法およびサービス制御プログラム
Ling et al. Critical zone recognition of smellscape of a Chinese traditional market based on the sensitivity–coordination matrix
US11816147B2 (en) Enhanced image-search using contextual tags
KR101602342B1 (ko) 의미 태깅된 자연어 질의의 의도에 부합하는 정보 추출 및 제공 방법 및 시스템
CN108140034A (zh) 使用主题模型基于接收的词项选择内容项目
JP6632564B2 (ja) 違法コンテンツ探索装置、違法コンテンツ探索方法、及びプログラム
Sheikh et al. Study of entity-topic models for OOV proper name retrieval.

Legal Events

Date Code Title Description
WWE Wipo information: entry into national phase

Ref document number: 15771057

Country of ref document: US

NENP Non-entry into the national phase

Ref country code: DE

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17798586

Country of ref document: EP

Kind code of ref document: A1

122 Ep: pct application non-entry in european phase

Ref document number: 17798586

Country of ref document: EP

Kind code of ref document: A1