WO2017206492A1 - 二元特征词典的构建方法和装置 - Google Patents

二元特征词典的构建方法和装置 Download PDF

Info

Publication number
WO2017206492A1
WO2017206492A1 PCT/CN2016/111571 CN2016111571W WO2017206492A1 WO 2017206492 A1 WO2017206492 A1 WO 2017206492A1 CN 2016111571 W CN2016111571 W CN 2016111571W WO 2017206492 A1 WO2017206492 A1 WO 2017206492A1
Authority
WO
WIPO (PCT)
Prior art keywords
binary
binary feature
statistic
preset
features
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2016/111571
Other languages
English (en)
French (fr)
Inventor
周坤胜
何径舟
石磊
冯仕堃
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Baidu Netcom Science and Technology Co Ltd
Original Assignee
Beijing Baidu Netcom Science and Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Baidu Netcom Science and Technology Co Ltd filed Critical Beijing Baidu Netcom Science and Technology Co Ltd
Priority to US16/306,488 priority Critical patent/US10831993B2/en
Publication of WO2017206492A1 publication Critical patent/WO2017206492A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/95—Retrieval from the web
    • G06F16/951—Indexing; Web crawling techniques
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
    • G06F17/10—Complex mathematical operations
    • G06F17/18—Complex mathematical operations for evaluating statistical data, e.g. average values, frequency distributions, probability functions, regression analysis
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/20—Natural language analysis
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/20—Natural language analysis
    • G06F40/237—Lexical tools
    • G06F40/242—Dictionaries
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/20—Natural language analysis
    • G06F40/279—Recognition of textual entities
    • G06F40/284—Lexical analysis, e.g. tokenisation or collocates
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/30—Semantic analysis
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0499—Feedforward networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/09—Supervised learning

Definitions

  • the present application relates to the field of natural language processing technologies, and in particular, to a method and an apparatus for constructing a binary feature dictionary.
  • a search engine is a retrieval system that feeds back results to the user.
  • One implementation of the search engine is based on a semantic similarity model.
  • the semantic similarity model can be used to calculate the similarity between the query retrieved by the user and the title of the candidate, and then sorted according to the similarity to return. Search Results.
  • the semantic similarity model can be constructed using neural network training.
  • the training data is usually information of a basic granularity of information or a phrase granularity, wherein the basic granularity information is obtained by cutting a sentence corpus.
  • phrase granular information Includes multiple minimum-grained word segments, such as "Brazil Portuguese.”
  • new granularity information can also be introduced in the training data. When it is necessary to introduce new granular information, it is necessary to solve the problem of how to extract new granular information.
  • the present application aims to solve at least one of the technical problems in the related art to some extent.
  • an object of the present application is to propose a method for constructing a binary feature dictionary, which can construct a binary feature dictionary, so that the binary feature dictionary can be used to extract binary features from the model corpus, further
  • the binary feature can be used as a new feature introduced in the training data of the semantic similarity model.
  • Another object of the present application is to propose a device for constructing a binary feature dictionary.
  • a method for constructing a binary feature dictionary according to the first aspect of the present application includes: extracting a binary feature from a corpus; calculating a preset statistic of the binary feature; The quantity selects the preset number of binary features in order to form a binary feature dictionary.
  • the method for constructing the binary feature dictionary proposed by the embodiment of the first aspect of the present application can realize the binary feature dictionary by calculating a preset statistic of the binary feature and selecting a binary feature to form a binary feature dictionary according to a preset statistic.
  • the binary feature is extracted from the model corpus by using the binary feature dictionary. Further, the binary feature can be used as a new feature introduced in the training data of the semantic similarity model.
  • the apparatus for constructing a binary feature dictionary proposed by the embodiment of the second aspect of the present application,
  • the method includes: an extraction module, configured to extract a binary feature from the corpus; a calculation module, configured to calculate a preset statistic of the binary feature; and a generating module, configured to sequentially select a preset according to the preset statistic
  • the binary feature of the number constitutes a binary feature dictionary.
  • the apparatus for constructing a binary feature dictionary proposed by the second aspect of the present application can realize a binary feature dictionary by calculating a preset statistic of a binary feature and selecting a binary feature to form a binary feature dictionary according to a preset statistic.
  • the binary feature is extracted from the model corpus by using the binary feature dictionary. Further, the binary feature can be used as a new feature introduced in the training data of the semantic similarity model.
  • An embodiment of the present application provides a terminal, including: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to: extract a binary feature from a corpus; calculate the binary feature Presetting statistics; selecting a preset number of binary features according to the preset statistics to form a binary feature dictionary.
  • the embodiment of the present application proposes a non-transitory computer readable storage medium, when the instructions in the storage medium are executed by a processor of the terminal, enabling the terminal to perform a method, the method comprising: extracting two from the corpus a meta-feature; calculating a preset statistic of the binary feature; sequentially selecting a preset number of binary features according to the preset statistic to form a binary feature dictionary.
  • the embodiment of the present application provides a computer program product, when an instruction in the computer program product is executed by a processor, performing a method, the method comprising: extracting a binary feature from a corpus; and calculating the binary feature a preset statistic; selecting a preset number of binary according to the preset statistic Features that form a binary feature dictionary.
  • FIG. 1 is a schematic flow chart of a method for constructing a binary feature dictionary according to an embodiment of the present application
  • FIG. 2 is a schematic flow chart of a method for constructing a binary feature dictionary according to another embodiment of the present application
  • FIG. 3 is a schematic flow chart of a method for constructing a binary feature dictionary according to another embodiment of the present application.
  • FIG. 4 is a schematic diagram of a neural network used in an embodiment of the present application.
  • FIG. 5 is a schematic structural diagram of a device for constructing a binary feature dictionary according to an embodiment of the present invention.
  • FIG. 6 is a schematic structural diagram of a device for constructing a binary feature dictionary according to an embodiment of the present invention.
  • the sentence corpus may be first cut, and then the information required to be granular, such as basic granular information or phrase granularity information, may be obtained according to the word segmentation.
  • new granularity information can be introduced.
  • the newly introduced information of the new granularity may be referred to as a binary feature.
  • Each binary feature consists of two basic granularity information or two phrase granularity information.
  • FIG. 1 is a schematic flow chart of a method for constructing a binary feature dictionary according to an embodiment of the present application.
  • the process of this embodiment includes:
  • the corpus when constructing the binary feature dictionary, can be collected first, similar to the construction of the semantic similarity model.
  • the corpus collected when constructing the binary feature dictionary can also be specific sentence corpus, that is, a large number of sentences are collected as corpus.
  • the sentence corpus can be segmented to obtain a term.
  • the term may be basic granularity, or may be a phrase granularity, but the above two terms need to have the same granularity, for example, both are basic granularity, or both are phrases. granularity.
  • the cutting of sentences into basic-grained terms or phrase-grained terms can employ various prior art and will not be described in detail herein.
  • any two adjacent terms can be regarded as a binary feature. Therefore, each sentence can be corresponding. If the sentence includes n terms, n-1 can be extracted. Binary features. After adding the binary features corresponding to all the sentences in the corpus, multiple binary features can be obtained.
  • the preset statistic is a t statistic as an example.
  • the t statistic is a statistic that tests a single hypothesis of model parameters.
  • the formula for calculating the t statistic of a binary feature is:
  • ⁇ p 1 ⁇ p 2
  • p 1 is the probability of occurrence of one entry in the binary feature in the corpus
  • p 2 is the probability of occurrence of another entry in the binary feature in the corpus
  • S13 Select a preset number of binary features according to the preset statistic to form a binary feature dictionary.
  • a preset number of binary features can be selected according to the t statistic from large to small, and the selected binary features are combined into a binary feature dictionary.
  • a binary feature may be composed of two basic granular terms, or a binary feature may be composed of two phrase granular terms.
  • the binary features (including the binary features obtained according to the basic granularity of the terms and the binary features obtained according to the phrase granularity) are uniformly sorted, so that the binary feature dictionary can be Binary features derived from terms of basic granularity and/or binary features derived from terms of phrase granularity are included.
  • the dictionary After constructing a binary feature dictionary, the dictionary can be used to generate binary features for training in semantic similarity models.
  • the method in this embodiment may further include:
  • a binary feature included in the binary feature dictionary and which can be composed of two adjacent word segments can be used as the extracted binary feature.
  • the above-mentioned word segmentation may be a basic granularity word segmentation or a phrase granularity word segmentation.
  • a corpus of the semantic similarity module is “Baidu Brazilian Portuguese”. After the word is cut, the basic word segmentation is taken as an example. Each participle is “Baidu”, “Brazil”, “Portuguese”, and the constructed binary One of the binary features in the feature dictionary is “Brazil Portuguese”, and one binary feature extracted is “Brazil Portuguese”.
  • the binary feature dictionary does not include the binary features composed of any two adjacent word segments, the word segmentation is directly used as the extracted binary feature. Still taking the above corpus as an example, assuming that the constructed binary feature dictionary does not include “Brazil Portuguese” or “Baidu Brazil”, the extracted binary features are original word segments, such as “Baidu” and “Brazil”. ""Portuguese.”
  • the method in this embodiment may further include:
  • S15 Perform neural network training according to the training data to generate a semantic similarity model.
  • the neural network may be as shown in FIG. 4. Unlike the usual semantic similarity model, the input of this embodiment may include a binary feature.
  • the semantic similarity model can be trained by introducing binary features.
  • the binary feature dictionary can be constructed, so that the binary feature dictionary can be adopted from the model.
  • the binary feature is extracted from the corpus. Further, the binary feature can be used as a new feature introduced in the training data of the semantic similarity model.
  • FIG. 5 is a schematic structural diagram of a device for constructing a binary feature dictionary according to an embodiment of the present invention.
  • the apparatus of this embodiment includes an extraction module 51, a calculation module 52, and a generation module 53.
  • An extracting module 51 configured to extract a binary feature from the corpus
  • a calculation module 52 configured to calculate a preset statistic of the binary feature
  • the generating module 53 is configured to sequentially select a preset number of binary features according to the preset statistic to form a binary feature dictionary.
  • the extraction module 51 is specifically configured to:
  • the preset statistic calculated by the calculation module 52 is a t statistic.
  • the apparatus of this embodiment may further include:
  • the obtaining module 54 is configured to extract the binary feature from the segmentation of the semantic similarity model by using the binary feature dictionary as the training data of the semantic similarity model.
  • the apparatus of this embodiment may further include:
  • the training module 55 is configured to perform neural network training according to the training data to generate a semantic similarity model.
  • the feature constitutes a binary feature dictionary, which can realize the construction of the binary feature dictionary, so that the binary feature dictionary can be used to extract the binary feature from the model corpus. Further, the binary feature can be used as the training data of the semantic similarity model. New features introduced in .
  • An embodiment of the present application provides a terminal, including: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to: extract a binary feature from a corpus; calculate the binary feature Presetting statistics; selecting a preset number of binary features according to the preset statistics to form a binary feature dictionary.
  • the embodiment of the present application proposes a non-transitory computer readable storage medium, when the instructions in the storage medium are executed by a processor of the terminal, enabling the terminal to perform a method, the method comprising: extracting two from the corpus a meta-feature; calculating a preset statistic of the binary feature; sequentially selecting a preset number of binary features according to the preset statistic to form a binary feature dictionary.
  • the embodiment of the present application provides a computer program product, when an instruction in the computer program product is executed by a processor, performing a method, the method comprising: extracting a binary feature from a corpus; and calculating the binary feature a preset statistic; selecting a preset number of binary features in order according to the preset statistic to form a binary feature dictionary.
  • portions of the application can be implemented in hardware, software, firmware, or a combination thereof.
  • multiple steps or methods may be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system.
  • a suitable instruction execution system For example, if implemented in hardware, as in another embodiment, it can be implemented by any one or combination of the following techniques well known in the art: having logic gates for implementing logic functions on data signals. Discrete logic circuits, application specific integrated circuits with suitable combinational logic gates, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc.
  • each functional unit in each embodiment of the present application may be integrated into one processing module, or each unit may exist physically separately, or two or more units may be integrated into one module.
  • the above integrated modules can be implemented in the form of hardware or in the form of software functional modules.
  • the integrated modules, if implemented in the form of software functional modules and sold or used as stand-alone products, may also be stored in a computer readable storage medium.
  • the above mentioned storage medium may be a read only memory, a magnetic disk or an optical disk or the like.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Computational Linguistics (AREA)
  • General Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Health & Medical Sciences (AREA)
  • Data Mining & Analysis (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Mathematical Physics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Software Systems (AREA)
  • Databases & Information Systems (AREA)
  • Biophysics (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • Evolutionary Computation (AREA)
  • Biomedical Technology (AREA)
  • Computational Mathematics (AREA)
  • Mathematical Analysis (AREA)
  • Mathematical Optimization (AREA)
  • Pure & Applied Mathematics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Evolutionary Biology (AREA)
  • Operations Research (AREA)
  • Probability & Statistics with Applications (AREA)
  • Algebra (AREA)
  • Machine Translation (AREA)

Abstract

一种二元特征词典的构建方法和装置，该二元特征词典的构建方法包括：从语料中提取二元特征（S11）；计算所述二元特征的预设统计量（S12）；根据所述预设统计量按序选择预设个数的二元特征，组成二元特征词典（S13）。该方法能够构建出二元特征词典，从而可以在语义相似度模型的训练数据中引入新的特征。

Description

二元特征词典的构建方法和装置
相关申请的交叉引用
本申请要求北京百度网讯科技有限公司于2016年5月31日提交的、发明名称为“二元特征词典的构建方法和装置”的、中国专利申请号“201610379719.X”的优先权。
技术领域
本申请涉及自然语言处理技术领域，尤其涉及一种二元特征词典的构建方法和装置。
背景技术
搜索引擎是一种检索系统，能够向用户反馈检索结果。搜索引擎的一种实现是基于语义相似度模型，通过语义相似度模型可以计算用户检索的检索词(query)与候选的标题(title)之间的相似度，再根据相似度进行排序，以返回检索结果。语义相似度模型可以采用神经网络训练构建。
相关技术中，采用神经网络构建语义相似度模型时，训练数据通常是基础(basic)粒度的信息或短语(phrase)粒度的信息，其中，basic粒度的信息是对句子语料进行切词后，得到的最小粒度的分词，如“百度”，phrase粒度的信息 包括多个最小粒度的分词，如“巴西葡语”。为了提高语义相似度模型的精准度，还可以在训练数据中引入新的粒度信息。当需要引入新的粒度信息时，需要解决如何提取新的粒度信息的问题。
发明内容
本申请旨在至少在一定程度上解决相关技术中的技术问题之一。
为此，本申请的一个目的在于提出一种二元特征词典的构建方法，该方法可以构建出二元特征词典，从而可以采用该二元特征词典从模型语料中提取出二元特征，进一步的，该二元特征可以作为语义相似度模型的训练数据中引入的新特征。
本申请的另一个目的在于提出一种二元特征词典的构建装置。
为达到上述目的，本申请第一方面实施例提出的二元特征词典的构建方法，包括：从语料中提取二元特征；计算所述二元特征的预设统计量；根据所述预设统计量按序选择预设个数的二元特征，组成二元特征词典。
本申请第一方面实施例提出的二元特征词典的构建方法，通过计算二元特征的预设统计量，根据预设统计量选择二元特征组成二元特征词典，可以实现二元特征词典的构建，从而可以采用该二元特征词典从模型语料中提取出二元特征，进一步的，该二元特征可以作为语义相似度模型的训练数据中引入的新特征。
为达到上述目的，本申请第二方面实施例提出的二元特征词典的构建装置， 包括：提取模块，用于从语料中提取二元特征；计算模块，用于计算所述二元特征的预设统计量；生成模块，用于根据所述预设统计量按序选择预设个数的二元特征，组成二元特征词典。
本申请第二方面实施例提出的二元特征词典的构建装置，通过计算二元特征的预设统计量，根据预设统计量选择二元特征组成二元特征词典，可以实现二元特征词典的构建，从而可以采用该二元特征词典从模型语料中提取出二元特征，进一步的，该二元特征可以作为语义相似度模型的训练数据中引入的新特征。
本申请实施例提出一种终端，包括：处理器；用于存储处理器可执行指令的存储器；其中，所述处理器被配置为：从语料中提取二元特征；计算所述二元特征的预设统计量；根据所述预设统计量按序选择预设个数的二元特征，组成二元特征词典。
本申请实施例提出一种非临时性计算机可读存储介质，当所述存储介质中的指令由终端的处理器执行时，使得终端能够执行一种方法，所述方法包括：从语料中提取二元特征；计算所述二元特征的预设统计量；根据所述预设统计量按序选择预设个数的二元特征，组成二元特征词典。
本申请实施例提出一种计算机程序产品，当所述计算机程序产品中的指令被处理器执行时，执行一种方法，所述方法包括：从语料中提取二元特征；计算所述二元特征的预设统计量；根据所述预设统计量按序选择预设个数的二元 特征，组成二元特征词典。
本申请附加的方面和优点将在下面的描述中部分给出，部分将从下面的描述中变得明显，或通过本申请的实践了解到。
附图说明
本申请上述的和/或附加的方面和优点从下面结合附图对实施例的描述中将变得明显和容易理解，其中：
图1是本申请一个实施例提出的二元特征词典的构建方法的流程示意图；
图2是本申请另一个实施例提出的二元特征词典的构建方法的流程示意图；
图3是本申请另一个实施例提出的二元特征词典的构建方法的流程示意图；
图4是本申请实施例采用的神经网络的示意图；
图5是本发明一个实施例提出的二元特征词典的构建装置的结构示意图；
图6是本发明一个实施例提出的二元特征词典的构建装置的结构示意图。
具体实施方式
下面详细描述本申请的实施例，所述实施例的示例在附图中示出，其中自始至终相同或类似的标号表示相同或类似的模块或具有相同或类似功能的模块。下面通过参考附图描述的实施例是示例性的，仅用于解释本申请，而不能理解为对本申请的限制。相反，本申请的实施例包括落入所附加权利要求书的精神和内涵范围内的所有变化、修改和等同物。
在获取语义相似模型的训练数据时，可以先对句子语料进行切词，之后根据分词获取需要粒度的信息，如basic粒度的信息或phrase粒度的信息。
如上所示，在构建语义相似度模型时，除了上述的basic粒度的信息或phrase粒度的信息之外，还可以引入新粒度的信息。该引入的新粒度的信息可以称为二元特征。每个二元特征由两个basic粒度的信息或两个phrase粒度的信息组成。
当需要引入二元特征时，可以根据二元特征词典获取对应的二元特征。因此，需要解决二元特征词典如何建立的问题。
图1是本申请一个实施例提出的二元特征词典的构建方法的流程示意图。
如图1所示，本实施例的流程包括：
S11：从语料中提取二元特征。
其中，在构建二元特征词典时，可以先收集语料，与语义相似度模型的构建类似，构建二元特征词典时收集的语料也可以具体是句子语料，即，收集大量的句子作为语料。
在语料中提取二元特征时，可以具体是将语料中任意两个相邻的词条，作为一个二元特征。
具体的，对应每个句子语料，可以对句子语料进行切分，得到词条(term)。
其中，词条可以是basic粒度，或者，也可以是phrase粒度，但是，上述的两个词条需要具有同种粒度，例如，均是basic粒度，或者，均是phrase 粒度。将句子切分成basic粒度的词条或phrase粒度的词条可以采用各种已有技术，在此不再详细说明。
在将句子切分成词条后，可以将任意两个相邻的词条作为一个二元特征，因此，可以对应每个句子，假设该句子包括n个词条，则可以提取出n-1个二元特征。将语料中的所有句子对应的二元特征相加后，可以得到多个二元特征。
S12：计算所述二元特征的预设统计量。
本实施例中，以预设统计量是t统计量为例。
t统计量是对模型参数的单个假设进行检验的一种统计量。
具体的，一个二元特征的t统计量的计算公式是：
Figure PCTCN2016111571-appb-000001
其中，f是该二元特征在语料中的出现概率，例如，语料中共有M个句子，其中m个句子中存在该二元特征，即该二元特征中的两个词条相邻且位置关系不变的出现在m个句子中，则f＝m/M；
μ＝p1×p2，p1是该二元特征中一个词条在语料中的出现概率，p2是该二元特征中另一个词条在语料中的出现概率；
本实施例中，假设s2＝f；
N是语料中所有二元特征的总数，N＝∑Ni，N是单个句子语料对应的二元特征的数量，假设某个句子语料包括n个词条，则该句子语料对应的二元特征的数量是n-1。
因此，通过上述计算公式，可以计算出每个二元特征的t统计量。
S13：根据所述预设统计量按序选择预设个数的二元特征，组成二元特征词典。
例如，在得到每个二元特征的t统计量后，可以根据t统计量从大到小的顺序选择预设个数的二元特征，将选择的这些二元特征组成二元特征词典。
进一步的，如上所示，一个二元特征可以是由两个basic粒度的词条组成，或者，一个二元特征也可以是由两个phrase粒度的词条组成。在上述排序时，是对这些二元特征(包括根据basic粒度的词条得到的二元特征以及根据phrase粒度的词条得到的二元特征)进行统一排序的，从而在二元特征词典中可以包括根据basic粒度的词条得到的二元特征和/或根据phrase粒度的词条得到的二元特征。
通过上述的流程，可以实现二元特征词典的构建。
在构建出二元特征词典后，可以使用该词典生成二元特征，以用于语义相似度模型的训练。
相应的，一些实施例中，参见图2，本实施例的方法还可以包括：
S14：采用所述二元特征词典，从语义相似度模型的分词中提取二元特征，作为语义相似度模型的训练数据。
其中，可以将二元特征词典中包括的且可以由相邻的两个分词组成的二元特征，作为提取的二元特征。
另外，上述的分词可以是basic粒度的分词或phrase粒度的分词。
例如，语义相似度模块的一个语料是“百度巴西葡语”，经过切词后，以basic粒度的分词为例，各分词是“百度”、“巴西”、“葡语”，构建的二元特征词典中的一个二元特征是“巴西葡语”，则提取的一个二元特征是“巴西葡语”。
进一步的，如何二元特征词典中不包括任意两个相邻分词组成的二元特征，则直接将分词作为提取的二元特征。依然以上述的语料为例，假设构建的二元特征词典中不包括“巴西葡语”，也不包括“百度巴西”，则提取的二元特征就是原始的分词，如“百度”、“巴西”、“葡语”。
因此，通过二元特征词典，可以实现二元特征的提取，从而在训练语义相似度模型时可以引入新的特征。
一些实施例中，参见图3，本实施例的方法还可以包括：
S15：根据所述训练数据进行神经网络训练，生成语义相似度模型。
其中，神经网络可以如图4所示，与通常的语义相似度模型不同的是，本实施例的输入可以包括二元特征。
因此，可以通过引入二元特征训练语义相似度模型。
本实施例中，通过计算二元特征的预设统计量，根据预设统计量选择二元特征组成二元特征词典，可以实现二元特征词典的构建，从而可以采用该二元特征词典从模型语料中提取出二元特征，进一步的，该二元特征可以作为语义相似度模型的训练数据中引入的新特征。
图5是本发明一个实施例提出的二元特征词典的构建装置的结构示意图。
参见图5，本实施例的装置包括：提取模块51、计算模块52和生成模块53。
提取模块51，用于从语料中提取二元特征；
计算模块52，用于计算所述二元特征的预设统计量；
生成模块53，用于根据所述预设统计量按序选择预设个数的二元特征，组成二元特征词典。
一些实施例中，所述提取模块51具体用于：
将语料中任意两个相邻的词条，作为一个二元特征。
一些实施例中，所述计算模块52计算的所述预设统计量是t统计量。
一些实施例中，参见图6，本实施例的装置还可以包括：
获取模块54，用于采用所述二元特征词典，从语义相似度模型的分词中提取二元特征，作为语义相似度模型的训练数据。
一些实施例中，参见图6，本实施例的装置还可以包括：
训练模块55，用于根据所述训练数据进行神经网络训练，生成语义相似度模型。
可以理解的是，本实施例的装置与方法实施例对应，具体内容可以参见方法实施例中的相关描述，在此不再详细说明。
本实施例中，通过计算二元特征的预设统计量，根据预设统计量选择二元 特征组成二元特征词典，可以实现二元特征词典的构建，从而可以采用该二元特征词典从模型语料中提取出二元特征，进一步的，该二元特征可以作为语义相似度模型的训练数据中引入的新特征。
可以理解的是，上述各实施例中相同或相似部分可以相互参考，在一些实施例中未详细说明的内容可以参见其他实施例中相同或相似的内容。
本申请实施例提出一种终端，包括：处理器；用于存储处理器可执行指令的存储器；其中，所述处理器被配置为：从语料中提取二元特征；计算所述二元特征的预设统计量；根据所述预设统计量按序选择预设个数的二元特征，组成二元特征词典。
本申请实施例提出一种非临时性计算机可读存储介质，当所述存储介质中的指令由终端的处理器执行时，使得终端能够执行一种方法，所述方法包括：从语料中提取二元特征；计算所述二元特征的预设统计量；根据所述预设统计量按序选择预设个数的二元特征，组成二元特征词典。
本申请实施例提出一种计算机程序产品，当所述计算机程序产品中的指令被处理器执行时，执行一种方法，所述方法包括：从语料中提取二元特征；计算所述二元特征的预设统计量；根据所述预设统计量按序选择预设个数的二元特征，组成二元特征词典。
需要说明的是，在本申请的描述中，术语“第一”、“第二”等仅用于描述目的，而不能理解为指示或暗示相对重要性。此外，在本申请的描述中，除 非另有说明，“多个”的含义是指至少两个。
流程图中或在此以其他方式描述的任何过程或方法描述可以被理解为，表示包括一个或更多个用于实现特定逻辑功能或过程的步骤的可执行指令的代码的模块、片段或部分，并且本申请的优选实施方式的范围包括另外的实现，其中可以不按所示出或讨论的顺序，包括根据所涉及的功能按基本同时的方式或按相反的顺序，来执行功能，这应被本申请的实施例所属技术领域的技术人员所理解。
应当理解，本申请的各部分可以用硬件、软件、固件或它们的组合来实现。在上述实施方式中，多个步骤或方法可以用存储在存储器中且由合适的指令执行系统执行的软件或固件来实现。例如，如果用硬件来实现，和在另一实施方式中一样，可用本领域公知的下列技术中的任一项或他们的组合来实现：具有用于对数据信号实现逻辑功能的逻辑门电路的离散逻辑电路，具有合适的组合逻辑门电路的专用集成电路，可编程门阵列(PGA)，现场可编程门阵列(FPGA)等。
本技术领域的普通技术人员可以理解实现上述实施例方法携带的全部或部分步骤是可以通过程序来指令相关的硬件完成，所述的程序可以存储于一种计算机可读存储介质中，该程序在执行时，包括方法实施例的步骤之一或其组合。
此外，在本申请各个实施例中的各功能单元可以集成在一个处理模块中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个模块 中。上述集成的模块既可以采用硬件的形式实现，也可以采用软件功能模块的形式实现。所述集成的模块如果以软件功能模块的形式实现并作为独立的产品销售或使用时，也可以存储在一个计算机可读取存储介质中。
上述提到的存储介质可以是只读存储器，磁盘或光盘等。
在本说明书的描述中，参考术语“一个实施例”、“一些实施例”、“示例”、“具体示例”、或“一些示例”等的描述意指结合该实施例或示例描述的具体特征、结构、材料或者特点包含于本申请的至少一个实施例或示例中。在本说明书中，对上述术语的示意性表述不一定指的是相同的实施例或示例。而且，描述的具体特征、结构、材料或者特点可以在任何的一个或多个实施例或示例中以合适的方式结合。
尽管上面已经示出和描述了本申请的实施例，可以理解的是，上述实施例是示例性的，不能理解为对本申请的限制，本领域的普通技术人员在本申请的范围内可以对上述实施例进行变化、修改、替换和变型。

Claims (13)

  1. 一种二元特征词典的构建方法，其特征在于，包括：
    从语料中提取二元特征；
    计算所述二元特征的预设统计量；
    根据所述预设统计量按序选择预设个数的二元特征，组成二元特征词典。
  2. 根据权利要求1所述的方法，其特征在于，所述从语料中提取二元特征，包括：
    将语料中任意两个相邻的词条，作为一个二元特征。
  3. 根据权利要求1-2任一项所述的方法，其特征在于，所述预设统计量是t统计量。
  4. 根据权利要求1-3任一项所述的方法，其特征在于，还包括：
    采用所述二元特征词典，从语义相似度模型的分词中提取二元特征，作为语义相似度模型的训练数据。
  5. 根据权利要求4所述的方法，其特征在于，还包括：
    根据所述训练数据进行神经网络训练，生成语义相似度模型。
  6. 一种二元特征词典的构建装置，其特征在于，包括：
    提取模块，用于从语料中提取二元特征；
    计算模块，用于计算所述二元特征的预设统计量；
    生成模块，用于根据所述预设统计量按序选择预设个数的二元特征，组成二元特征词典。
  7. 根据权利要求6所述的装置，其特征在于，所述提取模块具体用于：
    将语料中任意两个相邻的词条，作为一个二元特征。
  8. 根据权利要求6-7任一项所述的装置，其特征在于，所述计算模块计算的所述预设统计量是t统计量。
  9. 根据权利要求6-8任一项所述的装置，其特征在于，还包括：
    获取模块，用于采用所述二元特征词典，从语义相似度模型的分词中提取二元特征，作为语义相似度模型的训练数据。
  10. 根据权利要求9所述的装置，其特征在于，还包括：
    训练模块，用于根据所述训练数据进行神经网络训练，生成语义相似度模型。
  11. 一种终端，其特征在于，包括：处理器；用于存储处理器可执行指令的存储器；其中，所述处理器被配置为：
    从语料中提取二元特征；
    计算所述二元特征的预设统计量；
    根据所述预设统计量按序选择预设个数的二元特征，组成二元特征词典。
  12. 一种非临时性计算机可读存储介质，其特征在于，当所述存储介质中的指令由终端的处理器执行时，使得终端能够执行一种方法，所述方法包括：
    从语料中提取二元特征；
    计算所述二元特征的预设统计量；
    根据所述预设统计量按序选择预设个数的二元特征，组成二元特征词典。
  13. 一种计算机程序产品，其特征在于，当所述计算机程序产品中的指令被处理器执行时，执行一种方法，所述方法包括：
    从语料中提取二元特征；
    计算所述二元特征的预设统计量；
    根据所述预设统计量按序选择预设个数的二元特征，组成二元特征词典。
PCT/CN2016/111571 2016-05-31 2016-12-22 二元特征词典的构建方法和装置 Ceased WO2017206492A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US16/306,488 US10831993B2 (en) 2016-05-31 2016-12-22 Method and apparatus for constructing binary feature dictionary

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201610379719.X 2016-05-31
CN201610379719.XA CN106021572B (zh) 2016-05-31 2016-05-31 二元特征词典的构建方法和装置

Publications (1)

Publication Number Publication Date
WO2017206492A1 true WO2017206492A1 (zh) 2017-12-07

Family

ID=57092120

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2016/111571 Ceased WO2017206492A1 (zh) 2016-05-31 2016-12-22 二元特征词典的构建方法和装置

Country Status (3)

Country Link
US (1) US10831993B2 (zh)
CN (1) CN106021572B (zh)
WO (1) WO2017206492A1 (zh)

Families Citing this family (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106021572B (zh) 2016-05-31 2019-05-31 北京百度网讯科技有限公司 二元特征词典的构建方法和装置
CN106815311B (zh) * 2016-12-21 2021-08-06 杭州朗和科技有限公司 一种问题匹配方法和装置
US10860803B2 (en) 2017-05-07 2020-12-08 8X8, Inc. System for semantic determination of job titles
CN109558781B (zh) * 2018-08-02 2021-07-02 北京市商汤科技开发有限公司 一种多视角视频识别方法及装置、设备和存储介质
US11636334B2 (en) 2019-08-20 2023-04-25 Micron Technology, Inc. Machine learning with feature obfuscation
US11392796B2 (en) 2019-08-20 2022-07-19 Micron Technology, Inc. Feature dictionary for bandwidth enhancement
US11755884B2 (en) 2019-08-20 2023-09-12 Micron Technology, Inc. Distributed machine learning with privacy protection
CN111291559B (zh) * 2020-01-22 2023-04-11 中国民航信息网络股份有限公司 姓名文本处理方法及装置、存储介质及电子设备
CN114492434B (zh) * 2022-01-27 2022-10-11 圆通速递有限公司 一种基于运单号自动识别模型智能识别运单号方法
CN115270774B (zh) * 2022-09-27 2023-01-03 吉奥时空信息技术股份有限公司 一种半监督学习的大数据关键词词典构建方法
CN119670745B (zh) * 2024-11-26 2025-10-21 中国工商银行股份有限公司 特征词典的构建方法及其装置、电子设备、程序产品

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140289238A1 (en) * 2013-03-21 2014-09-25 Kabushiki Kaisha Toshiba Document creation support apparatus, method and program
CN104375989A (zh) * 2014-12-01 2015-02-25 国家电网公司 自然语言文本关键词关联网络构建系统
CN104391963A (zh) * 2014-12-01 2015-03-04 北京中科创益科技有限公司 一种自然语言文本关键词关联网络构建方法
CN104899190A (zh) * 2015-06-04 2015-09-09 百度在线网络技术（北京）有限公司 分词词典的生成方法和装置及分词处理方法和装置
CN106021572A (zh) * 2016-05-31 2016-10-12 北京百度网讯科技有限公司 二元特征词典的构建方法和装置

Family Cites Families (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6173298B1 (en) * 1996-09-17 2001-01-09 Asap, Ltd. Method and apparatus for implementing a dynamic collocation dictionary
US7555523B1 (en) * 2004-05-06 2009-06-30 Symantec Corporation Spam discrimination by generalized Ngram analysis of small header fields
US20110035211A1 (en) * 2009-08-07 2011-02-10 Tal Eden Systems, methods and apparatus for relative frequency based phrase mining
US9483460B2 (en) * 2013-10-07 2016-11-01 Google Inc. Automated formation of specialized dictionaries
CN105488077B (zh) * 2014-10-10 2020-04-28 腾讯科技（深圳）有限公司 生成内容标签的方法和装置
US10115055B2 (en) * 2015-05-26 2018-10-30 Booking.Com B.V. Systems methods circuits and associated computer executable code for deep learning based natural language understanding
US20170085509A1 (en) * 2015-09-17 2017-03-23 Vicente Fernandez Semantics classification aggregation newsfeed, an automated distribution method
US9659248B1 (en) * 2016-01-19 2017-05-23 International Business Machines Corporation Machine learning and training a computer-implemented neural network to retrieve semantically equivalent questions using hybrid in-memory representations
US11023680B2 (en) * 2016-02-17 2021-06-01 The King Abdulaziz City For Science And Technology (Kacst) Method and system for detecting semantic errors in a text using artificial neural networks

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140289238A1 (en) * 2013-03-21 2014-09-25 Kabushiki Kaisha Toshiba Document creation support apparatus, method and program
CN104375989A (zh) * 2014-12-01 2015-02-25 国家电网公司 自然语言文本关键词关联网络构建系统
CN104391963A (zh) * 2014-12-01 2015-03-04 北京中科创益科技有限公司 一种自然语言文本关键词关联网络构建方法
CN104899190A (zh) * 2015-06-04 2015-09-09 百度在线网络技术（北京）有限公司 分词词典的生成方法和装置及分词处理方法和装置
CN106021572A (zh) * 2016-05-31 2016-10-12 北京百度网讯科技有限公司 二元特征词典的构建方法和装置

Also Published As

Publication number Publication date
CN106021572A (zh) 2016-10-12
US10831993B2 (en) 2020-11-10
CN106021572B (zh) 2019-05-31
US20190163737A1 (en) 2019-05-30

Similar Documents

Publication Publication Date Title
CN110874531B (zh) 一种话题分析方法、装置和存储介质
CN106528845B (zh) 基于人工智能的检索纠错方法及装置
US10831993B2 (en) Method and apparatus for constructing binary feature dictionary
WO2022095374A1 (zh) 关键词抽取方法、装置、终端设备及存储介质
CN104199833B (zh) 一种网络搜索词的聚类方法和聚类装置
CN110390006A (zh) 问答语料生成方法、装置和计算机可读存储介质
CN111291177A (zh) 一种信息处理方法、装置和计算机存储介质
CN108009228A (zh) 一种内容标签的设置方法、装置及存储介质
WO2016179938A1 (zh) 题目推荐方法和题目推荐装置
CN106446109A (zh) 语音文件摘要的获取方法和装置
CN108280164B (zh) 一种基于类别相关单词的短文本过滤与分类方法
CN107301170A (zh) 基于人工智能的切分语句的方法和装置
US11379527B2 (en) Sibling search queries
CN110598209B (zh) 用于提取关键词的方法、系统及存储介质
KR102685135B1 (ko) 영상 편집 자동화 시스템
CN108108353A (zh) 一种基于弹幕的视频语义标注方法、装置及电子设备
Twinandilla et al. Multi-document summarization using k-means and latent dirichlet allocation (lda)–significance sentences
CN110263345B (zh) 关键词提取方法、装置及存储介质
CN109033066A (zh) 一种摘要形成方法及装置
CN115757760A (zh) 文本摘要提取方法及系统、计算设备、存储介质
CN105630769A (zh) 文档主题词提取方法及装置
CN104615654B (zh) 一种文本摘要获取方法及装置
US11990131B2 (en) Method for processing a video file comprising audio content and visual content comprising text content
CN121188178A (zh) 查询语句的检索处理方法、装置、设备、介质及程序产品
CN111324705A (zh) 自适应性调整关连搜索词的系统及其方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16903872

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 16903872

Country of ref document: EP

Kind code of ref document: A1