WO2017215339A1 - 基于rbf神经网络的搜索集群优化方法及系统 - Google Patents

基于rbf神经网络的搜索集群优化方法及系统 Download PDF

Info

Publication number
WO2017215339A1
WO2017215339A1 PCT/CN2017/080780 CN2017080780W WO2017215339A1 WO 2017215339 A1 WO2017215339 A1 WO 2017215339A1 CN 2017080780 W CN2017080780 W CN 2017080780W WO 2017215339 A1 WO2017215339 A1 WO 2017215339A1
Authority
WO
WIPO (PCT)
Prior art keywords
node
neural network
rbf neural
search
data
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2017/080780
Other languages
English (en)
French (fr)
Inventor
白凡
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Wuhan Douyu Network Technology Co Ltd
Original Assignee
Wuhan Douyu Network Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Wuhan Douyu Network Technology Co Ltd filed Critical Wuhan Douyu Network Technology Co Ltd
Publication of WO2017215339A1 publication Critical patent/WO2017215339A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/09—Supervised learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/95—Retrieval from the web
    • G06F16/951—Indexing; Web crawling techniques
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0499—Feedforward networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods

Definitions

  • the invention relates to the field of big data search, in particular to a search cluster optimization method and system based on RBF neural network.
  • ElasticSearch is a Lucene-based search server.
  • Lucene is a sub-project of the 4jakarta project team of the Apache Software Foundation. It is an open source full-text search engine toolkit, which is not a complete full-text search engine, but a full-text search engine architecture that provides a complete query. Engine and index engine, part of the text analysis engine (English and German two Western languages). Lucene's goal is to provide software developers with an easy-to-use toolkit for easy full-text search in the target system, or to build a complete full-text search engine based on this.
  • ElasticSearch provides a distributed multi-user capable full-text search engine based on a RESTful web interface.
  • Elasticsearch was developed in Java and released as an open source under the Apache license. It is a popular enterprise search engine designed for cloud computing, enabling real-time search, stable, reliable, fast, and easy to install and use.
  • ElasticSearch uses Lucene as its core to implement all indexing and search functions. ElasticSearch hides the complexity of Lucene with a simple RESTful API, making full-text search easy.
  • Full-text search Different from the traditional fuzzy matching, it first matches the search words according to certain rules, then matches the word segment with the source data, and scores according to the number of occurrences of the word segmentation, the distance between the segmentation, the weight and other data to obtain the search result.
  • Search node A single ElasticSearch server node.
  • Management node The master node of ElasticSearch, used to manage cluster data synchronization, message distribution, etc.
  • ElasticSearch's data node which does not do any data management operations, and is only used to store cluster data.
  • Cluster cluster consists of multiple search nodes, including one management node and multiple data nodes.
  • Cluster A collection of complete ElasticSearch nodes consisting of multiple clusters of clusters.
  • RBF networks can approximate arbitrary nonlinear functions, can handle difficult-to-analyze regularities in the system, have good generalization ability, and have fast learning convergence speed. For nonlinear function approximation, time series analysis, data classification, pattern recognition, information processing, image processing, system modeling, control and fault diagnosis.
  • Sample set Sample data for RBF neural network training.
  • Training result set Convergence result data obtained by the RBF neural network through training of the sample set.
  • Search feedback Search effect feedback, that is, after the user enters the search word into the search page, whether there is a click on the page link or click the link after turning the page multiple times.
  • ElasticSearch has three main features:
  • (3) can be extended to hundreds of servers to handle PB-level structured or unstructured data.
  • ElasticSearch can easily achieve horizontal expansion of server nodes, and the ElasticSearch node can specifically set the characteristics of the master (management node) and data (data node) to facilitate the management of big data search clusters.
  • the object of the present invention is to overcome the deficiencies of the above background art, and to provide a search cluster optimization method and system based on RBF neural network.
  • a management node fails, the training result data of the failed data node under the cluster is automatically extracted, and A management node suitable for the failed node is found, and a connection request process is established between the failed data node and the management node, and the cluster is automatically applied for joining.
  • the invention provides a search cluster optimization method based on RBF neural network, which comprises the following steps:
  • the management node sends a request to join the cluster
  • step S2 determining whether the management node has a response within a predetermined time period. If the management node does not respond within a predetermined time period, returning to step S1, performing RBF neural network training on the failed data node; if the management node has a specified duration In response, go to step S3;
  • the management node analyzes the number of data nodes of the current cluster group, and determines the cluster group. Whether the number of data nodes reaches the upper limit, if the upper limit is reached, return to step S1, and perform RBF neural network training on the failed data node; if the upper limit is not reached, go to step S4;
  • the management node replies to the data node applying for joining, establishes a connection with the data node, and notifies the entire search cluster.
  • the search cluster is performed based on the RBF neural network, and several radial basis functions are selected, and each radial basis function corresponds to one training data.
  • X is the input, the number of inputs is m, m is a positive integer; P is the number of radial basis functions, P is a positive integer, and P>m; input X is an m-dimensional vector, sample size is P; p is the Pth input, Is the Pth radial basis function and X p is the radial basis function Center of ⁇ P is the Pth weight vector, (
  • the index number of the data node, the search feedback, the search CPU consumption, the search response duration, and the node load pressure data are used as input vector sets, and timing is performed. Training and calculation, set the custom error range, number of iterations and weights, train to get the clusters and management nodes that are most suitable for you.
  • the invention also provides a search cluster optimization system based on RBF neural network, the system comprises a data node, a management node and an RBF neural network training unit, wherein:
  • the RBF neural network training unit uses the index number of the data node, the search feedback, the search CPU consumption, the search response duration, and the node load pressure data as the training sample set, performs the RBF neural network training, matches the optimal management node, and obtains the invalidation.
  • the RBF neural network training unit performs RBF neural network training on the failed data node; if the management node has a specified duration
  • the management node analyzes the number of data nodes of the current cluster to determine whether the number of data nodes of the cluster reaches an upper limit. If the upper limit is reached, the RBF neural network training unit performs RBF neural network training on the failed data node; If the upper limit is not reached, the management node replies to the requesting data node, establishes a connection with the data node, and notifies the entire search cluster.
  • the RBF neural network training unit performs RBF neural network training, searches for clusters based on the RBF neural network, selects several radial basis functions, and each radial basis function corresponds to one training data. .
  • X is the input, the number of inputs is m, m is a positive integer; P is the number of radial basis functions, P is a positive integer, and P>m; input X is an m-dimensional vector, sample size is P; p is the Pth input, Is the Pth radial basis function and X p is the radial basis function Center of ⁇ P is the Pth weight vector, (
  • the RBF neural network training unit performs the RBF neural network training process, and uses the index number of the data node, the search feedback, the search CPU consumption, the search response duration, and the node load pressure data as input vectors. Set, regularly train and calculate, set the custom error range, iteration number and weight, train to get the cluster and management node that is most suitable for you.
  • the system automatically extracts the training result data of the failed data node under the cluster, and finds a management node suitable for the failed node, and establishes between the failed data node and the management node.
  • the connection request process automatically applies to join the cluster, and the upper limit of the cluster node is customized.
  • the invention performs search clustering based on the RBF neural network, and uses the data node index number, search feedback, search CPU consumption, search response duration, node load pressure and the like as an input vector set, regularly performs training and calculation, and sets a custom error range.
  • the number of iterations and the weights are trained to get the clusters and management nodes that are most suitable for them.
  • the system automatically extracts the training result data of the failed data node under the cluster, finds the management node suitable for the failed node, establishes a connection request process between the failed data node and the management node, and automatically applies for joining the cluster. Group, cluster node upper limit is custom.
  • FIG. 1 is a flowchart of a search cluster optimization method based on RBF neural network in an embodiment of the present invention.
  • Figure 2 is a schematic diagram of RBF neural network training.
  • an embodiment of the present invention provides a search cluster optimization method based on an RBF neural network, including the following steps:
  • a suitable management node sends a request to join the cluster
  • step S2 determining whether the management node has a response within a predetermined time period. If the management node does not respond within a predetermined time period, returning to step S1, performing RBF neural network training on the failed data node; if the management node has a specified duration In response, go to step S3;
  • the management node analyzes the number of data nodes of the current cluster group, determines whether the number of data nodes of the cluster group reaches an upper limit, and if the upper limit is reached, returns to step S1, and performs RBF neural network training on the failed data node; if not, Upper limit, then go to step S4;
  • the management node replies to the data node applying for joining, establishes a connection with the data node, and notifies the entire search cluster.
  • the embodiment of the invention further provides a search cluster optimization system based on the RBF neural network, comprising a data node, a management node and an RBF neural network training unit, wherein:
  • the RBF neural network training unit uses the data node index number, search feedback, search CPU consumption, search response duration, node load pressure and other data as the training sample set, performs RBF neural network training, matches the optimal management node, and obtains the invalid data.
  • a node sending a request to the appropriate management node to join the cluster;
  • the RBF neural network training unit determines whether the management node has a response within the specified length of time, if the management node is in the regulation There is no response within the duration, and the RBF neural network training unit performs RBF neural network training on the failed data node; if the management node has a response within the specified duration, the management node analyzes the number of data nodes of the current cluster, and determines the cluster. Whether the number of data nodes of the group reaches the upper limit. If the upper limit is reached, the RBF neural network training unit performs RBF neural network training on the failed data node; if the upper limit is not reached, the management node replies to the data node applying for joining, and The data node establishes a connection and notifies the entire search cluster.
  • a search cluster is performed based on the RBF neural network, and a plurality of radial basis functions are selected, and each radial basis function corresponds to one training data.
  • the radial basis function core algorithm is its interpolation function, and the formula of the interpolation function is as follows:
  • X is the input, the number of inputs is m, m is a positive integer; P is the number of radial basis functions, P is a positive integer, and P>m; input X is an m-dimensional vector, sample size is P; p is the Pth input, Is the Pth radial basis function and X p is the radial basis function Center of ⁇ P is the Pth weight vector, (
  • the data node indexes, search feedback, search CPU consumption, search response duration, and node load pressure are used as input vector sets, and timing is performed. Training and calculation, set the custom error range, number of iterations and weights, and train to get the clusters and management nodes that are most suitable for you.
  • the system automatically extracts the training result data of the failed data node under the cluster, finds the management node suitable for the failed node, establishes a connection request process between the failed data node and the management node, and automatically applies for joining the cluster.
  • cluster node upper limit is custom.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Biophysics (AREA)
  • Evolutionary Computation (AREA)
  • Biomedical Technology (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • Computational Linguistics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Databases & Information Systems (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)
  • Feedback Control In General (AREA)

Abstract

一种基于RBF神经网络的搜索集群优化方法及系统，涉及大数据搜索领域。该系统包括数据节点、管理节点和RBF神经网络训练单元，RBF神经网络训练单元基于RBF神经网络进行搜索集群，将数据节点的索引数量、搜索反馈、搜索CPU消耗、搜索响应时长、节点负载压力等数据，作为输入向量集，定时进行训练与计算，设置自定义误差范围、迭代次数及权值，训练得到最适合自己的簇群及管理节点。当某个管理节点失效，系统自动取出其簇群下失效数据节点的训练结果数据，并找到适合该失效节点的管理节点，在失效数据节点与管理节点之间建立连接请求流程，自动申请加入簇群。

Description

基于RBF神经网络的搜索集群优化方法及系统 技术领域
本发明涉及大数据搜索领域，具体是涉及一种基于RBF神经网络的搜索集群优化方法及系统。
背景技术
ElasticSearch是一个基于Lucene的搜索服务器。Lucene是apache软件基金会4jakarta项目组的一个子项目，是一个开放源代码的全文检索引擎工具包，即它不是一个完整的全文检索引擎，而是一个全文检索引擎的架构，提供了完整的查询引擎和索引引擎，部分文本分析引擎(英文与德文两种西方语言)。Lucene的目的是为软件开发人员提供一个简单易用的工具包，以方便的在目标系统中实现全文检索的功能，或者是以此为基础建立起完整的全文检索引擎。
ElasticSearch提供了一个分布式多用户能力的全文搜索引擎，基于RESTful web接口。Elasticsearch是用Java开发的，并作为Apache许可条款下的开放源码发布，是当前流行的企业级搜索引擎，设计用于云计算中，能够达到实时搜索，稳定，可靠，快速，安装使用方便。
ElasticSearch采用Lucene作为其核心，来实现所有索引和搜索的功能。ElasticSearch通过简单的RESTful API，来隐藏Lucene的复杂性，从而让全文搜索变得简单。
下面简单介绍一些ElasticSearch的相关术语。
全文检索：不同于传统的模糊匹配，其先对搜索词按一定规则分词后，再将分词与源数据进行匹配，根据出现分词次数，分词邻近距离，权重等数据进行打分，获得检索结果。
搜索节点：单个ElasticSearch服务器节点。
管理节点：ElasticSearch的master节点，用于管理集群数据同步，消息分发等。
数据节点：ElasticSearch的data节点，不做任何数据管理操作，仅用于存储集群数据。
集群簇：由多个搜索节点组成，包含一个管理节点及多个数据节点。
集群：由多个集群簇组成的完整ElasticSearch节点集合。
径向基函数神经网络(RBF)：RBF网络能够逼近任意的非线性函数，可以处理系统内的难以解析的规律性，具有良好的泛化能力，并有很快的学习收敛速度，已成功应用于非线性函数逼近、时间序列分析、数据分类、模式识别、信息处理、图像处理、系统建模、控制和故障诊断等。
样本集：RBF神经网络训练的样本数据。
训练结果集：RBF神经网络通过样本集的训练得到的收敛结果数据。
搜索反馈：搜索效果反馈，即用户输入搜索词进入搜索页面后，是否有点击页面链接亦或在多次翻页之后点击链接。
ElasticSearch主要具备以下三个特点：
(1)分布式的实时文件存储，每个字段都被索引并可被搜索；
(2)分布式的实时分析搜索引擎；
(3)可以扩展到上百台服务器，处理PB级结构化或非结构化 数据。
因此，ElasticSearch可以轻松实现服务器节点的水平拓展，并且，ElasticSearch节点可专门设置master(管理节点)以及data(数据节点)的特性，为管理大数据搜索集群提供便利。
然而，在大数据搜索集群中，由于数据量庞大，节点繁多，一但某个master(管理节点)挂掉，其下data(数据节点)如何有效自主地加入到其他管理节点的簇群是ElasticSearch搜索集群领域一个亟待解决的重要问题。
发明内容
本发明的目的是为了克服上述背景技术的不足，提供一种基于RBF神经网络的搜索集群优化方法及系统，当某个管理节点失效，自动取出其簇群下失效数据节点的训练结果数据，并找到适合该失效节点的管理节点，在失效数据节点与管理节点之间建立连接请求流程，自动申请加入簇群。
本发明提供一种基于RBF神经网络的搜索集群优化方法，包括以下步骤：
S1、以数据节点的索引数量、搜索反馈、搜索CPU消耗、搜索响应时长、节点负载压力数据作为训练样本集，进行RBF神经网络训练，匹配最优的管理节点，得到失效的数据节点，向合适的管理节点发送申请加入簇群的请求；
S2、判断管理节点在规定时长内是否有响应，若该管理节点在规定时长内无响应，则返回步骤S1，对失效的数据节点重新进行RBF神经网络训练；若该管理节点在规定时长内有响应，则转到步骤S3；
S3、管理节点对当前簇群的数据节点数量进行分析，判断簇群的 数据节点数量是否达到上限，若达到上限，则返回步骤S1，对失效的数据节点重新进行RBF神经网络训练；若未达到上限，则转到步骤S4；
S4、管理节点向申请加入的数据节点回复请求，与该数据节点建立连接，并通知整个搜索集群。
在上述技术方案的基础上，步骤S1中进行RBF神经网络训练的过程中，基于RBF神经网络进行搜索集群，选择若干个径向基函数，每个径向基函数对应一个训练数据。
在上述技术方案的基础上，所述径向基函数的插值函数的公式如下：
Figure PCTCN2017080780-appb-000001
其中，X是输入，输入的次数是m，m为正整数；P是径向基函数的数量，P为正整数，且P>m；输入X是个m维的向量，样本容量为P；Xp是第P次输入，
Figure PCTCN2017080780-appb-000002
是第P个径向基函数，Xp是径向基函数
Figure PCTCN2017080780-appb-000003
的中心；ωP是第P个权值向量，(||X-Xp||)表示差向量的模；y是输出的训练结果。
在上述技术方案的基础上，步骤S1中进行RBF神经网络训练的过程中，将数据节点的索引数量、搜索反馈、搜索CPU消耗、搜索响应时长、节点负载压力数据，作为输入向量集，定时进行训练与计算，设置自定义误差范围、迭代次数及权值，训练得到最适合自己的簇群及管理节点。
在上述技术方案的基础上，当某个管理节点失效，自动取出其簇群下失效数据节点的训练结果数据，并找到适合该失效节点的管理节点，在失效数据节点与管理节点之间建立连接请求流程，自动 申请加入簇群，簇群节点上限数为自定义。
本发明还提供一种基于RBF神经网络的搜索集群优化系统，该系统包括数据节点、管理节点和RBF神经网络训练单元，其中：
所述RBF神经网络训练单元以数据节点的索引数量、搜索反馈、搜索CPU消耗、搜索响应时长、节点负载压力数据作为训练样本集，进行RBF神经网络训练，匹配最优的管理节点，得到失效的数据节点，向合适的管理节点发送申请加入簇群的请求；
判断管理节点在规定时长内是否有响应，若该管理节点在规定时长内无响应，所述RBF神经网络训练单元对失效的数据节点重新进行RBF神经网络训练；若该管理节点在规定时长内有响应，管理节点对当前簇群的数据节点数量进行分析，判断簇群的数据节点数量是否达到上限，若达到上限，所述RBF神经网络训练单元对失效的数据节点重新进行RBF神经网络训练；若未达到上限，管理节点向申请加入的数据节点回复请求，与该数据节点建立连接，并通知整个搜索集群。
在上述技术方案的基础上，所述RBF神经网络训练单元进行RBF神经网络训练的过程中，基于RBF神经网络进行搜索集群，选择若干个径向基函数，每个径向基函数对应一个训练数据。
在上述技术方案的基础上，所述径向基函数的插值函数的公式如下：
Figure PCTCN2017080780-appb-000004
其中，X是输入，输入的次数是m，m为正整数；P是径向基函数的数量，P为正整数，且P>m；输入X是个m维的向量，样本容量为P；Xp是第P次输入，
Figure PCTCN2017080780-appb-000005
是第P个径向基函数，Xp是径向基函 数
Figure PCTCN2017080780-appb-000006
的中心；ωP是第P个权值向量，(||X-Xp||)表示差向量的模；y是输出的训练结果。
在上述技术方案的基础上，所述RBF神经网络训练单元进行RBF神经网络训练的过程中，将数据节点的索引数量、搜索反馈、搜索CPU消耗、搜索响应时长、节点负载压力数据，作为输入向量集，定时进行训练与计算，设置自定义误差范围、迭代次数及权值，训练得到最适合自己的簇群及管理节点。
在上述技术方案的基础上，当某个管理节点失效，系统自动取出其簇群下失效数据节点的训练结果数据，并找到适合该失效节点的管理节点，在失效数据节点与管理节点之间建立连接请求流程，自动申请加入簇群，簇群节点上限数为自定义。
与现有技术相比，本发明的优点如下：
本发明基于RBF神经网络进行搜索集群，将数据节点的索引数量、搜索反馈、搜索CPU消耗、搜索响应时长、节点负载压力等数据，作为输入向量集，定时进行训练与计算，设置自定义误差范围、迭代次数及权值，训练得到最适合自己的簇群及管理节点。当某个管理节点失效，系统自动取出其簇群下失效数据节点的训练结果数据，并找到适合该失效节点的管理节点，在失效数据节点与管理节点之间建立连接请求流程，自动申请加入簇群，簇群节点上限数为自定义。
附图说明
图1是本发明实施例中基于RBF神经网络的搜索集群优化方法的流程图。
图2是RBF神经网络训练的示意图。
具体实施方式
下面结合附图及具体实施例对本发明作进一步的详细描述。
参见图1所示，本发明实施例提供一种基于RBF神经网络的搜索集群优化方法，包括以下步骤：
S1、以数据节点的索引数量、搜索反馈、搜索CPU消耗、搜索响应时长、节点负载压力等数据作为训练样本集，进行RBF神经网络训练，匹配最优的管理节点，得到失效的数据节点，向合适的管理节点发送申请加入簇群的请求；
S2、判断管理节点在规定时长内是否有响应，若该管理节点在规定时长内无响应，则返回步骤S1，对失效的数据节点重新进行RBF神经网络训练；若该管理节点在规定时长内有响应，则转到步骤S3；
S3、管理节点对当前簇群的数据节点数量进行分析，判断簇群的数据节点数量是否达到上限，若达到上限，则返回步骤S1，对失效的数据节点重新进行RBF神经网络训练；若未达到上限，则转到步骤S4；
S4、管理节点向申请加入的数据节点回复请求，与该数据节点建立连接，并通知整个搜索集群。
本发明实施例还提供一种基于RBF神经网络的搜索集群优化系统，包括数据节点、管理节点和RBF神经网络训练单元，其中：
RBF神经网络训练单元以数据节点的索引数量、搜索反馈、搜索CPU消耗、搜索响应时长、节点负载压力等数据作为训练样本集，进行RBF神经网络训练，匹配最优的管理节点，得到失效的数据节点，向合适的管理节点发送申请加入簇群的请求；
判断管理节点在规定时长内是否有响应，若该管理节点在规定 时长内无响应，所述RBF神经网络训练单元对失效的数据节点重新进行RBF神经网络训练；若该管理节点在规定时长内有响应，管理节点对当前簇群的数据节点数量进行分析，判断簇群的数据节点数量是否达到上限，若达到上限，所述RBF神经网络训练单元对失效的数据节点重新进行RBF神经网络训练；若未达到上限，管理节点向申请加入的数据节点回复请求，与该数据节点建立连接，并通知整个搜索集群。
本发明实施例的原理阐述如下：
本发明实施例进行RBF神经网络训练的过程中，基于RBF神经网络进行搜索集群，选择若干个径向基函数，每个径向基函数对应一个训练数据。
由于距离是径向同性的，因此称为径向基函数。径向基函数核心算法为其插值函数，插值函数的公式如下：
Figure PCTCN2017080780-appb-000007
其中，X是输入，输入的次数是m，m为正整数；P是径向基函数的数量，P为正整数，且P>m；输入X是个m维的向量，样本容量为P；Xp是第P次输入，
Figure PCTCN2017080780-appb-000008
是第P个径向基函数，Xp是径向基函数
Figure PCTCN2017080780-appb-000009
的中心；ωP是第P个权值向量，(||X-Xp||)表示差向量的模(或者叫2范数)；y是输出的训练结果。
参见图2所示，本发明实施例在进行RBF神经网络训练的过程中，将数据节点的索引数量、搜索反馈、搜索CPU消耗、搜索响应时长、节点负载压力等数据，作为输入向量集，定时进行训练与计算，设置自定义误差范围、迭代次数及权值，训练得到最适合自己的簇群及管理节点。
当某个管理节点失效，系统自动取出其簇群下失效数据节点的训练结果数据，并找到适合该失效节点的管理节点，在失效数据节点与管理节点之间建立连接请求流程，自动申请加入簇群，簇群节点上限数为自定义。
本领域的技术人员可以对本发明实施例进行各种修改和变型，倘若这些修改和变型在本发明权利要求及其等同技术的范围之内，则这些修改和变型也在本发明的保护范围之内。
说明书中未详细描述的内容为本领域技术人员公知的现有技术。

Claims (10)

  1. 一种基于RBF神经网络的搜索集群优化方法，其特征在于，包括以下步骤：
    S1、以数据节点的索引数量、搜索反馈、搜索CPU消耗、搜索响应时长、节点负载压力数据作为训练样本集，进行RBF神经网络训练，匹配最优的管理节点，得到失效的数据节点，向合适的管理节点发送申请加入簇群的请求；
    S2、判断管理节点在规定时长内是否有响应，若该管理节点在规定时长内无响应，则返回步骤S1，对失效的数据节点重新进行RBF神经网络训练；若该管理节点在规定时长内有响应，则转到步骤S3；
    S3、管理节点对当前簇群的数据节点数量进行分析，判断簇群的数据节点数量是否达到上限，若达到上限，则返回步骤S1，对失效的数据节点重新进行RBF神经网络训练；若未达到上限，则转到步骤S4；
    S4、管理节点向申请加入的数据节点回复请求，与该数据节点建立连接，并通知整个搜索集群。
  2. 如权利要求1所述的基于RBF神经网络的搜索集群优化方法，其特征在于：步骤S1中进行RBF神经网络训练的过程中，基于RBF神经网络进行搜索集群，选择若干个径向基函数，每个径向基函数对应一个训练数据。
  3. 如权利要求2所述的基于RBF神经网络的搜索集群优化方法，其特征在于：所述径向基函数的插值函数的公式如下：
    Figure PCTCN2017080780-appb-100001
    其中，X是输入，输入的次数是m，m为正整数；P是径向基函数的数量，P为正整数，且P>m；输入X是个m维的向量，样本容量为P；Xp是第P次输入，
    Figure PCTCN2017080780-appb-100002
    是第P个径向基函数，Xp是径向基函数
    Figure PCTCN2017080780-appb-100003
    的中心；ωP是第P个权值向量，(||X-Xp||)表示差向量的模；y是输出的训练结果。
  4. 如权利要求3所述的基于RBF神经网络的搜索集群优化方法，其特征在于：步骤S1中进行RBF神经网络训练的过程中，将数据节点的索引数量、搜索反馈、搜索CPU消耗、搜索响应时长、节点负载压力数据，作为输入向量集，定时进行训练与计算，设置自定义误差范围、迭代次数及权值，训练得到最适合自己的簇群及管理节点。
  5. 如权利要求4所述的基于RBF神经网络的搜索集群优化方法，其特征在于：当某个管理节点失效，自动取出其簇群下失效数据节点的训练结果数据，并找到适合该失效节点的管理节点，在失效数据节点与管理节点之间建立连接请求流程，自动申请加入簇群，簇群节点上限数为自定义。
  6. 一种基于RBF神经网络的搜索集群优化系统，其特征在于，该系统包括数据节点、管理节点和RBF神经网络训练单元，其中：
    所述RBF神经网络训练单元以数据节点的索引数量、搜索反馈、搜索CPU消耗、搜索响应时长、节点负载压力数据作为训练样本集，进行RBF神经网络训练，匹配最优的管理节点，得到失效的数据节点，向合适的管理节点发送申请加入簇群的请求；
    判断管理节点在规定时长内是否有响应，若该管理节点在规定时长内无响应，所述RBF神经网络训练单元对失效的数据节点重新进行RBF神经网络训练；若该管理节点在规定时长内有响应，管理 节点对当前簇群的数据节点数量进行分析，判断簇群的数据节点数量是否达到上限，若达到上限，所述RBF神经网络训练单元对失效的数据节点重新进行RBF神经网络训练；若未达到上限，管理节点向申请加入的数据节点回复请求，与该数据节点建立连接，并通知整个搜索集群。
  7. 如权利要求6所述的基于RBF神经网络的搜索集群优化系统，其特征在于：所述RBF神经网络训练单元进行RBF神经网络训练的过程中，基于RBF神经网络进行搜索集群，选择若干个径向基函数，每个径向基函数对应一个训练数据。
  8. 如权利要求7所述的基于RBF神经网络的搜索集群优化系统，其特征在于：所述径向基函数的插值函数的公式如下：
    Figure PCTCN2017080780-appb-100004
    其中，X是输入，输入的次数是m，m为正整数；P是径向基函数的数量，P为正整数，且P>m；输入X是个m维的向量，样本容量为P；Xp是第P次输入，
    Figure PCTCN2017080780-appb-100005
    是第P个径向基函数，Xp是径向基函数
    Figure PCTCN2017080780-appb-100006
    的中心；ωP是第P个权值向量，(||X-Xp||)表示差向量的模；y是输出的训练结果。
  9. 如权利要求8所述的基于RBF神经网络的搜索集群优化系统，其特征在于：所述RBF神经网络训练单元进行RBF神经网络训练的过程中，将数据节点的索引数量、搜索反馈、搜索CPU消耗、搜索响应时长、节点负载压力数据，作为输入向量集，定时进行训练与计算，设置自定义误差范围、迭代次数及权值，训练得到最适合自己的簇群及管理节点。
  10. 如权利要求9所述的基于RBF神经网络的搜索集群优化系 统，其特征在于：当某个管理节点失效，系统自动取出其簇群下失效数据节点的训练结果数据，并找到适合该失效节点的管理节点，在失效数据节点与管理节点之间建立连接请求流程，自动申请加入簇群，簇群节点上限数为自定义。
PCT/CN2017/080780 2016-06-14 2017-04-17 基于rbf神经网络的搜索集群优化方法及系统 Ceased WO2017215339A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201610415351.8 2016-06-14
CN201610415351.8A CN105956190A (zh) 2016-06-14 2016-06-14 基于rbf神经网络的搜索集群优化方法及系统

Publications (1)

Publication Number Publication Date
WO2017215339A1 true WO2017215339A1 (zh) 2017-12-21

Family

ID=56906398

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2017/080780 Ceased WO2017215339A1 (zh) 2016-06-14 2017-04-17 基于rbf神经网络的搜索集群优化方法及系统

Country Status (2)

Country Link
CN (1) CN105956190A (zh)
WO (1) WO2017215339A1 (zh)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105956190A (zh) * 2016-06-14 2016-09-21 武汉斗鱼网络科技有限公司 基于rbf神经网络的搜索集群优化方法及系统

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102122306A (zh) * 2011-03-28 2011-07-13 中国人民解放军国防科学技术大学 一种数据处理方法及应用该方法的分布式文件系统
CN104123300A (zh) * 2013-04-26 2014-10-29 上海云人信息科技有限公司 数据分布式存储系统及方法
WO2015196911A1 (zh) * 2014-06-27 2015-12-30 华为技术有限公司 数据挖掘方法和节点
CN105550323A (zh) * 2015-12-15 2016-05-04 北京国电通网络技术有限公司 一种分布式数据库负载均衡预测方法和预测分析器
CN105956190A (zh) * 2016-06-14 2016-09-21 武汉斗鱼网络科技有限公司 基于rbf神经网络的搜索集群优化方法及系统

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102122306A (zh) * 2011-03-28 2011-07-13 中国人民解放军国防科学技术大学 一种数据处理方法及应用该方法的分布式文件系统
CN104123300A (zh) * 2013-04-26 2014-10-29 上海云人信息科技有限公司 数据分布式存储系统及方法
WO2015196911A1 (zh) * 2014-06-27 2015-12-30 华为技术有限公司 数据挖掘方法和节点
CN105550323A (zh) * 2015-12-15 2016-05-04 北京国电通网络技术有限公司 一种分布式数据库负载均衡预测方法和预测分析器
CN105956190A (zh) * 2016-06-14 2016-09-21 武汉斗鱼网络科技有限公司 基于rbf神经网络的搜索集群优化方法及系统

Also Published As

Publication number Publication date
CN105956190A (zh) 2016-09-21

Similar Documents

Publication Publication Date Title
US11720822B2 (en) Gradient-based auto-tuning for machine learning and deep learning models
WO2023134062A1 (zh) 基于人工智能的药物靶点作用关系确定方法及装置
CN107066621B (zh) 一种相似视频的检索方法、装置和存储介质
WO2022068196A1 (zh) 跨模态的数据处理方法、装置、存储介质以及电子装置
CN112396106B (zh) 内容识别方法、内容识别模型训练方法及存储介质
CN110941698B (zh) 一种基于bert下卷积神经网络的服务发现方法
CN106446162A (zh) 一种面向领域的本体知识库文本检索方法
CN111782817B (zh) 一种面向信息系统的知识图谱构建方法、装置及电子设备
CN113128526B (zh) 图像识别方法、装置、电子设备和计算机可读存储介质
CN114490923B (zh) 相似文本匹配模型的训练方法、装置、设备及存储介质
WO2021190668A1 (zh) 智能学习系统的知识图谱生成方法
CN118821921A (zh) 基于嵌入同步和对齐的多模态知识图谱补全方法及系统
CN106202288A (zh) 一种人机交互系统知识库的优化方法及系统
CN102981913B (zh) 支持大规模分布式增量计算的推理控制方法及推理控制系统
CN118626631A (zh) 基于大语言模型的专利文本推荐方法、系统、设备及介质
CN113392220A (zh) 一种知识图谱生成方法、装置、计算机设备及存储介质
CN120952003A (zh) 一种基于动态知识图谱与rag模型的文档分析方法
CN120822046A (zh) 基于知识图谱与图书文化解析的图书资源匹配方法
CN110456985A (zh) 面向多模态网络大数据的层次型存储方法及系统
CN110399564A (zh) 帐号分类方法和装置、存储介质及电子装置
CN110941706A (zh) 答案确定方法及系统
CN105956190A (zh) 基于rbf神经网络的搜索集群优化方法及系统
Miao et al. LightTR+: A Lightweight Incremental Framework for Federated Trajectory Recovery
Wang et al. Community detection in complex networks using improved artificial bee colony algorithm
Li Eye centre localisation with hybrid regression framework

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17812451

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 17812451

Country of ref document: EP

Kind code of ref document: A1