CN102622414B - Peer-to-peer structure based distributed high-dimensional indexing parallel query framework - Google Patents
Peer-to-peer structure based distributed high-dimensional indexing parallel query framework Download PDFInfo
- Publication number
- CN102622414B CN102622414B CN 201210038115 CN201210038115A CN102622414B CN 102622414 B CN102622414 B CN 102622414B CN 201210038115 CN201210038115 CN 201210038115 CN 201210038115 A CN201210038115 A CN 201210038115A CN 102622414 B CN102622414 B CN 102622414B
- Authority
- CN
- China
- Prior art keywords
- index
- website
- module
- working terminal
- index block
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000001360 synchronised effect Effects 0.000 claims abstract description 16
- 238000001514 detection method Methods 0.000 claims abstract description 8
- 238000000034 method Methods 0.000 claims description 63
- 230000008569 process Effects 0.000 claims description 57
- 230000004044 response Effects 0.000 claims description 16
- 238000011068 loading method Methods 0.000 claims description 12
- 238000011160 research Methods 0.000 claims description 12
- 238000012360 testing method Methods 0.000 claims description 11
- 230000005540 biological transmission Effects 0.000 claims description 7
- 238000009826 distribution Methods 0.000 claims description 7
- 230000011218 segmentation Effects 0.000 claims description 3
- 238000000638 solvent extraction Methods 0.000 claims description 3
- 230000008901 benefit Effects 0.000 abstract description 7
- 238000012544 monitoring process Methods 0.000 description 16
- 230000008859 change Effects 0.000 description 11
- 230000006870 function Effects 0.000 description 11
- 239000013598 vector Substances 0.000 description 10
- 238000005516 engineering process Methods 0.000 description 9
- 238000007726 management method Methods 0.000 description 9
- 230000002159 abnormal effect Effects 0.000 description 5
- 238000004891 communication Methods 0.000 description 5
- 238000012217 deletion Methods 0.000 description 4
- 230000037430 deletion Effects 0.000 description 4
- 238000001914 filtration Methods 0.000 description 3
- 230000010354 integration Effects 0.000 description 3
- 230000007246 mechanism Effects 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- 238000003860 storage Methods 0.000 description 3
- 108020005210 Integrons Proteins 0.000 description 2
- 238000007792 addition Methods 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 2
- 239000012141 concentrate Substances 0.000 description 2
- 230000008878 coupling Effects 0.000 description 2
- 238000010168 coupling process Methods 0.000 description 2
- 238000005859 coupling reaction Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000008034 disappearance Effects 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000012423 maintenance Methods 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000008520 organization Effects 0.000 description 2
- 230000009467 reduction Effects 0.000 description 2
- 230000003044 adaptive effect Effects 0.000 description 1
- HUTDUHSNJYTCAR-UHFFFAOYSA-N ancymidol Chemical compound C1=CC(OC)=CC=C1C(O)(C=1C=NC=NC=1)C1CC1 HUTDUHSNJYTCAR-UHFFFAOYSA-N 0.000 description 1
- 230000004888 barrier function Effects 0.000 description 1
- 230000033228 biological regulation Effects 0.000 description 1
- 238000012512 characterization method Methods 0.000 description 1
- 238000000151 deposition Methods 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 238000011049 filling Methods 0.000 description 1
- 239000012467 final product Substances 0.000 description 1
- 239000012634 fragment Substances 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000004806 packaging method and process Methods 0.000 description 1
- 238000011002 quantification Methods 0.000 description 1
- 238000011105 stabilization Methods 0.000 description 1
- 238000005303 weighing Methods 0.000 description 1
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a peer-to-peer structure based distributed high-dimensional indexing parallel query framework, which comprises an index creation module, a monitor module, a peer-to-peer site cluster and a soft load balance module. The index creation module is used for creating an index and storing index block files. The monitor module is used for detecting information of available memory of operation sites in the peer-to-peer site cluster and corresponding index block information of each operation site so as to coordinate the operation sites according to detection results. The soft load balance module is used for controlling load balance for the each operation site, and control is regularly synchronized through the monitor module so that the soft load balance module can timely regulate and update a lift of the operation sites currently available. The peer-to-peer structure based distributed high-dimensional indexing parallel query framework has the advantages of capability of reliably searching massive candidates in real time under support of dynamic increase and decrease, and is fast in query speed.
Description
Technical field
The present invention relates to the computer information retrieval technical field, particularly a kind of distributed high dimensional indexing parallel query framework based on peering structure.
Background technology
Developing rapidly of the technology such as CBIR, video frequency searching makes High-dimensional Index Technology obtain developing more and more widely and using.The application of High-dimensional Index Technology is mainly in order to accelerate retrieval rate, and then the inquiry that promotes the user is experienced.In the present existing overwhelming majority was used, in fact the inquiry of high dimensional indexing was exactly front k the arest neighbors (k Nearest Neighbors is called for short kNN) of retrieval and inquisition object on semantic level.
Generally speaking, at first the content-based retrieval technology needs the candidate target of magnanimity (image, video etc.) is converted into the high dimensional feature vector, with the form that the quantizes feature of conservation object as far as possible itself.Then, need to carry out index to the magnanimity high dimension vector that extracts.When the submit queries object, be converted into the high dimensional feature vector of equal dimension through quantification, and utilize high-dimensional index structure, quick-searching to go out its kNN, and result is returned.Under the major applications scene, the establishment of high dimensional indexing is to allow off-line, there is no too high requirement for real-time, but stability, reliability and integrality are had relatively high expectations.And for the inquiry of high dimensional indexing, the demand in practical application is very clear and definite and strict, must be as far as possible accurately and fast, reliably, and supports the dynamically retrieval of the magnanimity candidate target of increase and decrease.This is emphatically thinking and the problem that solves of the present invention just also.
Along with the popularization of types of applications, High-dimensional Index Technology has all caused concern widely in academia and industry member, and has emerged in large numbers algorithm and the scheme of a large amount of lifting high dimensional indexings establishments and search efficiency.The improved efficiency strategy of high dimensional indexing can be divided into three major types on current overall context.The first kind is that the cost that reduces the kNN degree of accuracy is raised the efficiency, that is only inquiry is similar to kNN.Equations of The Second Kind is to promote establishment and the inquiry velocity of index by changing index structure, still through the research of decades, not yet finds to support very efficiently at present the index structure that dimensions up to a hundred are even more high-dimensional.The 3rd class is that the framework from directory system improves, and the parallel index system improving creates and the speed of inquiry by introducing, yet aspect the research of in real time reliable parallel system, due to the existence of technology barriers, correlation technique is difficult to be widely used at present.In addition, traditional master-slave mode search index framework lacks self-organization, and performance bottleneck easily occurs.
Summary of the invention
The present invention is intended to one of solve the problems of the technologies described above at least.
For this reason, one object of the present invention is to propose a kind of distributed high dimensional indexing parallel query framework based on peering structure, this distributed high dimensional indexing parallel query framework is based on peering structure, have advantages of real-time, reliable, the dynamically magnanimity candidate target retrieval of increase and decrease of support, and inquiry velocity is fast.
to achieve these goals, embodiments of the invention have proposed a kind of distributed high dimensional indexing parallel query framework based on peering structure, comprise: the index creation module, monitor module, equity website cluster and the soft balance module of load, wherein, described index creation module is for the magnanimity candidate target being cut apart and being that each partitioning portion creates index to obtain a plurality of index block files, and described index block file is stored, wherein, described index block file comprises index block information, described monitor module for detection of the free memory information of the working terminal in described reciprocity website cluster and index block information corresponding to each working terminal according to testing result, the index block that each working terminal was loaded coordinated and to be sent reciprocity site list update instruction to each working terminal, working terminal in described reciprocity website cluster carries out load or unload according to the index block information of self to corresponding index block file, and inquire about and Query Result is integrated and exported in corresponding index block file according to the query requests that the user sends, the soft balance module of described load is used for obtaining the work at present site list of described monitor module according to described working terminal list, the work at present website is carried out load balancing control, and the soft balance module of described load is regularly undertaken synchronous so that the soft balance module of described load is adjusted and upgrades current available working terminal list by described monitor module.
Adopt the high index structure of efficient according to the distributed high dimensional indexing parallel query framework based on peering structure of the embodiment of the present invention, particularly, adopt Hybrid Spill Tree as basic index structure (high-dimensional index structure), and adopt the framework mode of reciprocity website, all done comparatively comprehensively ensureing at aspects such as real-time, reliability, stability, extensibility, self-organizations.Can efficiently inquire about, and have stronger extensibility, can move to easily in various content-based search systems, and have advantages of that inquiry velocity is fast.In addition, should be clear based on the distributed high dimensional indexing parallel query framed structure of peering structure, be easy to realize.
In addition, the distributed high dimensional indexing parallel query framework based on peering structure according to the above embodiment of the present invention can also have following additional technical characterictic:
In one embodiment of the invention, described index creation module further comprises: index creation submodule, described index creation submodule adopt Map Reduce framework to create index to obtain described a plurality of index block file to a plurality of Data Segmentation part parallels; Distributed memory system, described distributed memory system is used for preserving described a plurality of index block file.
In one embodiment of the invention, described monitor module also is used for after the index block file update being detected inoperative set of sites pocket transmission index load instructions so that described inoperative website cluster enters duty, the complete rear described monitor module of described index block file load will carry out cluster and switch, and use the working terminal cluster that has loaded up-to-date index, original working terminal cluster is placed in off working state.
In one embodiment of the invention, described monitor module judges whether there is the inefficacy website in current working terminal according to described testing result, and sends reciprocity site list update instruction to current available working terminal detecting when having the inefficacy website.
In one embodiment of the invention, register to described monitor module when having the inoperative website to add described working terminal cluster, so that described monitor module upgrades described working terminal list, and the working terminal list after described renewal is sent to the soft balance module of described load, the reciprocity site list after upgrading simultaneously sends to all described working terminals.
In one embodiment of the invention, when each working terminal in described reciprocity website cluster receives query requests, create host process and corresponding subprocess a plurality of according to the index block information creating of self, so that each subprocess inquires about from the index block of loading of correspondence, and Query Result is sent to host process and integrates transmission by host process.
In one embodiment of the invention, described host process also is used for the query requests that other working terminal of response is distributed.
In one embodiment of the invention, described subprocess is monitored host process, so as when described host process accident to withdraw from autoshutdown.
In one embodiment of the invention, the soft balance module of described load comprises the soft balanced website of a plurality of loads, and the soft balance module of described load is safeguarded described current available working terminal list.
In one embodiment of the invention, described index block file adopts Hybird Spill Tree data structure.
Additional aspect of the present invention and advantage part in the following description provide, and part will become obviously from the following description, or recognize by practice of the present invention.
Description of drawings
Above-mentioned and/or additional aspect of the present invention and advantage are from obviously and easily understanding becoming the description of embodiment in conjunction with following accompanying drawing, wherein:
Fig. 1 is the schematic diagram based on the distributed high dimensional indexing parallel query framework of peering structure of the embodiment of the present invention; And
Fig. 2 is the Organization Chart based on the distributed high dimensional indexing parallel query framework of peering structure of the embodiment of the present invention.
Embodiment
The below describes embodiments of the invention in detail, and the example of described embodiment is shown in the drawings, and wherein same or similar label represents same or similar element or the element with identical or similar functions from start to finish.Be exemplary below by the embodiment that is described with reference to the drawings, only be used for explaining the present invention, and can not be interpreted as limitation of the present invention.
in description of the invention, it will be appreciated that, term " " center ", " vertically ", " laterally ", " on ", D score, " front ", " afterwards ", " left side ", " right side ", " vertically ", " level ", " top ", " end ", " interior ", orientation or the position relationship of indications such as " outward " are based on orientation shown in the drawings or position relationship, only the present invention for convenience of description and simplified characterization, rather than device or the element of indication or hint indication must have specific orientation, with specific orientation structure and operation, therefore can not be interpreted as limitation of the present invention.In addition, term " first ", " second " only are used for describing purpose, and can not be interpreted as indication or hint relative importance.
In description of the invention, need to prove, unless clear and definite regulation and restriction are separately arranged, term " installation ", " being connected ", " connection " should be done broad understanding, for example, can be to be fixedly connected with, and can be also to removably connect, or connect integratedly; Can be mechanical connection, can be also to be electrically connected to; Can be directly to be connected, also can indirectly be connected by intermediary, can be the connection of two element internals.For the ordinary skill in the art, can concrete condition understand above-mentioned term concrete meaning in the present invention.
The distributed high dimensional indexing parallel query framework based on peering structure of the embodiment of the present invention is described below in conjunction with accompanying drawing.
Referring to Fig. 1 and Fig. 2, comprise index creation module Builder, monitor module Monitor, reciprocity website cluster and the soft balance module Balancer of load according to the distributed high dimensional indexing parallel query framework 100 based on peering structure of the embodiment of the present invention.Wherein:
Index creation module Builder is used for the magnanimity candidate target is cut apart and be each partitioning portion establishment index obtaining a plurality of index block files, and described index block file is stored, and wherein, the index block file comprises index block information.
Monitor module Monitor for detection of the free memory information of the working terminal in described reciprocity website cluster and index block information corresponding to each working terminal according to testing result, the index block that each working terminal was loaded coordinated and to be sent reciprocity site list update instruction to each working terminal.Particularly, monitor module Monitor monitors, coordinates and manages the work between each working terminal.On the one hand, for example monitor module Monitor is by free memory information and index block information on each working terminal of mode timing acquisition that is similar to the heartbeat packet detection, and according to the result that obtains, analyze and whether have repetition index or disappearance index, and carry out intelligent coordinated according to free memory information, require the index block of some working terminal unloading index block or load miss, with lowest redundancy and the integrality of guaranteeing index block.On the other hand, monitor module Monitor will provide the management function of reciprocity website cluster, heartbeat packet detects can find the website that lost efficacy, and notify other websites to carry out the adjustment of reciprocity site list, and after new website adds, also can register from trend monitor module Monitor, and notify other working terminals to adjust reciprocity site list by Monitor, can be close to guarantee to the full extent each website the site information of knowing in real time in same index cluster.In addition, monitor module Monitor be responsible for load balancing module Balancer between synchronous working.Effective site list in the current index cluster that monitor module Monitor detects oneself is synchronized to the soft balanced website of each load in load balancing module Balancer, can go out effective available work website as all query selections take proof load balancing equipment Balancer.
Working terminal in equity website cluster carries out load or unload according to the index block information of self to corresponding index block file, and inquires about and Query Result is integrated and exported in corresponding index block file according to the query requests that the user sends.
The soft balance module Balancer of load controls according to described working terminal list, the work at present website is carried out load balancing for the work at present site list that obtains monitor module Monitor, and the soft balance module Balancer of load will regularly be undertaken synchronous so that the soft balance module Balancer of load adjusts and upgrades current available working terminal list by monitor module Monitor.
Adopt the high index structure of efficient according to the distributed high dimensional indexing parallel query framework 100 based on peering structure of the embodiment of the present invention, particularly, adopt Hybrid Spill Tree as basic index structure (high-dimensional index structure), and adopt the framework mode of reciprocity website, all done comparatively comprehensively ensureing at aspects such as real-time, reliability, stability, extensibility, self-organizations.Can efficiently inquire about, and have stronger extensibility, can move to easily in various content-based search systems, and have advantages of that inquiry velocity is fast.In addition, should be clear based on the distributed high dimensional indexing parallel query framed structure of peering structure, be easy to realize.
In one embodiment of the invention, index creation module Builder further comprises index creation submodule and distributed memory system.Wherein:
The index creation submodule adopts Map Reduce framework to create index to obtain described a plurality of index block file to a plurality of Data Segmentation part parallels.Distributed memory system is used for preserving described a plurality of index block file.
Particularly, consider that index creation allows off-line to carry out, not high for requirement of real-time, and require comparatively strict for reliability and stability, so directly adopt ripe Map Reduce framework, be used for parallel establishment index, and by means of distributed memory system, with the storage organization transparence of bottom, be used for depositing the index block file.
It will be appreciated that, in index creation module Builder, the magnanimity candidate target is divided into a plurality of parts, and be that every part creates corresponding index block, can lay the foundation for follow-up distributed parallel inquiry.Adopt Map Reduce framework, regularly walking abreast efficiently creates index, and after new index creation success, covers old index file, to adapt to the change of candidate target.
In one embodiment of the invention, described monitor module Monitor also is used for after the index block file update being detected inoperative set of sites pocket transmission index load instructions so that described inoperative website cluster enters duty, and will carry out the cluster switching at the complete rear described monitor module of index block file load, and use has loaded the working terminal cluster of up-to-date index, original working terminal cluster is placed in off working state, sends up-to-date working terminal list to the soft balance module of load simultaneously.That is to say, monitor the change conditions of index file, and after the indexed file vary stable, require the inoperative website to carry out index and load, load to complete and carry out the cluster switching afterwards, make it to become working terminal, the working terminal before replacing it.
In one embodiment of the invention, described monitor module judges whether there is the inefficacy website in current working terminal according to described testing result, and sends reciprocity site list update instruction to current available working terminal detecting when having the inefficacy website.The heartbeat packet that is monitor module Monitor by above-described embodiment detects can find the website that lost efficacy, and notifies other websites to carry out the adjustment of reciprocity site list.
Further, to register to monitor module Monitor when having the inoperative website to add the working terminal cluster, so that monitor module Monitor to working terminal list upgrade, and the working terminal list after described renewal is sent to the soft balance module of described load, send simultaneously other working terminals of command request and carry out the renewal of reciprocity site list.
As a concrete example, when each working terminal in reciprocity website cluster receives query requests, create host process and corresponding subprocess a plurality of according to the index block information creating of self, so that each subprocess inquires about from the index block that has loaded of correspondence, and Query Result is sent to host process and integrates transmission by host process.Further, host process also is used for the query requests that other working terminal of response is distributed, that is will further distribute query requests after certain working terminal receives user's query requests, requires other websites to assist completing of inquiry.In this example, subprocess is monitored host process, so as when described host process accident to withdraw from autoshutdown.
Particularly, the host process of working terminal mainly is responsible for the work of three parts.
1, the heartbeat detection of response monitor module Monitor, the memory information that provides the work at present website to use, and the index block information of being responsible for etc.Therefore,, should initiatively register to monitor module Monitor when starting service when working terminal.And host process need to be safeguarded index block information, when needs load the assigned indexes piece, creates corresponding subprocess index block is managed, and when needs unloading assigned indexes piece, be that the subprocess of being responsible for this index block is closed.In addition, host process needs the antithetical phrase state of a process to monitor, when finding that subprocess lost efficacy, and can be in time to monitor module Monitor feedback index block deletion condition.
2, respond the query requests of other working terminal distributions, and according to query requests, subprocess is submitted in request and processed, the result of then in the time threshold of appointment, subprocess being returned is integrated, and finally returns to the working terminal of distribution query requests.
3, accept user's query requests, the parallel working terminal that is distributed to other carries out secondary integration to returning results of each working terminal (comprising) itself at last, and returns to the user in the time threshold of appointment.
The work that the subprocess of working terminal is responsible for is as follows:
1, load the index block file, and the index block file is carried out internal memory launch, shone upon the one-to-one relationship between proper vector and candidate target.
2, the query requests of response host process, the most similar Query Result of the predetermined quantity of inquiry appointed object in index structure, and result is returned to host process.
3, the activity of monitoring host process,, can close, in order to avoid take excess resource when the host process accident withdraws from voluntarily with assurance.
It will be appreciated that, what the index block management in the embodiment of the present invention was adopted is the subprocess pattern, but not thread mode, mainly in order to guarantee in the situation that low memory to load new index block, adopt the mode of subprocess can avoid the situation that causes host process to be collapsed fully because loading new index block failure, finally be unlikely to cause all the index block of loaded all lost efficacy.So also in order to guarantee to the full extent Systems balanth and reliability.
In one embodiment of the invention, the soft balance module of described load comprises the soft balanced website of a plurality of loads.The soft balanced website of each load act as suitable scheduling query requests, make the load of each working terminal reach generally balance, to avoid occurring the overweight and phenomenon of other website underloads of certain site load.The soft balance module Balancer of load obtains effectively working terminal list in current index cluster by the synchronous effect of monitor module Monitor, and receives the request of query interface, selects suitable working terminal to return for query interface and carries out query calls.
Need to prove, synchronous smooth and easy in order to guarantee, and working terminal is similar, can initiatively register to the Monitor website when the soft balanced website of load starts service.Generally speaking, if the hardware facility situation of each working terminal is close, the mode that the soft balanced website of load can be taked to choose is at random carried out the selection of working terminal, and returns to query interface and carry out further query calls.
In embodiments of the invention, can also have query interface module Searcher, query interface module Searcher mainly is responsible for the function of two aspects.
1, query object is quantized, be converted into the vector with equal dimension, and submit to this vector to inquire about, then Query Result is converted into the required form of external interface.
2, the scheduling relation with bottom encapsulates, upwards shielding, and with the coupling of reduction framework, and the extensibility of Hoisting System.In fact, the query manipulation of this part can be divided into two steps: obtain effective working terminal from the soft balance module Balancer of load, inquiry is submitted to working terminal and obtained corresponding result.
In an example of the present invention, in order to guarantee requirement of real-time, query portion is not suitable for adopting Map Reduce framework, but is responsible in conjunction with more efficient modes of intelligence transmission such as MPI communication and the coordination that website is asked.
Embodiments of the invention have following advantage:
1, highly-parallel, inquiry is efficient, and real-time is better, and degree of accuracy is higher.
What adopt is distributed search index framework, and has adopted high-dimensional index structure Hybrid SpillTree comparatively efficiently, simultaneously in conjunction with efficient communication mechanisms such as MPI, all guarantees as far as possible in all its bearings the quick response of inquiry.In addition, Hybrid Spill Tree under the stand-alone environment, the distributed index structure of the embodiment of the present invention can effectively improve degree of accuracy, main cause is when carrying out the approximate NN Query of predetermined quantity at every turn, adopted redundancy strategy, and finally result has been carried out two stage integration and filtration, thereby can guarantee that the result of finally returning can cover the accurate neighbour of the predetermined quantity of the overwhelming majority.Wherein to filter be that result that working terminal returns local subprocess is when integrating in the integration of phase one, filtering a part of redundant results, subordinate phase is when sending result that the working terminal of global query request returns other working terminals and integrating, further the filtering redundant results.Need to prove in addition, Hybrid Spill Tree itself possesses the adjustable characteristics of parameter, can select in actual applications suitable parameter, in order to obtain good compromise between efficient and degree of accuracy.
2, system reliability and stability are higher.
Can find out from the module analysis of framework, the present invention has introduced the load condition that monitor module Monitor is used for monitoring and manages index block, and working terminal is managed accordingly, carry out intelligent coordinated according to the internal memory situation of website, with lowest redundancy and the integrality that guarantees as much as possible global index, thus the reliability of assurance inquiry.In addition, the multiple abnormal conditions that may occur have been considered, comprise that the machine of delaying, subprocess withdraw from extremely, host process withdraws from etc. extremely, and for above problem, introduced comparatively complicated monitoring mechanism and the coordination system, thereby guaranteed in multiple abnormal generation, in even parallel situation about occuring, system is comparatively robust ground running still, and guarantees reception and the feedback of external inquiry.What need to mention is a bit that on the surface, the monitor module Monitor that the present invention introduces and the home site comparing class in traditional client/server scheme seemingly may become the bottleneck of whole framework.But in fact, can know by analyzing, the monitor module Monitor in the present invention, its load is obviously lower than the home site in host-guest architecture.And, by adopting the technology such as traditional Hot Spare or cluster, can guarantee to a greater extent that the same time has Monitor website and Balancer website to run well, thereby provide powerful guarantee for Systems balanth.In addition, the architectural schemes that is based on peering structure that the embodiment of the present invention adopts, less for the dependence of bottleneck point, even if in the situation that Monitor complete failure, as long as have a Balancer website and several working terminals to work, system still can respond inquiry.
3, extensibility, scalability are better.
The high dimensional indexing parallel query framework that the embodiment of the present invention proposes is not intended for certain specific application, but directly processing feature is vectorial, this makes this framework have stronger extensibility, both can be applied to CBIR, can also be applied in the fields such as video frequency searching, audio retrieval.In addition, in the present invention, for working terminal and the soft balanced website of load, after they start service, all can initiatively register to Monitor, nullify application and also all can submit to Monitor when withdrawing from, and carry out follow-up notice or synchronous by Monitor, this mechanism makes dynamic increase or deletes website convenient, need not to intervene other websites or restarts system, only need configure by the website of additions and deletions to get final product.These reflect that all framework proposed by the invention is possessing stronger adaptive faculty aspect the dynamic retractility of system.In addition, in the high dimensional indexing parallel query framework based on peering structure that the embodiment of the present invention proposes, arbitrary working terminal can also can be used as the assistance website of global query simultaneously as the responsible website of global query, operates more flexible.
4, self-organization, intelligent management ability are stronger
Distributed high dimensional indexing parallel query framework based on peering structure proposed by the invention is in the management of website and coordinate to have done two-way guarantee.On the one hand, the startup of working terminal or the soft balanced website of load or withdraw from all and can register or nullify to the Monitor website, and notified and the follow-up work such as synchronous by the Monitor website; On the other hand, the Monitor website itself is being safeguarded all soft balanced site lists of load, safeguarding simultaneously the whole working terminal information in the reciprocity website cluster (index cluster) of all reciprocity website cluster informations and current use, and these websites are coordinated and managed (comprising going heavily, filling a vacancy etc. of index block).In addition, the inquiry framework that the embodiment of the present invention proposes is supported the combination of isomery website, and can be according to the free memory situation of each website, carry out intelligently the distribution of index block and other scheduling, co-ordination, make the resource of system can obtain abundant, appropriate utilization.And the introducing of the soft balanced website of load is also adjusted the load of each working terminal on the aspect of soft equilibrium.Above-mentionedly reflected that all framework proposed by the invention possesses stronger self-organization and intelligent management ability.
5, support the dynamically mass data of increase and decrease
The present invention is in conjunction with comparatively ripe Map Reduce framework and corresponding distributed memory system, and establishment and the inquiry of high dimensional indexing connected.After introducing Map Reduce framework carries out parallelization to index creation, solved to a certain extent the large limitation of high-dimensional index structure additions and deletions cost, but carried out the renewal of high dimensional indexing by the mode of Fast Reconstruction index.In addition, introduce distributed memory system, can be with the storage system transparence of bottom, the subprocess that is more convenient in working terminal carries out reading of index block, and need not to carry out complicated configuration.The combination of above two aspects is laid a good foundation for the high-dimensional index structure that mass data is provided creates, and provides safeguard for feasibility.The present invention is connected with inquiry the establishment of high dimensional indexing by distributed memory system, the high dimensional indexing creation module only is responsible for regularly rebuilding index, change conditions with the reflection data, the renewal that the Monitor website regularly carries out index file detects, and after stable the renewal being detected, carry out the loading of index and the switching of index cluster, make system can in time respond the dynamic change situation of mass data.
[embodiment]
In actual applications, its validity and feasibility have been verified in the actual test of 120 dimensional feature vectors by 1,000,000 images.As shown in the table, be the distributed high dimensional indexing parallel query framework experimental result based on peering structure.Listed the response speed situation of the present invention's several times inquiry in the reality test.It should be noted that, in this experimentation, the time threshold of all inquiries is set to 1 second, it is 5 parts and the index block file that has created equal number that all images is split, equity website cluster comprises the ordinary PC of 5 isomeries, and (wherein four PC's is configured to the 2G internal memory, Pentium E5300 double-core CPU, other one is configured to the 4G internal memory, Pentium E5200 double-core CPU).In addition, in experimentation, the parameter of Hybrid Spill Tree is not carried out tuning, and adopt empirical value to test, to reflect truly more comprehensively the responding ability of system.
| |
1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | Average |
| Inquire about (millisecond) consuming time | 139 | 121 | 77 | 64 | 88 | 57 | 90 | 66 | 65 | 85 | 85.2 |
Table 1
As can be seen from Table 1, there is certain wave phenomenon in query time, but the global response time is all still very short, and average response time is 85.2 milliseconds, substantially can satisfy the Search Requirement in practical application.And what can look forward to is, Hybrid Spill Tree is being carried out the parameter tuning, introduces more powerful server, sets under the less conditions such as inquiry time limit, and query time will have further reduction, thereby make system's highly effective more.
Based on the more specific implementation of the distributed high dimensional indexing parallel query framework of peering structure, as follows:
The realization of index creation Model Implement (high dimensional indexing creation module Builder):
The work that the high dimensional indexing creation module mainly is responsible for is regularly the candidate target of magnanimity to be carried out the reconstruction of index, to guarantee in time to reflect the change conditions of candidate target.This module is not due to high to requirement of real-time, and it is comparatively strict to stability and reliability requirement, in this embodiment, realize by means of relatively mature and stable Map Reduce framework, and in conjunction with distributed memory system, the process of index creation is carried out highly-parallel, to guarantee to rebuild the efficient and reliable of index.
Therefore, the Builder Model Implement concentrates on comparatively ripe building of Open Framework and building of distributed memory system on the one hand, to concentrate on timed task is set on the other hand, magnanimity candidate target to appointment regularly carries out appropriate deblocking, and carrying out the parallelization establishment of index by Map Reduce task, each index block file is stored in transparent distributed memory system the most at last.
In addition, the Builder module also needs to be responsible for the maintenance of index block file, particularly, exactly after new index all creates successfully, remove old index block file, and new index block file is all moved under the path of appointment in distributed memory system, in order to arrived by the Monitor module monitors.What need explanation is a bit, here the mistiming that covers the index block file can not cause occurring monitoring the situation of makeing mistakes, because the monitoring function in the Monitor module is not loading and the switching that begins immediately new index after finding for the first time to change, but can continue new index file is carried out the tracking of several times, and only in the situation that judge the basicly stable just switching of execution index of index file.And even if the situation of abnormal too early loading occurred, along with the change of follow-up index file, system will reload, and finally guarantees the accuracy that index switches.
The realization of monitor module (Monitor):
The work that the Monitor module is responsible for mainly comprises three parts: the renewal monitoring of index file, the monitoring of working terminal (Peer website), coordination and management, and the soft balanced website of synchronized loading (Balancer website).And communicating by letter between Monitor module and other modules, website, and the communication between other modules, between website, in this embodiment, all mainly depend on efficient communication interface RMI (the Remote Method Invocation of Java language, the Java of MPI realizes, is widely used in the exploitation of parallel environment).According to the implementation feature of RMI, the Monitor module in the process of carrying out above-mentioned three some works, is the side of being called of RMI service, is again simultaneously the called side (Peer website, subprocess etc. are also similar) of RMI service.
As the side of being called of RMI service, the service interface that Monitor need to provide comprise following these: registerClusterPeer, registerBalancer, unregisterClusterPeer, unregisterBalancer.Wherein registerClusterPeer and registerBalancer service interface are mainly used in the registration of Peer website and Balancer website, reciprocity website cluster under wherein the registration of Peer website need to indicate, unregisterClusterPeer and unregisterBalancer service interface are mainly used in the de-registration request of Peer website and Balancer website.By above four service interfaces, Monitor website self maintained Peer site information and all Balancer site information of all reciprocity website clusters.
As the called side of RMI service, Monitor mainly depends on the service interface that calls other modules or website when carrying out the work that it is responsible for, thereby realizes corresponding function.
The renewal monitoring aspect of indexed file, Monitor website need to start regularly thread and be used at set intervals index file being detected, and confirm the change conditions of index file by comparison.And after finding change first, the Monitor website will further be followed the tracks of, until confirm that finally the index file change just starts the index task switching after stable.The switching of index is actually the idle reciprocity website cluster of Monitor website notice and carries out the loading of new index block, and judges whether to load successfully according to predetermined strategy.In the situation that load successfully, directly current index cluster is switched on new cluster, and synchronous Balancer website.
Detection, coordination and management aspect at working terminal, the Monitor website need to be safeguarded regularly thread equally, for the free memory of monitoring current all Peer websites of index cluster, index block situation etc., and will repeatedly not have the website of response to regard as the inefficacy website according to the strategy of appointment, report to managerial personnel.In addition, when four service interfaces that provide when the Monitor website were called, Monitor website itself also need to be safeguarded corresponding Peer site list and Balancer site list in internal memory, and notice is to other websites.Particularly, when certain Peer site registration or cancellation, after Monitor website response request, can notify other websites in same reciprocity website cluster to upgrade its reciprocity site list information.In addition, free memory information and index block information that Monitor can feed back according to each Peer website, therefrom find the index block of repetition or the index block of disappearance, and according to the free memory information of Peer website, select intelligently suitable website to carry out unloading or the loading of index block, with lowest redundancy and the integrality that guarantees whole index.
Synchronized loading soft balanced aspect, be actually the Monitor website and start regularly thread, and the service that provides by directly calling the Balancer website, the available site list information of its current index cluster of safeguarding is synchronized to all Balancer websites, make it in time to reflect the situation of index change, and be consistent with the Monitor website, thereby can provide suitable Peer website as the inquiry home site for the query interface module.
What need explanation is a bit that the Monitor Model Implement need to have corresponding exterior arrangement file, so that the Monitor module is carried out initialization.Wherein of paramount importance deploy content comprises: index monitoring time interval, Peer website monitoring time interval, index file is deposited the path in distributed memory system, available reciprocity website cluster and inner Peer site information thereof (comprising IP address, RMI serve port etc.), whole Balancer site information, and with RMI or relevant some configuration items of coordination strategy etc.
The realization of host process in the equity website:
Equity website host process is the main provider of inquiry service, this process mainly is responsible for the function of three aspects:: the monitoring request of response Monitor website is also safeguarded reciprocity site list information and local index block message, respond the global search request of other websites and the Query Result of integron process, distribute global search requests and integrate the Query Result that each reciprocity website returns to other reciprocity websites.
In the monitoring request of response Monitor website with safeguard aspect the local index piece, the Peer website need to be registered to Monitor when starting, to inform Monitor, and then inform other reciprocity websites under same reciprocity website cluster, this is the Peer website as one of place of RMI service call side.After startup, the Peer website need to provide following service: Ping, addFederal, removeFederal, loadIndex, unloadIndex at least for Monitor.Wherein the Ping service interface is used for returning the free memory information of Peer website itself and the index block information of managing; Certain reciprocity site information in the reciprocity site list that addFederal and removeFederal service interface be used for to add and deletion Peer website internal memory is safeguarded is to guarantee the accurate distribution of global search request; LoadIndex and unloadIndex service interface are used for loading and unloading the index block of appointment, in actual the realization, loadIndex starts a subprocess of being in charge of this index block by host process, unloadIndex directly closes corresponding subprocess by force, with the corresponding index of unloading from internal memory.Meanwhile, host process can remain the monitoring to subprocess, when finding that certain subprocess meets the expiration policy of appointment (such as continuous several times response timeout etc.), assert that directly it is the inefficacy subprocess, and feeds back to the Monitor website.Accordingly, host process need to start PingForChildProcess service for subprocess, with the monitoring request of response subprocess, thereby keeps the running status of subprocess.
Aspect the Query Result of the global search request that responds other websites and integron process, each Peer website need to be opened search service for other reciprocity websites, to receive the global search request of other reciprocity websites.After the global search request that receives other websites, the Peer website will call the service interface of subprocess, and retrieval request is further submitted to all subprocesss of oneself safeguarding.After getting the Query Result that each subprocess returns, the Peer website will be responsible for result is integrated from high to low according to degree of accuracy, and remove redundancy section, then return to the reciprocity website of distribution global query request.
To other reciprocity websites distribution global search requests and integrate aspect the Query Result that each reciprocity website returns, the Peer website need to provide the globalSearch service to call for the query interface module.When receiving the query requests of query interface module, the Peer website will be by a plurality of threads, further distribute retrieval request to other Peer websites in same reciprocity website cluster, oneself also start local retrieval simultaneously, submit to retrieval request to all subprocesss of local maintenance.The Peer website had both needed to integrate the result for retrieval of local subprocess after sending the global search request, need again to integrate the result for retrieval that other reciprocity websites return.The process need of integrating is according to degree of accuracy order from high to low, and needs removal redundancy, and result returns to the query interface module the most at last.
The Peer website needs the exterior arrangement file to carry out initialized setting equally, wherein main configuration item comprises: the RMI serve port scope that subprocess can be used, the query time restriction, local RMI service relevant configuration item (IP address and service port number etc.), the parameter that the promoter process the is relevant monitoring time of operational factor, the subprocess (interval etc.), the Peer site information of same index cluster (IP address, RMI service port number etc.), and Monitor site information (IP address, RMI service port number etc.) etc.
The realization of equity website subprocess (Child Process):
Equity website subprocess is related is actually the subprocess that is started by reciprocity website host process, management and the inquiry of an index block of each this type of subprocess individual responsibility.Therefore, single Peer website may have a plurality of subprocesss, simultaneously a plurality of index blocks is managed and inquires about.
Equity website subprocess is created by host process and when starting, and is specified port, the index block file path in distributed memory system of RMI service by host process, and the service port number of host process etc.After subprocess starts, will obtain by the interface of distributed memory system corresponding index block file, and it will be launched in internal memory, shone upon the relation between query object and proper vector, for follow-up inquiry is prepared.
As the called side of RMI service, subprocess is mainly by calling the RMI service of host process, host process is carried out heartbeat detection, can close voluntarily when host process withdraws from extremely with assurance, avoiding taking too much resource.
And as the side of being called of RMI service, subprocess mainly provides the search service interface, in fact be exactly to accept query vector and the association requests that host process is given, inquire about in Hybrid Spill Tree index structure in internal memory, and final result is returned to host process, carry out result by host process and integrate and send.
The configuration section of subprocess is actually by host process and passes into by the process parameter, need not the exterior arrangement file.
The realization of the soft balance module of load (Balancer):
The work that the soft office of load weighing apparatus module is born in system mainly comprises two parts, is respectively the synchronous requirement of accepting the Monitor website, and for the query interface module provides suitable available Peer website, as the home site of inquiry.
As the called side of RMI service, the Balancer website is mainly after starting service, registers by the service of calling the Monitor website, and is synchronous to inform that the Monitor website begins to accept the Peer site list.
And as the side of being called of RMI service, the service interface that the Balancer website provides mainly comprises: SynchronizeCluster and getAProperPeer.Wherein the SynchronizeCluster service interface is mainly called by the Monitor website, thereby realizes the synchronizing function of effective Peer site list in current index cluster; The getAProperPeer service interface is called by query interface, returns to Peer website available in current index cluster.
The configuration information of Balancer website mainly comprises: Monitor site information (comprising IP address and RMI serve port etc.), and Balancer website itself starts some relevant parameters of RMI service etc.
The realization of query interface module (Searcher):
The work that query interface is responsible for is to accept the external inquiry request, and query object is quantified as proper vector, and the Peer website from the Balancer station for acquiring can be used as the entrance of search, and is converted into the required form of external interface with final Search Results.The query interface module is the coupling part between the scheduling of upper layer application and bottom, by communication and the call relation of packaging bottom layer, for the exploitation of upper layer application provides interface easily.Therefore, in fact this part of module can be used as chained library or third party's bag, offers the upper strata developer and further develops.
Particularly, the query interface module need to realize tetrameric function: accept and quantize query object, calling the Peer website that the Balancer station for acquiring can be used, calling the Peer website and carry out search, and the processing of Search Results.
The configuration of query interface relates generally to the information (comprising the IP address of Balancer website and serve port etc.) of Balancer website.
The connection of system and overall operation:
Distributed high dimensional indexing parallel query framework based on peering structure proposed by the invention has stronger self-organization and intelligent management ability, does not have strict sequence requirement for the boot sequence of system's modules.Suggestion first operates the index creation module, then starts Monitor module, Balancer module and reciprocity website cluster, starts at last the query interface module and begins to accept the external inquiry request but generally speaking.In addition, aspect system's connection, the planning of cluster is first carried out in suggestion, and the configuration file to modules carries out correct configuration, with the assurance system can after startup with under velocity-stabilization faster, and complete the work such as loading of index, with the response external query requests.
The whole modules of system need to comparatively comprehensively be tested the overall operation situation of system after starting, particularly to the test of the various abnormal conditions that may exist, to weigh the reliability and stability of system.Through actual test analysis, framework provided by the invention is done fairly perfectly aspect reliability and stability, can resist preferably that the multiple machine of delaying, host process withdraw from extremely, subprocess withdraws from extremely etc. abnormal, and overall operation is all right.
Describe and to be understood in process flow diagram or in this any process of otherwise describing or method, expression comprises module, fragment or the part of code of the executable instruction of the step that one or more is used to realize specific logical function or process, and the scope of the preferred embodiment of the present invention comprises other realization, and this should be understood by the embodiments of the invention person of ordinary skill in the field.
Should be appreciated that each several part of the present invention can realize with hardware, software, firmware or their combination.In the above-described embodiment, a plurality of steps or method can realize with being stored in storer and by software or firmware that suitable instruction execution system is carried out.
Those skilled in the art are appreciated that and realize that all or part of step that above-described embodiment method is carried is to come the relevant hardware of instruction to complete by program, described program can be stored in a kind of computer-readable recording medium, this program comprises step of embodiment of the method one or a combination set of when carrying out.
In addition, each functional unit in each embodiment of the present invention can be integrated in a processing module, can be also that the independent physics of unit exists, and also can be integrated in a module two or more unit.Above-mentioned integrated module both can adopt the form of hardware to realize, also can adopt the form of software function module to realize.If described integrated module realizes with the form of software function module and during as independently production marketing or use, also can be stored in a computer read/write memory medium.
The above-mentioned storage medium of mentioning can be ROM (read-only memory), disk or CD etc.
In the description of this instructions, the description of reference term " embodiment ", " some embodiment ", " example ", " concrete example " or " some examples " etc. means to be contained at least one embodiment of the present invention or example in conjunction with specific features, structure, material or the characteristics of this embodiment or example description.In this manual, the schematic statement of above-mentioned term not necessarily referred to identical embodiment or example.And the specific features of description, structure, material or characteristics can be with suitable mode combinations in any one or more embodiment or example.
Although illustrated and described embodiments of the invention, for the ordinary skill in the art, be appreciated that without departing from the principles and spirit of the present invention and can carry out multiple variation, modification, replacement and modification to these embodiment, scope of the present invention is by claims and be equal to and limit.
Claims (10)
1. the distributed high dimensional indexing Research on Parallel Query System based on peering structure, is characterized in that, comprising: index creation module, monitor module, reciprocity website cluster and the soft balance module of load, wherein,
Described index creation module is used for candidate target is cut apart and be each partitioning portion establishment index obtaining a plurality of index block files, and described index block file is stored, and wherein, described index block file comprises index block information,
Described monitor module for detection of the free memory information of the working terminal in described reciprocity website cluster and index block information corresponding to each working terminal according to testing result, the index block information that each working terminal was loaded coordinated and to be sent reciprocity site list update instruction to each working terminal
Working terminal in described reciprocity website cluster carries out load or unload according to the index block information of self to corresponding index block file, and inquire about and Query Result is integrated and exported in corresponding index block file according to the query requests that the user sends
The soft balance module of described load is controlled according to described working terminal list, the work at present website is carried out load balancing for the work at present site list that obtains described monitor module, and the soft balance module of described load will regularly be undertaken synchronous so that the soft balance module of described load is adjusted and upgrades current available working terminal list by described monitor module.
2. the distributed high dimensional indexing Research on Parallel Query System based on peering structure according to claim 1, is characterized in that, described index creation module further comprises:
Index creation submodule, described index creation submodule adopt Map Reduce framework to create index to obtain described a plurality of index block file to a plurality of Data Segmentation part parallels;
Distributed memory system, described distributed memory system is used for preserving described a plurality of index block file.
3. the distributed high dimensional indexing Research on Parallel Query System based on peering structure according to claim 1, is characterized in that,
Described monitor module also is used for after the index block file update being detected inoperative set of sites pocket transmission index load instructions so that described inoperative website cluster enters duty, the complete rear described monitor module of described index block file load will carry out cluster and switch, and use the working terminal cluster that has loaded up-to-date index, original working terminal cluster is placed in off working state.
4. the distributed high dimensional indexing Research on Parallel Query System based on peering structure according to claim 1, it is characterized in that, described monitor module judges whether there is the inefficacy website in current working terminal according to described testing result, and sends reciprocity site list update instruction to current available working terminal detecting when having the inefficacy website.
5. the distributed high dimensional indexing Research on Parallel Query System based on peering structure according to claim 1, it is characterized in that, to register to described monitor module when having the inoperative website to add the working terminal cluster, so that described monitor module upgrades described working terminal list, and the working terminal list after described renewal is sent to the soft balance module of described load, the reciprocity site list after upgrading simultaneously sends to all described working terminals.
6. the distributed high dimensional indexing Research on Parallel Query System based on peering structure according to claim 1, it is characterized in that, when each working terminal in described reciprocity website cluster receives query requests, host process and corresponding subprocess a plurality of according to the index block information creating of self will be created, so that each subprocess inquires about from the index block of the loading information of correspondence, and Query Result is sent to host process and integrates transmission by host process.
7. the distributed high dimensional indexing Research on Parallel Query System based on peering structure according to claim 6, is characterized in that, described host process also is used for the query requests of other working terminal distribution of response.
8. the distributed high dimensional indexing Research on Parallel Query System based on peering structure according to claim 6, is characterized in that, described subprocess is monitored host process, so as when described host process accident to withdraw from autoshutdown.
9. the distributed high dimensional indexing Research on Parallel Query System based on peering structure according to claim 1, it is characterized in that, the soft balance module of described load comprises the soft balanced website of a plurality of loads, and the soft balance module of described load is safeguarded described current available working terminal list.
10. the described distributed high dimensional indexing Research on Parallel Query System based on peering structure of according to claim 1-9 any one, is characterized in that, described index block file adopts Hybird Spill Tree data structure.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 201210038115 CN102622414B (en) | 2012-02-17 | 2012-02-17 | Peer-to-peer structure based distributed high-dimensional indexing parallel query framework |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 201210038115 CN102622414B (en) | 2012-02-17 | 2012-02-17 | Peer-to-peer structure based distributed high-dimensional indexing parallel query framework |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN102622414A CN102622414A (en) | 2012-08-01 |
| CN102622414B true CN102622414B (en) | 2013-11-06 |
Family
ID=46562333
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN 201210038115 Active CN102622414B (en) | 2012-02-17 | 2012-02-17 | Peer-to-peer structure based distributed high-dimensional indexing parallel query framework |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102622414B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106970837A (en) * | 2017-03-29 | 2017-07-21 | 联想(北京)有限公司 | A kind of information processing method and electronic equipment |
Families Citing this family (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103455531B (en) * | 2013-02-01 | 2016-12-28 | 深圳信息职业技术学院 | A kind of parallel index method supporting high dimensional data to have inquiry partially in real time |
| CN104572785B (en) * | 2013-10-29 | 2018-07-03 | 阿里巴巴集团控股有限公司 | A kind of distributed method and apparatus for creating index |
| EP3449437A4 (en) * | 2016-04-29 | 2019-12-25 | Cisco Technology, Inc. | Dynamic streaming of query responses |
| CN106681794B (en) * | 2016-12-07 | 2020-04-10 | 长春市三昧动漫设计有限公司 | Interest behavior based distributed virtual environment cache management method |
| CN108279943B (en) * | 2017-01-05 | 2020-09-11 | 腾讯科技（深圳）有限公司 | Index loading method and device |
| CN106686117B (en) * | 2017-01-20 | 2020-04-03 | 郑州云海信息技术有限公司 | Data storage processing system and method of distributed computing cluster |
| CN106844752A (en) * | 2017-02-16 | 2017-06-13 | 中电海康集团有限公司 | A kind of entity relationship searching method and device based on data correlation network model |
| CN108681592B (en) * | 2018-05-15 | 2021-05-25 | 北京三快在线科技有限公司 | Index switching method, device and system and index switching central control device |
| CN108920552B (en) * | 2018-06-19 | 2022-04-29 | 浙江工业大学 | Distributed index method for multi-source large data stream |
| CN109710642B (en) * | 2018-12-18 | 2021-07-27 | 中科曙光国际信息产业有限公司 | Index aggregation parallel processing system based on big data architecture |
| CN111338560B (en) * | 2018-12-19 | 2024-09-17 | 北京奇虎科技有限公司 | Cache reconstruction method and device |
| CN109788068B (en) * | 2019-02-14 | 2020-11-03 | 腾讯科技（深圳）有限公司 | Heartbeat state information reporting method, device and equipment and computer storage medium |
| CN110221910B (en) * | 2019-06-19 | 2022-08-02 | 北京百度网讯科技有限公司 | Method and apparatus for performing MPI jobs |
| CN113568906A (en) * | 2021-07-02 | 2021-10-29 | 浙江工业大学 | Distributed index structure and load balancing method for high-throughput data stream |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8230070B2 (en) * | 2007-11-09 | 2012-07-24 | Manjrasoft Pty. Ltd. | System and method for grid and cloud computing |
| CN101853283B (en) * | 2010-05-21 | 2012-01-04 | 南京邮电大学 | Construction method for multidimensional data-oriented semantic indexing peer-to-peer network |
-
2012
- 2012-02-17 CN CN 201210038115 patent/CN102622414B/en active Active
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106970837A (en) * | 2017-03-29 | 2017-07-21 | 联想(北京)有限公司 | A kind of information processing method and electronic equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN102622414A (en) | 2012-08-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102622414B (en) | Peer-to-peer structure based distributed high-dimensional indexing parallel query framework | |
| US10747714B2 (en) | Scalable distributed data store | |
| Ahmad et al. | A survey on virtual machine migration and server consolidation frameworks for cloud data centers | |
| CN109831500B (en) | Synchronization method for configuration file and Pod in Kubernetes cluster | |
| CN102855284B (en) | The data managing method of a kind of cluster storage system and system | |
| US20170048123A1 (en) | System for controlling switch devices, and device and method for controlling system configuration | |
| CN100451989C (en) | Software testing system and testing method | |
| US10833933B2 (en) | Systems and methods for optimizing a cloud deployed service based on performance metrics of the cloud service | |
| CN102694868B (en) | A kind of group system realizes and task dynamic allocation method | |
| Großmann et al. | Monitoring container services at the network edge | |
| US9262148B2 (en) | Modular architecture for distributed system management | |
| CN101930472A (en) | Parallel query method for distributed database | |
| CN106462575A (en) | Design and implementation of clustered in-memory database | |
| CN104506589A (en) | Resource migration scheduling method based on super fusion storage | |
| US10430723B1 (en) | Storage system with machine learning based skew prediction | |
| CN103383689A (en) | Service process fault detection method, device and service node | |
| CN102377598A (en) | Internet application custody system, equipment and method | |
| CN108173840B (en) | Intelligent logistics terminal integration system based on cloud platform | |
| CN108369404A (en) | The distributed embedded data and Knowledge Management System of integrated PLC historical records | |
| CN111625383B (en) | Process exception event processing method and device, electronic equipment and storage medium | |
| DE102022203266A1 (en) | METHODS AND DEVICES FOR AUTOMATICALLY UPDATING ARTIFICIAL INTELLIGENCE MODELS FOR AUTONOMOUS FACTORIES | |
| US10481800B1 (en) | Network data management protocol redirector | |
| CN115048188A (en) | Network isolation control system and method for container, electronic device and storage medium | |
| CN105069024A (en) | Parallel data acquisition-oriented write access method for distributed file system | |
| CN102339303A (en) | Distributed system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant |