CN106294472A - The querying method of a kind of Hadoop data base HBase and device - Google Patents
The querying method of a kind of Hadoop data base HBase and device Download PDFInfo
- Publication number
- CN106294472A CN106294472A CN201510299125.3A CN201510299125A CN106294472A CN 106294472 A CN106294472 A CN 106294472A CN 201510299125 A CN201510299125 A CN 201510299125A CN 106294472 A CN106294472 A CN 106294472A
- Authority
- CN
- China
- Prior art keywords
- callqueue
- request queue
- queue
- resource
- request
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24564—Applying rules; Deductive queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2453—Query optimisation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/248—Presentation of query results
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Abstract
The invention provides querying method and the device of a kind of Hadoop data base HBase, relate to database field, the problem solving existing HBase querying method existing defects, the method includes: receive the inquiry request of user, and inquiry request antitone sequence is melted into Call inquiry job；Obtain the User Priority of Call inquiry job institute home subscriber；According to the User Priority pre-set and the corresponding relation of Query priority, Call inquiry job is put in a request queue CallQueue in multiple request queue CallQueue；According to the Query priority of request queue CallQueue, process the Call inquiry job in request queue CallQueue, obtain Query Result；Query Result is returned to user.The solution of the present invention, while ensureing overall customer experience, improves the service experience of high-priority users.
Description
Technical field
The present invention relates to database field, particularly to the querying method of a kind of Hadoop data base HBase
And device.
Background technology
HBase (Hadoop Database, Hadoop data base) be one increase income, distributed, towards
Row storage, telescopic non-relational database, HBase builds at HDFS (Hadoop Distributed File
System, Hadoop distributed file system) on, for the access random, real-time to mass data.
HBase uses RPC (Remote Procedure Call Protocol, remote procedure call protocol) mechanism to make
For Primary communication means.Client Client by RPC mechanism with HMaster (Hadoop Master,
Hadoop manager) and HRegionServer (Hadoop Region Server, the service of Hadoop area
Device) communicate to send operation requests.The operation of HMaster response management class, HRegionServer
The read-write operation of response data.
The querying method of existing HBase uses priority scheduling strategy based on single request queue CallQueue,
Job queue maintenance costs is big.When there being new job to arrive, need, to whole queue reordering, to add
Wait the maintenance costs of job queue.And the most according to priority order fulfils assignment scheduling, it is also possible to lead
The operation causing low priority can not get performing for a long time so that the service quality of whole HBase cluster cannot be protected
Card.And do not consider how to provide better service to experience into high-priority users.
The querying method of existing HBase is mainly for single application, the QueryTicket business of user.When many
When individual user uses HBase to do QueryTicket simultaneously, it is impossible to neatly according to user identity, priority,
Distribute suitable query resource, it is impossible to realize the big data, services of cloud computing form.
Summary of the invention
The technical problem to be solved in the present invention be to provide a kind of Hadoop data base HBase querying method and
Device, solves the querying method of HBase in prior art and cannot ensure the experience of overall user, it is impossible to for
High-priority users provides more good service, it is impossible to the problem realizing the big data, services of cloud computing body.
For solving above-mentioned technical problem, embodiments of the invention provide a kind of Hadoop data base HBase's
Querying method, including:
Receive the inquiry request of user, and described inquiry request antitone sequence is melted into Call inquiry job；
Obtain the User Priority of described Call inquiry job institute home subscriber；
According to the User Priority pre-set and the corresponding relation of Query priority, described Call is inquired about
Operation is put in a request queue CallQueue in multiple request queue CallQueue, the most each
Request queue CallQueue has a Query priority, looking between different request queue CallQueue
Inquiry priority is different；
According to the Query priority of described request queue CallQueue, process described request queue CallQueue
In Call inquiry job, obtain Query Result；And
Described Query Result is returned to user.
Wherein, described querying method also includes:
Set up a resource tracing thread, monitor the queuing message of each request queue CallQueue, and root
According to described queuing message, described request queue CallQueue is carried out dynamic resource scheduling.
Wherein, described queuing message includes load number and average response delay, and described dynamic resource includes please
Seek the queue resource of queue CallQueue；
Correspondingly, described set up a resource tracing thread, monitor the team of each request queue CallQueue
Column information, and according to described queuing message, described request queue CallQueue is carried out dynamic resource scheduling,
Specifically include:
Set up a resource tracing thread, monitor the load number of each request queue CallQueue with average
Response delay；
According to described load number and described average response time delay, the team to described request queue CallQueue
Row resource is scheduling.
Wherein, described according to described load number with described average response time delay, to described request queue
The queue resource of CallQueue is scheduling, and specifically includes:
According to described load number and described average response time delay, from multiple request queue CallQueue,
Obtain the maximum threshold that load number limits less than queue resource, and average response time delay is more than predetermined threshold value
The idle queues resource of request queue CallQueue；
Described idle queues resource is distributed to load number more than queue resource limit maximum threshold or
Average response time delay is less than the request queue CallQueue of predetermined threshold value.
Wherein, each request queue CallQueue is provided with different process thread resources, described team in advance
Column information includes Call inquiry job quantity and average response delay, and described dynamic resource includes request queue
The process thread resources of CallQueue；
Correspondingly, set up a resource tracing thread, monitor the queue letter of each request queue CallQueue
Breath, and according to described queuing message, described request queue CallQueue is carried out dynamic resource scheduling, specifically
Including:
Set up a resource tracing thread, monitor the Call inquiry job number of each request queue CallQueue
Amount and average response delay；
According to described Call inquiry job quantity and described average response time delay, to described request queue
The process thread resources of CallQueue is scheduling.
Wherein, described according to described Call inquiry job quantity and described average response time delay, ask described
The process thread resources seeking queue CallQueue is scheduling, and specifically includes:
According to described Call inquiry job quantity and described average response time delay, from multiple request queues
In CallQueue, obtain Call inquiry job quantity less than processing the number of threads that thread resources has, and
Average response time delay processes thread resources more than the idle of the request queue CallQueue of predetermined threshold value；
The described idle thread resources that processes is distributed to Call inquiry job quantity more than processing thread resources tool
Some number of threads or average response time delay are less than the request queue CallQueue of predetermined threshold value.
Wherein, the User Priority of described acquisition described Call inquiry job institute home subscriber, particularly as follows:
According to the network interconnection Protocol IP address of described Call inquiry job, obtain described Call inquiry job
The User Priority of institute's home subscriber.
For solving above-mentioned technical problem, embodiments of the invention also provide for a kind of Hadoop data base HBase
Inquiry unit, including:
Receiver module, for receiving the inquiry request of user, and is melted into Call by described inquiry request antitone sequence
Inquiry job；
Acquisition module, for obtaining the User Priority of described Call inquiry job institute home subscriber；
Storage module, the User Priority pre-set for basis and the corresponding relation of Query priority, will
Described Call inquiry job is put into a request queue CallQueue in multiple request queue CallQueue
In, the most each request queue CallQueue has a Query priority, different request queues
Query priority between CallQueue is different；
Processing module, for the Query priority according to described request queue CallQueue, processes described asking
Seek the Call inquiry job in queue CallQueue, obtain Query Result；And
Return module, for described Query Result is returned to user.
Wherein, described inquiry unit also includes:
Scheduler module, is used for setting up a resource tracing thread, monitors each request queue CallQueue's
Queuing message, and according to described queuing message, described request queue CallQueue is carried out dynamic resource scheduling.
Wherein, described queuing message includes load number and average response delay, and described dynamic resource includes please
Seek the queue resource of queue CallQueue；
Correspondingly, described scheduler module specifically includes:
First monitors module, is used for setting up a resource tracing thread, monitors each request queue CallQueue
Load number and average response delay；
First scheduling sublayer module, for according to described load number and described average response time delay, asking described
The queue resource seeking queue CallQueue is scheduling.
Wherein, each request queue CallQueue is provided with different process thread resources, described team in advance
Column information includes Call inquiry job quantity and average response delay, and described dynamic resource includes request queue
The process thread resources of CallQueue；
Correspondingly, described scheduler module specifically includes:
Second monitors module, is used for setting up a resource tracing thread, monitors each request queue CallQueue
Call inquiry job quantity and average response delay；
Second scheduling sublayer module, is used for according to described Call inquiry job quantity and described average response time delay,
The process thread resources of described request queue CallQueue is scheduling.
Wherein, described acquisition module is further used for the network interconnection agreement according to described Call inquiry job
IP address, obtains the User Priority of described Call inquiry job institute home subscriber.
Having the beneficial effect that of the technique scheme of the present invention:
In embodiments of the present invention, first the inquiry request antitone sequence chemical conversion Call of the user received is inquired about
Operation；Then the User Priority of Call inquiry job institute home subscriber is obtained；Further according to pre-set
User Priority and the corresponding relation of Query priority, be put into multiple request queue by Call inquiry job
In a request queue CallQueue in CallQueue, the most each request queue CallQueue has
One Query priority, the Query priority between different request queue CallQueue is different；Then basis
The Query priority of request queue CallQueue, the Call inquiry processed in request queue CallQueue is made
Industry, obtains Query Result；Finally Query Result is returned to user.By using multi-request queue
CallQueue dispatches, and makes user's inquiry request of different priorities enjoy different query resource, is ensureing
While overall customer experience, provide better service for high-priority users and experience, it is achieved that cloud meter
Calculate the big data, services of form.
Accompanying drawing explanation
Fig. 1 is the querying method flow chart of the Hadoop data base HBase of the embodiment of the present invention；
Fig. 2 is that the querying method one of the Hadoop data base HBase of the embodiment of the present invention implements system
The first structural representation；
Fig. 3 is that the querying method one of the Hadoop data base HBase of the embodiment of the present invention implements system
The second structural representation；
Fig. 4 is that the querying method application one of the Hadoop data base HBase of the embodiment of the present invention implements
The flow chart of system；
Fig. 5 is the structural representation of the inquiry unit of the Hadoop data base HBase of the embodiment of the present invention.
Detailed description of the invention
For making the technical problem to be solved in the present invention, technical scheme and advantage clearer, below in conjunction with attached
Figure and specific embodiment are described in detail.
The querying method of the Hadoop data base HBase of the embodiment of the present invention, is ensureing overall customer experience
While, provide better service for high-priority users and experience.
As it is shown in figure 1, the querying method of the Hadoop data base HBase of the embodiment of the present invention, including:
Step 11, receives the inquiry request of user, and by the chemical conversion Call inquiry of described inquiry request antitone sequence
Operation.
Step 12, obtains the User Priority of described Call inquiry job institute home subscriber.
Step 13, according to the User Priority pre-set and the corresponding relation of Query priority, by described
Call inquiry job is put in a request queue CallQueue in multiple request queue CallQueue,
The most each request queue CallQueue has a Query priority, different request queue CallQueue
Between Query priority different.
Here, User Priority and Query priority are according to priority level one_to_one corresponding, such as the highest user priority
The level the highest Query priority of correspondence, the corresponding minimum User Priority of minimum User Priority.And each request team
Row CallQueue enjoys the query resource of different size according to the difference of Query priority, as having the highest looking into
The request queue CallQueue asking priority enjoys highest query resource, has minimum inquiry preferential
The request queue CallQueue of level enjoys the query resource of minimum specification.
Step 14, according to the Query priority of described request queue CallQueue, processes described request queue
Call inquiry job in CallQueue, obtains Query Result；And
Step 15, returns to user by described Query Result.
The querying method of the Hadoop data base HBase of the embodiment of the present invention, by using multi-request queue
CallQueue dispatches, and can distribute suitable query resource neatly according to User Priority, make difference
User's inquiry request of priority enjoys the query resource of different size, thus finally controls different priorities and use
The response delay of the inquiry request at family is variant.While ensureing overall customer experience, use for high priority
Family provides better service and experiences, it is achieved that the big data, services of cloud computing form.
In order to make multiple request queue CallQueue all keep higher operational efficiency, thus ensure overall use
The experience at family, in the specific embodiment of the present invention, described querying method can also include:
Step 16, sets up a resource tracing thread, monitors the queue letter of each request queue CallQueue
Breath, and according to described queuing message, described request queue CallQueue is carried out dynamic resource scheduling.
Of course, it should be understood that step 16 can be for request queue mentioned in above-mentioned steps 13
CallQueue carries out dynamic resource scheduling.
Now, by request queue CallQueue is carried out dynamic resource scheduling, it is achieved that multiple request teams
Dynamic resource balance between row CallQueue, makes multiple request queue CallQueue can be maintained at relatively
High operational efficiency, it is ensured that the experience of overall user.
Here, the queuing message listened to can be stored in journal file, convenient reading and use.
Wherein, described queuing message can include load number and average response delay, and described dynamic resource can wrap
Include the queue resource of request queue CallQueue；
Correspondingly, above-mentioned steps 16 specifically may include that
Step 161, sets up a resource tracing thread, monitors the load of each request queue CallQueue
Quantity and average response delay；
Step 162, according to described load number and described average response time delay, to described request queue
The queue resource of CallQueue is scheduling.
Now, by monitoring the load number of request queue CallQueue and average response delay, with to asking
The queue resource seeking queue CallQueue is scheduling, and can make the request queue CallQueue of heavier loads
Also there is the queue resource of abundance.Ensure that the operational efficiency of the request queue CallQueue of heavier loads,
The request queue CallQueue operational efficiency avoiding overload is the slowest, the problem affecting Consumer's Experience,
Thus it is effectively increased the experience of overall user.
Further, above-mentioned steps 162 specifically may include that
Step 1621, according to described load number and described average response time delay, from multiple request queues
In CallQueue, obtain the maximum threshold that load number limits less than queue resource, and during average response
Prolong the idle queues resource of the request queue CallQueue more than predetermined threshold value.
Here, owing to enjoying the response delay that the request queue CallQueue of different size query resource requires
Also different, therefore can set the average of each request queue CallQueue requirement previously according to Query priority
The threshold value of response delay, with the average response time delay according to this threshold decision request queue CallQueue whether
Meet the requirement of corresponding Query priority.Thus ensure the Consumer's Experience of different priority users.
Here, if the load number of certain request queue CallQueue is less than the gate that queue resource limits
Limit value, and average response time delay is more than predetermined threshold value, then this request queue CallQueue has unnecessary team
Row resource, and meet the requirement of self Query priority.
Step 1622, distributes to described idle queues resource load number and limits more than queue resource
Big threshold value or average response time delay are less than the request queue CallQueue of predetermined threshold value.
Here, if the load number of certain request queue CallQueue is more than the gate that queue resource limits
Limit value, and average response time delay is less than predetermined threshold value, the then heavier loads of this request queue CallQueue,
Queue resource is not enough, and cannot meet the requirement of self Query priority.
Now, by will obtain idle queues resource to the request queue CallQueue of heavier loads.Protect
Demonstrate,prove the operational efficiency of the request queue CallQueue of heavier loads, it is to avoid the request queue of overload
CallQueue operational efficiency is the slowest, the problem affecting Consumer's Experience, thus is effectively increased overall user's
Experience.
Wherein, each request queue CallQueue is provided with different process thread resources, described team in advance
Column information can include Call inquiry job quantity and average response delay, and described dynamic resource can include request
The process thread resources of queue CallQueue；
Correspondingly, above-mentioned steps 16 specifically may include that
Step 163, sets up a resource tracing thread, monitors the Call of each request queue CallQueue
Inquiry job quantity and average response delay；
Step 164, according to described Call inquiry job quantity and described average response time delay, to described request
The process thread resources of queue CallQueue is scheduling.
Now, by monitoring the Call inquiry job quantity of request queue CallQueue and average response delay,
It is scheduling with the process thread resources to request queue CallQueue, can make that there is more Call inquiry and make
The request queue CallQueue of industry quantity has the process thread resources of abundance, thus ensure that each request
Being smoothed out of queue CallQueue inquiry job, and then ensure that Consumer's Experience.
Further, above-mentioned steps 164 specifically may include that
Step 1641, according to described Call inquiry job quantity and described average response time delay, asks from multiple
Ask in queue CallQueue, obtain Call inquiry job quantity less than processing the Thread Count that thread resources has
Amount, and the idle process thread resources that average response time delay is more than the request queue CallQueue of predetermined threshold value.
Step 1642, distributes to Call inquiry job quantity more than processing by the described idle thread resources that processes
Number of threads that thread resources has or average response time delay are less than the request queue CallQueue of predetermined threshold value.
Now, a fairly large number of by the idle process thread resources obtained is distributed to Call inquiry job
Request queue CallQueue, it is ensured that Call inquiry job a fairly large number of request queue CallQueue looks into
What inquiry processed is smoothed out, thus ensure that treatment effeciency and Consumer's Experience.
Preferably, after above-mentioned steps 1642, if the described idle request queue processing thread resources place
The Call inquiry job quantity of CallQueue increases to the number of threads having more than self process thread resources,
Then regain the described idle thread resources that processes to this request queue CallQueue.
In the specific embodiment of the present invention, above-mentioned steps 12 concretely:
Step 121, according to the network interconnection Protocol IP address of described Call inquiry job, obtains described Call
The User Priority of inquiry job institute home subscriber.
Now, can be accurately obtained what Call inquiry job was belonged to by the IP address of Call inquiry job
User, and then get the User Priority of Call inquiry job institute home subscriber, improve the standard of operation
Really property and treatment effeciency.
To the present invention one implements embodiment and is illustrated below below.
As Figure 2-3, the querying method of the Hadoop data base HBase of the embodiment of the present invention can pass through
Answer Listener module, connect Connection module, CallQueues module, process Handlers mould
Block, request queue monitor CallQueues Tracker module, Hadoop interprocess communication scheduling
HBaseIPCScheduler module and response Responder module realize.
Wherein, above-mentioned steps 11 can realize by answering Listener module.Listener module includes many
Individual sub-thread reads Reader module and a main thread Listener module.Main thread Listener module is born
Duty receives the inquiry request of user.Sub-line journey Reader module is responsible for connecting concrete inquiry operation, will look into
Ask request antitone sequence chemical conversion Call inquiry job.
Wherein, above-mentioned steps 12 and step 13 can realize by connecting Connection module.Concrete,
Connection module reads Call inquiry job from Listener module, and according to Call inquiry job
The User Priority of IP address acquisition institute home subscriber, finally according to the User Priority pre-set and looking into
Ask the corresponding relation of priority, Call inquiry job is put in multiple request queue CallQueue
In request queue CallQueue.
Here, multiple request queue CallQueue can be managed by CallQueues module, and request
The quantity of queue CallQueue can be configured by the manager of HBase data base.Due to User Priority
With the Query priority one_to_one corresponding of request queue CallQueue, so manager is by arranging user priority
Quantity n of level, just can determine the quantity of request queue CallQueue.If the quantity of User Priority is n
Individual, then the quantity of request queue CallQueue is also n: P1CallQueue, P2CallQueue ...,
PnCallQueue, wherein n is the integer more than or equal to 1.
Further, above-mentioned steps 14 can realize by processing Handlers module.Wherein process Handlers
Module includes multiple Handlers unit corresponding with request queue CallQueue, each Handlers
Unit, according to the Query priority of corresponding requests queue CallQueue, processes this request queue CallQueue
In Call inquiry job.
As request queue CallQueue includes: P1CallQueue, P2CallQueue ..., PnCallQueue,
Then Handlers module includes: P1Handlers, P2Handlers ..., PnHandlers.And P1Handlers
Processing the inquiry job in P1CallQueue queue, P1Handlers processes in P1CallQueue queue
Inquiry job, PnHandlers processes the inquiry job in PnCallQueue queue.As it is shown on figure 3, it is false
Determining n is 3, then the quantity of request queue CallQueue be 3: P1CallQueue, P2CallQueue,
P3CallQueue, and Handlers module includes: P1Handlers, P2Handlers, P3Handlers.
Here, each request queue CallQueue is provided with different process thread resources, this process in advance
Thread resources i.e. Handlers unit is according to the Query priority of request queue CallQueue, pre-assigned
Process thread resources.Wherein, the process thread resources summation of all Handlers unit not can exceed that
The total resources that Handlers module has: P1Handlers+P2Handlers+ ...+PnHandlers≤
hbase.regionserver.handler.count。
Wherein, above-mentioned steps 16 can monitor CallQueues Tracker module, Hadoop by request queue
Interprocess communication scheduling HBaseIPCScheduler module realizes.
The newly-built resource tracing thread of CallQueues Tracker module, in real time to each request queue
The load number of CallQueue, Call inquiry job quantity and average response delay etc. are added up, and will
Statistical result is stored in journal file, as the foundation of dynamic resource scheduling.
When HBaseIPCScheduler module is according to load number, Call inquiry job quantity and average response
Prolong, request queue CallQueue is carried out dynamic resource scheduling.Here, HBaseIPCScheduler mould
The scheduling strategy of block can refer to the model realization of capacity scheduling device Yarn, uses different priority users not
Same strategy, it is ensured that high-priority users inquiry job first carries out, and takes into account low priority user inquiry job
Fairness.
Concrete, HBaseIPCScheduler module is according to load number and average response delay, it is judged that
When certain request queue CallQueue has idle queues resource, idle queues Resource dynamic allocation can be given
The request queue CallQueue of heavier loads, with the place of the heavier request queue CallQueue of proof load
Reason efficiency.HBaseIPCScheduler module according to Call inquiry job quantity and average response delay,
Judge that the process Handlers of certain request queue CallQueue has idle when processing thread resources, can be by
Process thread resources and be dynamically assigned to Call inquiry job a fairly large number of request queue CallQueue, to protect
Being smoothed out of card request queue CallQueue query processing, improves treatment effeciency, it is ensured that overall use
The experience at family.Also can make in the Call inquiry of the request queue CallQueue that the free time processes thread resources place
When industry quantity increases, regain this free time process thread resources.
As when Call inquiry job quantity pending in request queue P1CallQueue is less than P1Handlers
The number of threads having, and when average response time delay is more than predetermined threshold value, then can be by unnecessary process thread money
Source is distributed to P2CallQueue or P3CallQueue of heavier loads and is used.When P1CallQueue loads
During increase, then regain the process thread resources separated, for processing the operation of P1CallQueue.
Further, above-mentioned steps 15 can be realized by response Responder module, Responder module
Query Result is returned to user.
As shown in Figure 4, application system as shown in Figure 2, the Hadoop data base of the embodiment of the present invention
The querying method of HBase, including:
Step S01, configuration parameter: the total resources that quantity n of User Priority, Handlers module have
(hbase.regionserver.handler.count), each request queue CallQueue (P1CallQueue,
P2CallQueue ..., PnCallQueue) corresponding Handlers unit (P1Handlers,
P2Handlers ..., PnHandlers) process thread resources etc..Such as: n=3,
Hbase.regionserver.handler.count=100, P1Handlers=50, P2Handlers=30,
P3Handlers=20.Then according to configuration parameter, CallQueues module and Handlers module are initialized.
Step S02, Listener module receives the inquiry request of user, and inquiry request antitone sequence is melted into Call
Inquiry job.
Step S03, Connection module reads Call inquiry job from Listener module, according to Call
The User Priority of the IP address acquisition institute home subscriber of inquiry job, and according to User Priority and inquiry
The corresponding relation of priority, puts into Call inquiry job in the CallQueue queue of correspondence.
Step S04, CallQueueTrackers module newly-built resource tracing thread, timing is to each request team
The row load number of CallQueue, Call inquiry job quantity, average response time etc. are added up, will
Result is stored in journal file, and feeds back to HBaseIPCScheduler module, as dynamic adjustresources
Foundation.
Step S05, HBaseIPCScheduler module according to load number, Call inquiry job quantity and
Multiple request queue CallQueue are scheduling by average response time.
Step S06, Handlers module performs the inquiry job of respective request queue CallQueue, will look into
Ask result and return to Responder module.
Step S07, Query Result is returned to user by Responder module.
The querying method of the Hadoop data base HBase of the embodiment of the present invention, by using multi-request queue
CallQueue dispatches, and can distribute suitable query resource neatly according to User Priority, make difference
User's inquiry request of priority enjoys the query resource of different size, thus finally controls different priorities and use
The response delay of the inquiry request at family is variant.Provide better service for high-priority users to experience,
Achieve the big data, services of cloud computing form.And make multiple request queue CallQueue be held at higher
Operational efficiency, the experience of overall user has been effectively ensured.
In prior art, the RPC communication of HRegionServer end is implemented as a reactor Reactor
The request/response queuing model of pattern.All of inquiry request is put into after HRegionServer end is decoded
Request queue CallQueue, gives processor Handler thread pool and processes, and Handler thread pool is constantly
From request queue CallQueue, obtain inquiry request, give HRegionServer and perform inquiry, and will
Execution result, with response queue's ResponseQueue form, returns to inquiring client terminal successively.
The querying method of existing HBase uses priority scheduling strategy based on single request queue CallQueue,
The experience of overall user cannot be ensured, it is impossible to provide more good service for high-priority users, it is impossible to realize cloud
Calculate the big data, services of body.
The querying method of the Hadoop data base HBase of the embodiment of the present invention, solves prior art not
Foot, while ensureing overall customer experience, provides better service for high-priority users and experiences,
Achieve the big data, services of cloud computing form.
As it is shown in figure 5, embodiments of the invention additionally provide the inquiry of a kind of Hadoop data base HBase
Device, including:
Receiver module, for receiving the inquiry request of user, and is melted into Call by described inquiry request antitone sequence
Inquiry job；
Acquisition module, for obtaining the User Priority of described Call inquiry job institute home subscriber；
Storage module, the User Priority pre-set for basis and the corresponding relation of Query priority, will
Described Call inquiry job is put into a request queue CallQueue in multiple request queue CallQueue
In, the most each request queue CallQueue has a Query priority, different request queues
Query priority between CallQueue is different；
Processing module, for the Query priority according to described request queue CallQueue, processes described asking
Seek the Call inquiry job in queue CallQueue, obtain Query Result；And
Return module, for described Query Result is returned to user.
The inquiry unit of the Hadoop data base HBase of the embodiment of the present invention, by using multi-request queue
CallQueue dispatches, and can distribute suitable query resource neatly according to User Priority, make difference
User's inquiry request of priority enjoys the query resource of different size, thus finally controls different priorities and use
The response delay of the inquiry request at family is variant.While ensureing overall customer experience, use for high priority
Family provides better service and experiences, it is achieved that the big data, services of cloud computing form.
In the specific embodiment of the present invention, described inquiry unit also includes:
Scheduler module, is used for setting up a resource tracing thread, monitors each request queue CallQueue's
Queuing message, and according to described queuing message, described request queue CallQueue is carried out dynamic resource scheduling.
Wherein, described queuing message includes load number and average response delay, and described dynamic resource includes please
Seek the queue resource of queue CallQueue；
Correspondingly, described scheduler module specifically includes:
First monitors module, is used for setting up a resource tracing thread, monitors each request queue CallQueue
Load number and average response delay；
First scheduling sublayer module, for according to described load number and described average response time delay, asking described
The queue resource seeking queue CallQueue is scheduling.
Further, described first scheduling sublayer module includes:
Second acquisition module, for according to described load number and described average response time delay, from multiple requests
In queue CallQueue, obtain the maximum threshold that load number limits less than queue resource, and the loudest
Time delay is answered to be more than the idle queues resource of request queue CallQueue of predetermined threshold value；
First distribution module, limits more than queue resource for described idle queues resource is distributed to load number
Fixed maximum threshold or average response time delay are less than the request queue CallQueue of predetermined threshold value.
Wherein, each request queue CallQueue is provided with different process thread resources, described team in advance
Column information includes Call inquiry job quantity and average response delay, and described dynamic resource includes request queue
The process thread resources of CallQueue；
Correspondingly, described scheduler module specifically includes:
Second monitors module, is used for setting up a resource tracing thread, monitors each request queue CallQueue
Call inquiry job quantity and average response delay；
Second scheduling sublayer module, is used for according to described Call inquiry job quantity and described average response time delay,
The process thread resources of described request queue CallQueue is scheduling.
Further, described second scheduling sublayer module includes:
3rd acquisition module, is used for according to described Call inquiry job quantity and described average response time delay,
From multiple request queue CallQueue, obtain Call inquiry job quantity and have less than processing thread resources
Number of threads, and the idle process that average response time delay is more than the request queue CallQueue of predetermined threshold value
Thread resources；
Second distribution module, for distributing to Call inquiry job quantity by the described idle thread resources that processes
The number of threads having more than process thread resources or average response time delay are less than the request queue of predetermined threshold value
CallQueue。
Wherein, described acquisition module is further used for the network interconnection agreement according to described Call inquiry job
IP address, obtains the User Priority of described Call inquiry job institute home subscriber.
To the present invention one implements embodiment and is illustrated below below.
As Figure 2-3, above-mentioned receiver module can be realized by Listener module, above-mentioned acquisition and depositing
Can be realized by Connection module, above-mentioned processing module can be realized by Handlers module, above-mentioned
Scheduler module can monitor CallQueues Tracker module and Hadoop interprocess communication by request queue
Scheduling HBaseIPCScheduler module realizes, and above-mentioned return module can be realized by Responder module.
The querying method of the Hadoop data base HBase of the embodiment of the present invention, by using multi-request queue
CallQueue dispatches, and can distribute suitable query resource neatly according to User Priority, make difference
User's inquiry request of priority enjoys the query resource of different size, thus finally controls different priorities and use
The response delay of the inquiry request at family is variant.Provide better service for high-priority users to experience,
Achieve the big data, services of cloud computing form.And make multiple request queue CallQueue be held at higher
Operational efficiency, the experience of overall user has been effectively ensured.
The inquiry unit of the Hadoop data base HBase of the embodiment of the present invention, solves prior art not
Foot, while ensureing overall customer experience, provides better service for high-priority users and experiences,
Achieve the big data, services of cloud computing form.
It should be noted that this inquiry unit realizing Hadoop data base HBase is and above-mentioned Hadoop
The device that the querying method of data base HBase is corresponding, wherein all realization sides in said method embodiment
Formula, all be applicable to the embodiment of this device, also can reach same technique effect.
The above is the preferred embodiment of the present invention, it is noted that for the common skill of the art
For art personnel, on the premise of without departing from principle of the present invention, it is also possible to make some improvements and modifications,
These improvements and modifications also should be regarded as protection scope of the present invention.
Claims (12)
1. the querying method of a Hadoop data base HBase, it is characterised in that including:
Receive the inquiry request of user, and described inquiry request antitone sequence is melted into Call inquiry job；
Obtain the User Priority of described Call inquiry job institute home subscriber；
According to the User Priority pre-set and the corresponding relation of Query priority, described Call is inquired about
Operation is put in a request queue CallQueue in multiple request queue CallQueue, the most each
Request queue CallQueue has a Query priority, looking between different request queue CallQueue
Inquiry priority is different；
According to the Query priority of described request queue CallQueue, process described request queue CallQueue
In Call inquiry job, obtain Query Result；And
Described Query Result is returned to user.
Querying method the most according to claim 1, it is characterised in that described querying method also includes:
Set up a resource tracing thread, monitor the queuing message of each request queue CallQueue, and root
According to described queuing message, described request queue CallQueue is carried out dynamic resource scheduling.
Querying method the most according to claim 2, it is characterised in that described queuing message includes bearing
Carrying quantity and average response delay, described dynamic resource includes the queue resource of request queue CallQueue；
Correspondingly, described set up a resource tracing thread, monitor the team of each request queue CallQueue
Column information, and according to described queuing message, described request queue CallQueue is carried out dynamic resource scheduling,
Specifically include:
Set up a resource tracing thread, monitor the load number of each request queue CallQueue with average
Response delay；
According to described load number and described average response time delay, the team to described request queue CallQueue
Row resource is scheduling.
Querying method the most according to claim 3, it is characterised in that described according to described load number
Amount and described average response time delay, be scheduling the queue resource of described request queue CallQueue, tool
Body includes:
According to described load number and described average response time delay, from multiple request queue CallQueue,
Obtain the maximum threshold that load number limits less than queue resource, and average response time delay is more than predetermined threshold value
The idle queues resource of request queue CallQueue；
Described idle queues resource is distributed to load number more than queue resource limit maximum threshold or
Average response time delay is less than the request queue CallQueue of predetermined threshold value.
Querying method the most according to claim 2, it is characterised in that each request queue CallQueue
Being provided with different process thread resources in advance, described queuing message includes Call inquiry job quantity peace
All response delays, described dynamic resource includes the process thread resources of request queue CallQueue；
Correspondingly, set up a resource tracing thread, monitor the queue letter of each request queue CallQueue
Breath, and according to described queuing message, described request queue CallQueue is carried out dynamic resource scheduling, specifically
Including:
Set up a resource tracing thread, monitor the Call inquiry job number of each request queue CallQueue
Amount and average response delay；
According to described Call inquiry job quantity and described average response time delay, to described request queue
The process thread resources of CallQueue is scheduling.
Querying method the most according to claim 5, it is characterised in that described look into according to described Call
Ask operation quantity and described average response time delay, the process thread resources to described request queue CallQueue
It is scheduling, specifically includes:
According to described Call inquiry job quantity and described average response time delay, from multiple request queues
In CallQueue, obtain Call inquiry job quantity less than processing the number of threads that thread resources has, and
Average response time delay processes thread resources more than the idle of the request queue CallQueue of predetermined threshold value；
The described idle thread resources that processes is distributed to Call inquiry job quantity more than processing thread resources tool
Some number of threads or average response time delay are less than the request queue CallQueue of predetermined threshold value.
Querying method the most according to claim 1, it is characterised in that the described Call of described acquisition looks into
Ask the User Priority of operation institute home subscriber, particularly as follows:
According to the network interconnection Protocol IP address of described Call inquiry job, obtain described Call inquiry job
The User Priority of institute's home subscriber.
8. the inquiry unit of a Hadoop data base HBase, it is characterised in that including:
Receiver module, for receiving the inquiry request of user, and is melted into Call by described inquiry request antitone sequence
Inquiry job；
Acquisition module, for obtaining the User Priority of described Call inquiry job institute home subscriber；
Storage module, the User Priority pre-set for basis and the corresponding relation of Query priority, will
Described Call inquiry job is put into a request queue CallQueue in multiple request queue CallQueue
In, the most each request queue CallQueue has a Query priority, different request queues
Query priority between CallQueue is different；
Processing module, for the Query priority according to described request queue CallQueue, processes described asking
Seek the Call inquiry job in queue CallQueue, obtain Query Result；And
Return module, for described Query Result is returned to user.
Inquiry unit the most according to claim 8, it is characterised in that described inquiry unit also includes:
Scheduler module, is used for setting up a resource tracing thread, monitors each request queue CallQueue's
Queuing message, and according to described queuing message, described request queue CallQueue is carried out dynamic resource scheduling.
Inquiry unit the most according to claim 9, it is characterised in that described queuing message includes bearing
Carrying quantity and average response delay, described dynamic resource includes the queue resource of request queue CallQueue；
Correspondingly, described scheduler module specifically includes:
First monitors module, is used for setting up a resource tracing thread, monitors each request queue CallQueue
Load number and average response delay；
First scheduling sublayer module, for according to described load number and described average response time delay, asking described
The queue resource seeking queue CallQueue is scheduling.
11. inquiry units according to claim 9, it is characterised in that each request queue
CallQueue is provided with different process thread resources in advance, and described queuing message includes Call inquiry job
Quantity and average response delay, described dynamic resource includes the process thread resources of request queue CallQueue；
Correspondingly, described scheduler module specifically includes:
Second monitors module, is used for setting up a resource tracing thread, monitors each request queue CallQueue
Call inquiry job quantity and average response delay；
Second scheduling sublayer module, is used for according to described Call inquiry job quantity and described average response time delay,
The process thread resources of described request queue CallQueue is scheduling.
12. inquiry units according to claim 8, it is characterised in that described acquisition module is further
For the network interconnection Protocol IP address according to described Call inquiry job, obtain described Call inquiry job
The User Priority of institute's home subscriber.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510299125.3A CN106294472B (en) | 2015-06-03 | 2015-06-03 | A kind of querying method and device of Hadoop database HBase |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510299125.3A CN106294472B (en) | 2015-06-03 | 2015-06-03 | A kind of querying method and device of Hadoop database HBase |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106294472A true CN106294472A (en) | 2017-01-04 |
| CN106294472B CN106294472B (en) | 2019-08-02 |
Family
ID=57656173
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510299125.3A Active CN106294472B (en) | 2015-06-03 | 2015-06-03 | A kind of querying method and device of Hadoop database HBase |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106294472B (en) |
Cited By (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107943854A (en) * | 2017-11-03 | 2018-04-20 | 深圳前海微众银行股份有限公司 | Reference service inquiry method, apparatus and computer-readable recording medium |
| CN108241539A (en) * | 2018-01-03 | 2018-07-03 | 百度在线网络技术（北京）有限公司 | Interactive big data querying method, device, storage medium and terminal device based on distributed system |
| CN108459825A (en) * | 2018-01-30 | 2018-08-28 | 深圳神州数码云科数据技术有限公司 | A kind of storage system service method of quality control and device |
| CN108595254A (en) * | 2018-03-09 | 2018-09-28 | 北京永洪商智科技有限公司 | A kind of query scheduling method |
| CN109976910A (en) * | 2019-03-20 | 2019-07-05 | 跬云(上海)信息科技有限公司 | Querying method and device based on precomputation OLAP model |
| CN110489447A (en) * | 2019-07-16 | 2019-11-22 | 招联消费金融有限公司 | Data query method, apparatus, computer equipment and storage medium |
| CN110532227A (en) * | 2019-08-23 | 2019-12-03 | 广东浪潮大数据研究有限公司 | Operation quantity querying method and device |
| CN110704489A (en) * | 2019-09-29 | 2020-01-17 | 苏州浪潮智能科技有限公司 | Database query method, device, equipment and computer storage medium |
| CN110825732A (en) * | 2019-09-20 | 2020-02-21 | 广州亚美信息科技有限公司 | Data query method and device, computer equipment and readable storage medium |
| CN111078735A (en) * | 2019-11-04 | 2020-04-28 | 苏宁云计算有限公司 | Query request processing method and device |
| CN111385255A (en) * | 2018-12-28 | 2020-07-07 | 北京金山云网络技术有限公司 | Asynchronous call implementation method and device, server and server cluster |
| CN112003915A (en) * | 2020-08-14 | 2020-11-27 | 苏州浪潮智能科技有限公司 | File access method, device, equipment and readable medium |
| CN113111083A (en) * | 2021-03-31 | 2021-07-13 | 北京沃东天骏信息技术有限公司 | Method, device, equipment, storage medium and program product for data query |
| CN114466079A (en) * | 2022-02-09 | 2022-05-10 | 星环信息科技(上海)股份有限公司 | Request processing method, device, proxy server and storage medium |
| CN115051981A (en) * | 2022-05-12 | 2022-09-13 | 中国农业银行股份有限公司 | Zookeeper-based asynchronous downloading method and device |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101477568A (en) * | 2009-02-12 | 2009-07-08 | 清华大学 | Integrated retrieval method for structured data and non-structured data |
| US20100218191A1 (en) * | 2009-02-24 | 2010-08-26 | Kannan Shivkumar | Apparatus and Method for Processing Management Requests |
| CN102984275A (en) * | 2012-12-14 | 2013-03-20 | 北京奇虎科技有限公司 | Method and browser for web downloading |
| CN103036956A (en) * | 2012-11-30 | 2013-04-10 | 航天恒星科技有限公司 | Filing system and implement method of distributed configured massive data |
| CN103177035A (en) * | 2011-12-26 | 2013-06-26 | 中国银联股份有限公司 | Data query device and data query method in data base |
| CN103780505A (en) * | 2014-01-27 | 2014-05-07 | 中国联合网络通信集团有限公司 | Business handling method and device |
-
2015
- 2015-06-03 CN CN201510299125.3A patent/CN106294472B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101477568A (en) * | 2009-02-12 | 2009-07-08 | 清华大学 | Integrated retrieval method for structured data and non-structured data |
| US20100218191A1 (en) * | 2009-02-24 | 2010-08-26 | Kannan Shivkumar | Apparatus and Method for Processing Management Requests |
| CN103177035A (en) * | 2011-12-26 | 2013-06-26 | 中国银联股份有限公司 | Data query device and data query method in data base |
| CN103036956A (en) * | 2012-11-30 | 2013-04-10 | 航天恒星科技有限公司 | Filing system and implement method of distributed configured massive data |
| CN102984275A (en) * | 2012-12-14 | 2013-03-20 | 北京奇虎科技有限公司 | Method and browser for web downloading |
| CN103780505A (en) * | 2014-01-27 | 2014-05-07 | 中国联合网络通信集团有限公司 | Business handling method and device |
Cited By (22)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107943854A (en) * | 2017-11-03 | 2018-04-20 | 深圳前海微众银行股份有限公司 | Reference service inquiry method, apparatus and computer-readable recording medium |
| CN107943854B (en) * | 2017-11-03 | 2022-01-11 | 深圳前海微众银行股份有限公司 | Credit investigation service query method, apparatus and computer readable storage medium |
| CN108241539A (en) * | 2018-01-03 | 2018-07-03 | 百度在线网络技术（北京）有限公司 | Interactive big data querying method, device, storage medium and terminal device based on distributed system |
| CN108241539B (en) * | 2018-01-03 | 2021-05-07 | 百度在线网络技术（北京）有限公司 | Interactive big data query method and device based on distributed system, storage medium and terminal equipment |
| CN108459825A (en) * | 2018-01-30 | 2018-08-28 | 深圳神州数码云科数据技术有限公司 | A kind of storage system service method of quality control and device |
| CN108459825B (en) * | 2018-01-30 | 2020-12-29 | 深圳神州数码云科数据技术有限公司 | Storage system service quality control method and device |
| CN108595254A (en) * | 2018-03-09 | 2018-09-28 | 北京永洪商智科技有限公司 | A kind of query scheduling method |
| CN111385255A (en) * | 2018-12-28 | 2020-07-07 | 北京金山云网络技术有限公司 | Asynchronous call implementation method and device, server and server cluster |
| CN111385255B (en) * | 2018-12-28 | 2022-05-06 | 北京金山云网络技术有限公司 | Asynchronous call implementation method and device, server and server cluster |
| CN109976910A (en) * | 2019-03-20 | 2019-07-05 | 跬云(上海)信息科技有限公司 | Querying method and device based on precomputation OLAP model |
| CN110489447B (en) * | 2019-07-16 | 2022-05-27 | 招联消费金融有限公司 | Data query method and device, computer equipment and storage medium |
| CN110489447A (en) * | 2019-07-16 | 2019-11-22 | 招联消费金融有限公司 | Data query method, apparatus, computer equipment and storage medium |
| CN110532227A (en) * | 2019-08-23 | 2019-12-03 | 广东浪潮大数据研究有限公司 | Operation quantity querying method and device |
| CN110825732A (en) * | 2019-09-20 | 2020-02-21 | 广州亚美信息科技有限公司 | Data query method and device, computer equipment and readable storage medium |
| CN110704489A (en) * | 2019-09-29 | 2020-01-17 | 苏州浪潮智能科技有限公司 | Database query method, device, equipment and computer storage medium |
| CN111078735A (en) * | 2019-11-04 | 2020-04-28 | 苏宁云计算有限公司 | Query request processing method and device |
| CN111078735B (en) * | 2019-11-04 | 2023-06-30 | 苏宁云计算有限公司 | Query request processing method and device |
| CN112003915A (en) * | 2020-08-14 | 2020-11-27 | 苏州浪潮智能科技有限公司 | File access method, device, equipment and readable medium |
| CN113111083A (en) * | 2021-03-31 | 2021-07-13 | 北京沃东天骏信息技术有限公司 | Method, device, equipment, storage medium and program product for data query |
| CN114466079A (en) * | 2022-02-09 | 2022-05-10 | 星环信息科技(上海)股份有限公司 | Request processing method, device, proxy server and storage medium |
| CN114466079B (en) * | 2022-02-09 | 2024-02-06 | 星环信息科技(上海)股份有限公司 | Request processing method, device, proxy server and storage medium |
| CN115051981A (en) * | 2022-05-12 | 2022-09-13 | 中国农业银行股份有限公司 | Zookeeper-based asynchronous downloading method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106294472B (en) | 2019-08-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106294472A (en) | The querying method of a kind of Hadoop data base HBase and device | |
| US20210218842A1 (en) | Method, device, server and storage medium of agent allocation | |
| US9015227B2 (en) | Distributed data processing system | |
| US20210311781A1 (en) | Method and system for scalable job processing | |
| US9357047B2 (en) | Method and system for a multitenancy telephone network | |
| US9363198B2 (en) | Load balancing in cloud-based networks | |
| US9112809B2 (en) | Method and apparatus for controlling utilization in a horizontally scaled software application | |
| CN107241281B (en) | Data processing method and device | |
| CN101951411A (en) | Cloud scheduling system and method and multistage cloud scheduling system | |
| CN107426274A (en) | The method and system of service application and monitoring analysis scheduling based on sequential | |
| CN106790636A (en) | A kind of equally loaded system and method for cloud computing server cluster | |
| CN109542608B (en) | Cloud simulation task scheduling method based on hybrid queuing network | |
| CN111083519A (en) | VR content distribution system and method based on cloud and edge computing | |
| CN107124453A (en) | Platform Interworking GateWay stacks the SiteServer LBS and video call method of deployment | |
| CN113938435A (en) | Data transmission method, data transmission device, electronic device, storage medium, and program product | |
| CN109815008A (en) | Hadoop cluster user resource monitoring method and system | |
| CN105491253A (en) | Resource distribution method and system for call center | |
| CN111913784B (en) | Task scheduling method and device, network element and storage medium | |
| CN109670691A (en) | Method, equipment and the customer service system distributed for customer service queue management and customer service | |
| CN112749008A (en) | Cloud resource distribution system based on OpenStack and construction method thereof | |
| CN110753083A (en) | Cloud service resource uniform distribution system for multiple service providers | |
| CN109062707A (en) | The method that is communicated between electronic device and its limiting process, storage medium | |
| CN116755905B (en) | Data interaction method, device, equipment and storage medium based on message queue | |
| CN116307567A (en) | Resource allocation method and related device for customer service scene based on multidimensional data analysis | |
| CN116301709A (en) | Method for adjusting access downstream flow rate under high concurrency scene and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |