CN110275882A - Method for writing data, system, device and readable storage medium storing program for executing based on HBASE - Google Patents
Method for writing data, system, device and readable storage medium storing program for executing based on HBASE Download PDFInfo
- Publication number
- CN110275882A CN110275882A CN201910379089.XA CN201910379089A CN110275882A CN 110275882 A CN110275882 A CN 110275882A CN 201910379089 A CN201910379089 A CN 201910379089A CN 110275882 A CN110275882 A CN 110275882A
- Authority
- CN
- China
- Prior art keywords
- task
- write
- task queue
- hbase
- queue
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/221—Column-oriented storage; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/485—Task life-cycle, e.g. stopping, restarting, resuming execution
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Debugging And Monitoring (AREA)
Abstract
The present invention relates to technical field of data processing, more particularly to when carrying out write operation to HBASE, the method for writing data based on HBASE, system, device and the electronic device of write operation mode are controlled, which is a PostgreSQL database distributed, towards column.The realization of the method for writing data based on HBASE needs to complete by task queue, and timer and counter are arranged in task queue, time threshold is arranged in timer, amount threshold is arranged in counter；The processing time by the way that task is written in timer control task queue controls the maximum quantity of the write-in task in task queue per treatment by counter.Method for writing data, system, device and the readable storage medium storing program for executing based on HBASE, pass through the time of each write operation of task queue unified planning and the quantity of write-in task, increase dormancy time, prevent write-in task in single write operation excessive, the pressure of data writing systems of the mitigation based on HBASE.
Description
Technical field
The present invention relates to technical field of data processing, and in particular to the data when carrying out write operation to HBASE are written
Method, system, device and readable storage medium storing program for executing.
Background technique
With the universal infiltration of data processing technique, the data manipulation based on HBASE is more and more.Same HBASE cluster
It is likely to have both the work of read-write simultaneously, and frequent high-volume data write-in can generate pressure to HBASE cluster, and influence
Reading data performance.
During the reading and writing data based on HBASE, when operating between cluster, possible multiple clusters are simultaneously to a certain HBASE
The write operation or a cluster for carrying out data carry out multiple data write operation to some HBASE simultaneously.
When the data write operation number sometime to certain HBASE is excessive, that is, when the number being written is excessive, will generate
The peak value of write operation has multiple write-in tasks to work at the same time at this time, this will increase the pressure of cluster, when cluster is difficult to load
When, it be easy to cause data write error or system crash.
Summary of the invention
In order to solve in the prior art based on HBASE data write-in it is excessive when, cluster pressure is big, and data is be easy to cause to write
The problem of entering mistake or system crash, the present invention provide a kind of data based on HBASE of peak value that can eliminate write operation
Wiring method, system, device and readable storage medium storing program for executing.
Based on the one hand, the present invention provides a kind of method for writing data based on HBASE, includes the following steps:
Firstly, establishing write-in task corresponding with said write operation；
Then, said write task is added in the task queue；Wherein, meter is provided in the task queue
When device and counter, be provided with time threshold in the timer, be provided with amount threshold in the counter；
The counter is used to count the quantity of the said write task in the task queue, when in the task queue
When the quantity of said write task reaches the amount threshold, stop adding said write task into the task queue；
The timer is used for Dose times, described when the time of timer metering reaching the time threshold
Timer is reset, and is simultaneously emitted by write-in task processing signal；
Finally, the said write task in the task queue according to said write task processing signal processing.
Preferably, while the timer is cleared processing, the timer restarts timing.
Preferably, after the timer is cleared processing, the said write task in the task queue is detected, when described
After the completion of said write task in task queue all executes, the timer starts Dose times again.
Further preferably, said write task after timer Dose times again, and in the task queue is all held
After the completion of row, continue to add write-in task into the task queue.
Preferably, the task queue is configured with multiple, and each task queue is respectively positioned in the same thread pool.
Further preferably, the task queue has 25 to 35；The time threshold is in 90ms between 110ms；The number
Threshold value is measured between 900 to 1100.
Preferably, after the task queue receives said write task processing signal, said write task is added
To in the task queue, the said write task in the task queue is then handled.
Based on the other hand, the present invention also provides a kind of data writing systems based on HBASE, should the number based on HBASE
It include write-in task creation unit, task queue configuration unit, timing unit and counting unit, the task team according to writing system
Column configuration unit, timing unit and counting unit are associated with the task queue respectively；
Said write task creation unit is used for: establishing write-in task corresponding with said write operation；
The task queue configuration unit is used for: each write-in task is added in the task queue；
The timing unit is used for: according to preset time threshold, being reached in the time of timing unit metering
Write-in task is issued when time threshold and handles signal, and said write task processing signal is used to start the thread in the thread pool
Handle the write-in task in the task queue；
The counting unit is used for: the quantity of the said write task in the task queue is counted, according to presetting
Amount threshold, control the quantity of the said write task in the task queue.
Based in another aspect, the present invention provides a kind of electronic device, which includes memory and processor, in institute
It states and is stored with computer program in memory, when computer program is executed by processor, realize such as the above-mentioned data based on HBASE
Wiring method.
Based on the other hand, the present invention provides a kind of computer readable storage medium, the computer readable storage medium
It is inside stored with computer program, when computer program is executed by processor, realizes such as the above-mentioned data write-in side based on HBASE
Method.
Method for writing data based on HBASE, system, device and readable storage medium storing program for executing provided by the invention and the prior art
It compares, has the advantages that
A kind of above-mentioned method for writing data based on HBASE, system, device and readable storage medium storing program for executing, carry out to HBASE
When write operation, task queue of the setting configured with counter and timer first is respectively write by the task queue unified planning
Enter operation.When each write operation of the task queue unified planning, it is continuously written into operation by the prior art by timer and becomes
The time more measured carries out write operation when reaching time threshold, makes to be continuously written into operation and is changed to interrupted write operation, often
There is the suspend mode waiting time after secondary write operation, mitigates the pressure of the data writing systems based on HBASE；Meanwhile passing through counting
Device limits the quantity of the write-in task of each write operation, prevents while write-in task mistake that the write operation that runs is formed
More situations occurs, and prevents write operation peak value, mitigates the pressure of the data writing systems based on HBASE.
Detailed description of the invention
By reference to the explanation of following below scheme attached drawing and the content of claims, and with to it is of the invention more comprehensively
Understand, other objects and results of the present invention will be more clearly understood and understood.In the accompanying drawings:
Fig. 1 is the flow diagram according to the method for writing data based on HBASE of the present embodiment.
Fig. 2 is to be reached after time threshold under practice condition according to the addition of the write-in task of the present embodiment in the time of metering
The flow diagram of method for writing data based on HBASE.
Fig. 3 is to be reached before time threshold under practice condition according to the addition of the write-in task of the present embodiment in the time of metering
The flow diagram of method for writing data based on HBASE.
Fig. 4 is data based on HBASE under Dose times situation again immediately after being reset according to the timer of the present embodiment
The flow diagram of wiring method.
Fig. 5 is to be based under Dose times situation again after the completion of task execution is written according to the timer of the present embodiment
The flow diagram of the method for writing data of HBASE.
Fig. 6 is the flow diagram of write-in task quantity in counter controls task queue.
Fig. 7 is the structural schematic diagram according to the electronic device of the present embodiment.
There is following label: 1 electronic device in attached drawing；2 processors；3 memories；4 computer programs.
Identical label indicates similar or corresponding feature or function in all the appended drawings.
Specific embodiment
In the following description, for purposes of illustration, it in order to provide the comprehensive understanding to one or more embodiments, explains
Many details are stated.It may be evident, however, that these embodiments can also be realized without these specific details.
In other examples, one or more embodiments for ease of description, well known structure and equipment are shown in block form an.
Hereinafter, specific embodiments of the present invention will be described in detail with reference to the accompanying drawings.
Embodiment 1
Fig. 1 provides the flow diagram of the method for writing data based on HBASE according to the present embodiment, and Fig. 2 to Fig. 5 is
According to the flow diagram of the method for writing data based on HBASE of the present embodiment under different situations, Fig. 6 appoints for counter controls
The flow diagram of write-in task quantity in business queue.
As shown in Figures 1 to 6, the present invention provides a kind of method for writing data based on HBASE.The number based on HBASE
It according to the realization of wiring method, needs to complete by task queue, in order to realize the control to write operation, in the task queue
It is provided with timer and counter, is provided with time threshold in the timer, is provided with amount threshold in the counter；On
After stating task queue configuration timer and counter, the data write operation based on HBASE is carried out.
The method for writing data based on HBASE includes the following steps:
S110: when carrying out write operation to HBASE, write-in task corresponding with the write operation is established；
S120: the write-in task established in step S110 is added to the above-mentioned task team configured with counter and timer
In column.
Specifically, Fig. 6, which is provided, is written number of tasks in counter controls task queue according to an embodiment of the invention
The process of amount.
As shown in fig. 6, the quantity of the write-in task contained in the task queue can be limited by above-mentioned counter；It is above-mentioned
Counter counts the quantity for the write-in task being added in task queue, and will be added in above-mentioned task queue the task that is written
Quantity is compared with amount threshold pre-set in counter；When the write-in task contained in task queue is learnt in comparison
When quantity is not up to the amount threshold being arranged in counter, continue to add untreated write-in task into the task queue；When
Compare when learning that the quantity of the write-in task contained in task queue reaches the amount threshold being arranged in several devices, enters step
Stop adding untreated write-in task into the task queue in S310, step S310.
S130: as shown in Figures 2 and 3, in the initial phase for carrying out each write operation, timer starts Dose times,
When the time of timer metering reaching pre-set time threshold in timer, which is reset, while the timing
Device also issues write-in task processing signal.The control unit of above-mentioned task queue or task queue receives write-in task processing
When signal, the write-in task in above-mentioned task queue is started to process.The control unit of the task queue can be in thread pool
Thread, task queue are located in thread pool.
S110 to S130 can complete the circulation of write-once operation through the above steps, when there are also write-in task is untreated
Or there are also write operation it is pending when, timer Dose times again, the step of repeating S110 to S130.
It further include some specific steps in above-mentioned steps S110 into S130, the step of which can be discussed further below；
Certainly, step as described below is only to illustrate to the present embodiment, does not mean that unique step；As described below
Step only illustrates in an illustrative manner, should not be understood as the restriction to the present embodiment；Other those skilled in the art are not
Through creative work, the thinkable alternative solution of institute also should belong to protection scope of the present invention.
Specifically, Fig. 2, which provides the addition of write-in task, reaches after time threshold base under practice condition in the time of metering
In the process of the method for writing data of HBASE.
As shown in Fig. 2, the time of timer metering reaches time threshold, above-mentioned task queue receives timer sending
After write-in task handles signal, S120 is entered step, it will be untreated and do not enter the write-in task of task queue and be added to task
In queue；S130 is subsequently entered step, the write-in task in task queue is handled.It can be seen from the above content that above-mentioned S120
The step of can timer issue write-in task processing signal after execute, since write-in task is added to spent by task queue
Time it is few, the process can the control unit of task queue or task queue receive write-in task processing signal after hold
Row.
Specifically, Fig. 3, which provides the addition of write-in task, reaches before time threshold base under practice condition in the time of metering
In the flow diagram of the method for writing data of HBASE.
As shown in figure 3, the time of timer metering reaches time threshold, above-mentioned task queue receives timer sending
After write-in task handles signal, it is directly entered step S130, handles configured good write-in task in task queue.
It is based under Dose times situation again immediately specifically, Fig. 4 is provided after being reset according to the timer of the present embodiment
The flow diagram of the method for writing data of HBASE.
As shown in figure 4, the time of timer metering reaches time threshold in step S210；In the metering of above-mentioned timer
When time reaches time threshold, S130 and step S220 are entered step, in step S220, when which starts to measure again
Between.Certainly, this Dose times refer to Dose times again after resetting again, carry out the write operation processing of subsequent cycle.?
Timer starts Dose times again while the time of timer metering reaches time threshold, can accurately control each read-write
The specific time interval of operation, the time interval are the time threshold being arranged in timer.
Specifically, Fig. 5 provides according to the timer of the present embodiment after the completion of task execution is written Dose times again
The flow diagram of method for writing data under situation based on HBASE.
As shown in figure 5, Dose times can also be arranged in the following manner again: in step S210, timer metering when
Between reach time threshold；When the time of above-mentioned timer metering reaches time threshold, S130 is entered step, then judges S130's
Whether progress, i.e., the write-in task in task queue are disposed；When judgement learns that the write-in task in task queue has been handled
Bi Hou enters step S220, and in step S220, which starts Dose times again；When judgement is learnt in task queue
Write-in task it is also untreated after, return to step S130, continue with the write-in task in task queue.In above-mentioned task
Dose times again are carried out again after the completion of all write-in tasks processing in queue, can be accurately controlled dormancy time, should be stopped
The dormancy time refers to that last time write-in task all executes the time interval for completing to start up to next write operation.
, can be after above-mentioned timer again Dose times in the specific embodiment of the present invention, and above-mentioned
After the completion of all write-in tasks in queue of being engaged in all execute, untreated write-in task is added in task queue.On
Description of contents is stated, after the completion of certain write operation wherein executes, in next write operation, above-mentioned two condition should met
Afterwards, write-in task could be added into task queue, above-mentioned two condition is that necessity of write-in task is added into task queue
Condition；When, there are also when the write-in task handled, write-in that is other untreated and not entering task queue is appointed in task queue
Business does not allow to be added to task queue.
In addition, in order to accelerate the efficiency of write operation, above-mentioned task queue should be configured with multiple, each equal position of task queue
In in the same thread pool；When needing to handle write-in task, pass through respectively writing in the thread process task queue in thread pool
Enter task.
The setting of task queue can be according to the service ability and practical business demand flexible setting of system, for example, at this
In one specific embodiment of invention, above-mentioned task queue has 25 to 35；The time threshold being arranged in above-mentioned timer exists
90ms is between 110ms；The amount threshold being arranged in above-mentioned timer is between 900 to 1100.Above-mentioned task queue quantity
Setting can realize that the task queue quantity is most by setting task queue quantity maximum value and task queue quantity minimum value
Big value and task queue quantity minimum value select between 25 to 35, and task queue quantity maximum value should be not less than task queue number
Measure minimum value.Above-mentioned time threshold and amount threshold are arranged in timer and counter respectively.
In another specific embodiment of the invention, above-mentioned task queue has 30 (task queue quantity maximum values
It is that 30), the time threshold being arranged in above-mentioned timer is 100ms with task queue quantity minimum value；It is set in above-mentioned timer
The amount threshold set is 1000, can be immediately arrived at through the above, each write operation can handle 30 × 1000=
30000 write-in tasks.
While the time of each clearing or timer metering reaching time threshold, when timer starts to measure again
Between when, it is known that it is per second to realize 10 read-write operations, it is per second to execute 30000 × 10=300000 write-in task.When every
After the completion of write-in task whole after the time of secondary clearing or timer metering reaches time threshold, and in task queue executes
When timer starts Dose times again, it is known that each read-write operation is other than the time threshold of 100ms is as dormancy time, often
The execution of secondary write operation also needs to expend the time；The execution for estimating each write operation needs 100ms, then each write operation and
Dormancy time amounts to time-consuming 200ms, it is known that the read-write operation per second for realizing 5 times, it is per second to can handle 30000 × 5=150000
A write-in task.
Embodiment 2
The present embodiment provides a kind of method for writing data based on HBASE, and it is real for being somebody's turn to do the method for writing data based on HBASE
Apply the specific example of example 1.In concrete operations, the time threshold being arranged in above-mentioned timer is 100ms, is set in above-mentioned timer
The amount threshold set is 1000.
Timer Dose times contain 900 write-ins when the time of timer metering reaching 100ms in task queue
Task, and at this time without other untreated and do not enter the write-in task of task queue.Then timer issues the processing of write-in task
When the control unit of signal, above-mentioned task queue or task queue receives write-in task processing signal, start to process above-mentioned
900 write-in tasks in task queue, the control unit of the task queue can be the thread in thread pool.Work as task queue
In 900 write-in tasks be all finished after, Dose times, task queue receive other untreated timer again again
And the write-in task of task queue is not entered.Certainly, which can also be clear while issuing write-in task processing signal
Zero, and Dose times again；The process that above-mentioned write-in task is added to task queue can also appoint receiving write-in next time
It is carried out after business processing signal.
Embodiment 3
The present embodiment provides a kind of method for writing data based on HBASE, and it is real for being somebody's turn to do the method for writing data based on HBASE
Apply the specific example of example 1.In concrete operations, the time threshold being arranged in above-mentioned timer is 100ms, is set in above-mentioned timer
The amount threshold set is 1000.
Timer Dose times contain 1000 write-ins when the time of timer metering reaching 100ms in task queue
Task, and also several are untreated and do not enter the write-in task of task queue, counter controls task queue at this time is forbidden
This is untreated and does not enter the write-in task of task queue and is added to task queue.Then timer issues write-in task processing letter
Number, when the control unit of above-mentioned task queue or task queue receives write-in task processing signal, start to process above-mentioned
1000 write-in tasks being engaged in queue, the control unit of the task queue can be the thread in thread pool.Work as task queue
In 1000 write-in tasks be all finished after, timer Dose times again, task queue receives again other not to be located
Manage and do not enter the write-in task of task queue.Certainly, which can also be while issuing write-in task processing signal
It resets, and Dose times again；The process that above-mentioned write-in task is added to task queue can appoint receiving write-in next time
It is carried out after business processing signal.
Embodiment 4
The present embodiment provides a kind of data writing systems based on HBASE, in order to realize above-described embodiment 1 to embodiment 3
The method for writing data based on HBASE, being somebody's turn to do the data writing systems based on HBASE includes task queue, the task team
Column are used for each write operation of unified planning, the number for the write-in task that the time and each write operation for controlling write operation execute
Amount.
In order to control the quantity for the write-in task that write operation interval time and each write operation execute, this is based on
The data writing systems of HBASE further include write-in task creation unit, task queue configuration unit, timing unit and count single
Member, above-mentioned task queue configuration unit, timing unit and counting unit are associated with above-mentioned task queue respectively；
Above-mentioned write-in task creation unit is used for: when carrying out write operation to HBASE, being established opposite with the write operation
The write-in task answered；The write-in task creation unit is the process that pending write operation is converted to write-in task.
Above-mentioned task queue configuration unit is used for: each write-in task is added in above-mentioned task queue；The task queue
Configuration unit is in order to be associated with write-in task and task queue, convenient for controlling the execution of each write-in task by task queue.
Above-mentioned timing unit is used for: according to preset time threshold, when reaching the time of timing unit metering
Between threshold value when issue write-in task and handle signal, it is above-mentioned that write-in task processing signal is used for the thread process that starts in thread pool
Write-in task in task queue；The timing unit is associated with task queue, control processing task queue in be written task when
Between, after the time zero of the timing unit can be last write-in task execution, it is also possible to last timer
When the time of amount reaches time threshold.
Above-mentioned counting unit is used for: the quantity of the write-in task in above-mentioned task queue is counted, according to preset number
Measure threshold value, the quantity of the write-in task in control task queue.The counting unit should be associated with task queue, appoints for controlling
It is engaged in that the quantity of task is written in queue.Certainly, there are also just when the quantity of the task of write-in reaches preset threshold or in task queue
In the write-in task of processing, the quantity of task is written in the task queue even below amount threshold, which also answers
It is configured to that write-in task can not be added.
Specifically, in order to increase the treatment effeciency of write-in task, above-mentioned task queue is provided with multiple, multiple task
Queue is respectively positioned in the same thread pool, passes through the write-in task in the thread process task queue in thread pool.
Embodiment 5
Fig. 4 provides the structural schematic diagram of electronic device, as shown in figure 4, electronic device provided in this embodiment includes storage
Device 3 and processor 2 are stored with computer program 4 in the memory 3, when computer program 4 is executed by processor 2, realize such as
The above-mentioned method for writing data based on HBASE.
Above-mentioned electronic device 1 can be the calculating such as desktop PC, notebook, palm PC and cloud server and set
It is standby.Electronic device 1 may include, but be not limited only to, processor 2 and memory 3.It will be understood by those skilled in the art that Fig. 4 is only
It is the example of electronic device 1, does not constitute the restriction to electronic device 1, may include components more more or fewer than diagram, or
Person combines certain components or different components, such as electronic device 1 can also be set including input-output equipment, network insertion
Standby, bus etc..
Alleged processor 2 can be central processing unit (Central Processing Unit, CPU), can also be it
His general processor, digital signal processor (Digital Signal Processor, DSP), specific integrated circuit
(Application Specific Integrated Circuit, ASIC), ready-made programmable gate array (Field-
Programmable Gate Array, FPGA) either other programmable logic device, discrete gate or transistor logic,
Discrete hardware components etc..General processor can be controller, microcontroller, microprocessor or the processor and be also possible to
Any conventional processor etc..
Memory 3 can be the internal storage unit of electronic device 1, such as the hard disk or memory of electronic device 1.Memory
3 are also possible to the plug-in type hard disk being equipped on the External memory equipment of electronic device 1, such as electronic device 1, intelligent memory card
(Smart Media Card, SMC), secure digital (Secure Digital, SD) card, flash card (Flash Card), more matchmakers
Body card, card-type memory, magnetic storage, disk and CD etc..Further, memory can also both include terminal device
Internal storage unit also includes External memory equipment.Memory is for storing needed for computer program 4 and electronic equipment it
His program and data.Memory can be also used for temporarily storing the data that has exported or will export.
Embodiment 6
The present embodiment provides a kind of computer readable storage medium, computer is stored in the computer readable storage medium
Program when computer program is executed by processor, realizes such as the above-mentioned method for writing data based on HBASE.
It is apparent to those skilled in the art that for convenience of description and succinctly, only with above-mentioned each function
Can unit, module division progress for example, in practical application, can according to need and by above-mentioned function distribution by different
Functional unit, module are completed, i.e., the internal structure of device are divided into different functional unit or module, to complete above description
All or part of function.Each functional unit in embodiment, module can integrate in one processing unit, be also possible to
Each unit physically exists alone, and can also be integrated in one unit with two or more units, above-mentioned integrated unit
Both it can take the form of hardware realization, can also realize in the form of software functional units.In addition, each functional unit, mould
The specific name of block is also only for convenience of distinguishing each other, the protection scope being not intended to limit this application.It is single in above system
Member, the specific work process of module, can refer to corresponding processes in the foregoing method embodiment, details are not described herein.
In the above-described embodiments, it all emphasizes particularly on different fields to the description of each embodiment, is not described in detail or remembers in some embodiment
The part of load may refer to the associated description of other embodiments.
Those of ordinary skill in the art may be aware that list described in conjunction with the examples disclosed in the embodiments of the present disclosure
Member and algorithm steps can be realized with the combination of electronic hardware or computer software and electronic hardware.These functions are actually
It is implemented in hardware or software, the specific application and design constraint depending on technical solution.Professional technician
Each specific application can be used different methods to achieve the described function, but this realization is it is not considered that exceed
The scope of the present invention.
In embodiment provided by the present invention, it should be understood that disclosed device and method can pass through others
Mode is realized.For example, the apparatus embodiments described above are merely exemplary, for example, the division of the module or unit,
Only a kind of logical function partition, there may be another division manner in actual implementation, such as multiple units or components can be with
In conjunction with or be desirably integrated into another system, or some features can be ignored or not executed.Another point, it is shown or discussed
Mutual coupling or direct-coupling or communication connection can be through some interfaces, the INDIRECT COUPLING of device or unit or
Communication connection can be electrical property, mechanical or other forms.
Unit may or may not be physically separated as illustrated by the separation member, shown as a unit
Component may or may not be physical unit, it can and it is in one place, or may be distributed over multiple networks
On unit.It can some or all of the units may be selected to achieve the purpose of the solution of this embodiment according to the actual needs.
It, can also be in addition, the functional units in various embodiments of the present invention may be integrated into one processing unit
It is that each unit physically exists alone, can also be integrated in one unit with two or more units.Above-mentioned integrated list
Member both can take the form of hardware realization, can also realize in the form of software functional units.
If the integrated module/unit be realized in the form of SFU software functional unit and as independent product sale or
In use, can store in a computer readable storage medium.Based on this understanding, the present invention realizes above-mentioned implementation
All or part of the process in example method, can also instruct relevant hardware to complete, the meter by computer program
Calculation machine program can be stored in a computer readable storage medium, the computer program when being executed by processor, it can be achieved that on
The step of stating each embodiment of the method.Wherein, the computer program includes computer program code, the computer program generation
Code can be source code form, object identification code form, executable file or certain intermediate forms etc..The computer-readable medium
It may include: any entity or device, recording medium, USB flash disk, mobile hard disk, magnetic that can carry the computer program code
Dish, CD, computer storage, read-only memory (ROM, Read-Only Memory), random access memory (RAM,
Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium etc..It should be noted that described
The content that computer-readable medium includes can carry out increasing appropriate according to the requirement made laws in jurisdiction with patent practice
Subtract, such as in certain jurisdictions, according to legislation and patent practice, computer-readable medium do not include be electric carrier signal and
Telecommunication signal.
A kind of above-mentioned method for writing data based on HBASE, system, device and readable storage medium storing program for executing, carry out to HBASE
When write operation, task queue of the setting configured with counter and timer first is respectively write by the task queue unified planning
Enter operation.When each write operation of the task queue unified planning, it is continuously written into operation by the prior art by timer and becomes
The time more measured carries out write operation when reaching time threshold, makes to be continuously written into operation and is changed to interrupted write operation, often
There is the suspend mode waiting time after secondary write operation, mitigates the pressure of the data writing systems based on HBASE；Meanwhile passing through counting
Device limits the quantity of the write-in task of each write operation, prevents while write-in task mistake that the write operation that runs is formed
More situations occurs, and prevents write operation peak value, mitigates the pressure of the data writing systems based on HBASE.
It describes the method for writing data according to the present invention based on HBASE in an illustrative manner above with reference to attached drawing, be
System, device and readable storage medium storing program for executing.It will be understood by those skilled in the art, however, that for aforementioned present invention proposed based on
Method for writing data, system, device and the readable storage medium storing program for executing of HBASE, can also be on the basis of not departing from the content of present invention
Make various improvement.Therefore, protection scope of the present invention should be determined by the content of appended claims.
Claims (10)
1. a kind of method for writing data based on HBASE, which comprises the steps of:
Establish write-in task corresponding with write operation；
Said write task is added in task queue；Wherein,
It is provided with timer and counter in the task queue, time threshold is provided in the timer, described
Amount threshold is provided in counter；
The counter is used to count the quantity of the said write task in the task queue, described in the task queue
When the quantity of write-in task reaches the amount threshold, stop adding said write task into the task queue；
The timer is used for Dose times, when the time of timer metering reaching the time threshold, the timing
Device is reset, and is simultaneously emitted by write-in task processing signal；
According to the said write task in task queue described in said write task processing signal processing.
2. as described in claim 1 based on the method for writing data of HBASE, which is characterized in that the timer is cleared place
While reason, start Dose times again.
3. as described in claim 1 based on the method for writing data of HBASE, which is characterized in that the timer is cleared place
After reason, the said write task in the task queue is detected, when the said write task in the task queue all executes
After the completion, the timer starts Dose times again.
4. as claimed in claim 2 or claim 3 based on the method for writing data of HBASE, which is characterized in that when the timer again
Dose times, and after the completion of the said write task whole execution in the task queue, continue to add into the task queue
Add write-in task.
5. as described in claim 1 based on the method for writing data of HBASE, which is characterized in that the task queue is configured with
Multiple, each task queue is respectively positioned in the same thread pool.
6. as claimed in claim 5 based on the method for writing data of HBASE, which is characterized in that the task queue have 25 to
35；The time threshold is in 90ms between 110ms；The amount threshold is between 900 to 1100.
7. as described in claim 1 based on the method for writing data of HBASE, which is characterized in that received in the task queue
After handling signal to said write task, said write task is added in the task queue, the task is then handled
Said write task in queue.
8. a kind of data writing systems based on HBASE, which is characterized in that match including write-in task creation unit, task queue
Set unit, timing unit and counting unit, the task queue configuration unit, timing unit and counting unit respectively with described
Business queue is associated；
Said write task creation unit is used for: establishing write-in task corresponding with said write operation；
The task queue configuration unit is used for: each write-in task is added in the task queue；
The timing unit is used for: according to preset time threshold, reaching the time in the time of timing unit metering
Write-in task is issued when threshold value and handles signal, and said write task processing signal is used to start the thread process in the thread pool
Write-in task in the task queue；
The counting unit is used for: the quantity of the said write task in the task queue is counted, according to preset number
Threshold value is measured, the quantity of the said write task in the task queue is controlled.
9. a kind of electronic device, which includes memory and processor, which is characterized in that is stored in the memory
Computer program when computer program is executed by processor, is realized as claimed in any one of claims 1 to 7 based on HBASE
Method for writing data.
10. a kind of computer readable storage medium, which is characterized in that the computer-readable recording medium storage has computer journey
Sequence when computer program is executed by processor, realizes that the data as claimed in any one of claims 1 to 7 based on HBASE are write
Enter method.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910379089.XA CN110275882A (en) | 2019-05-08 | 2019-05-08 | Method for writing data, system, device and readable storage medium storing program for executing based on HBASE |
| PCT/CN2019/117895 WO2020224218A1 (en) | 2019-05-08 | 2019-11-13 | Hbase-based data writing method, system, device, and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910379089.XA CN110275882A (en) | 2019-05-08 | 2019-05-08 | Method for writing data, system, device and readable storage medium storing program for executing based on HBASE |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110275882A true CN110275882A (en) | 2019-09-24 |
Family
ID=67959749
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910379089.XA Pending CN110275882A (en) | 2019-05-08 | 2019-05-08 | Method for writing data, system, device and readable storage medium storing program for executing based on HBASE |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN110275882A (en) |
| WO (1) | WO2020224218A1 (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2020224218A1 (en) * | 2019-05-08 | 2020-11-12 | 平安科技（深圳）有限公司 | Hbase-based data writing method, system, device, and readable storage medium |
| CN113535616A (en) * | 2020-04-13 | 2021-10-22 | 慧荣科技股份有限公司 | Computer readable storage medium, method and device for controlling access of flash memory device |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100217905A1 (en) * | 2009-02-24 | 2010-08-26 | International Business Machines Corporation | Synchronization Optimized Queuing System |
| CN107078959A (en) * | 2014-12-27 | 2017-08-18 | 英特尔公司 | The portfolio guiding mitigated in distributed non-nuclear structure is poorly efficient |
| CN107229673A (en) * | 2017-04-20 | 2017-10-03 | 努比亚技术有限公司 | Method for writing data, Hbase terminals and the storage medium of Hbase databases |
| CN108038171A (en) * | 2017-12-07 | 2018-05-15 | 杭州电魂网络科技股份有限公司 | Method for writing data, device and data server |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107688634A (en) * | 2017-08-22 | 2018-02-13 | 阿里巴巴集团控股有限公司 | Method for writing data and device, electronic equipment |
| CN108829504A (en) * | 2018-06-28 | 2018-11-16 | 泰康保险集团股份有限公司 | A kind of method for scheduling task, device, medium and electronic equipment |
| CN110275882A (en) * | 2019-05-08 | 2019-09-24 | 平安科技（深圳）有限公司 | Method for writing data, system, device and readable storage medium storing program for executing based on HBASE |
-
2019
- 2019-05-08 CN CN201910379089.XA patent/CN110275882A/en active Pending
- 2019-11-13 WO PCT/CN2019/117895 patent/WO2020224218A1/en active Application Filing
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100217905A1 (en) * | 2009-02-24 | 2010-08-26 | International Business Machines Corporation | Synchronization Optimized Queuing System |
| CN107078959A (en) * | 2014-12-27 | 2017-08-18 | 英特尔公司 | The portfolio guiding mitigated in distributed non-nuclear structure is poorly efficient |
| CN107229673A (en) * | 2017-04-20 | 2017-10-03 | 努比亚技术有限公司 | Method for writing data, Hbase terminals and the storage medium of Hbase databases |
| CN108038171A (en) * | 2017-12-07 | 2018-05-15 | 杭州电魂网络科技股份有限公司 | Method for writing data, device and data server |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2020224218A1 (en) * | 2019-05-08 | 2020-11-12 | 平安科技（深圳）有限公司 | Hbase-based data writing method, system, device, and readable storage medium |
| CN113535616A (en) * | 2020-04-13 | 2021-10-22 | 慧荣科技股份有限公司 | Computer readable storage medium, method and device for controlling access of flash memory device |
| CN113535616B (en) * | 2020-04-13 | 2023-09-22 | 慧荣科技股份有限公司 | Computer readable storage medium, method and device for controlling access of flash memory device |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2020224218A1 (en) | 2020-11-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109647719B (en) | Method and device for sorting goods | |
| US11325780B2 (en) | Method and device for sorting cargo | |
| CN108595157B (en) | Block chain data processing method, device, equipment and storage medium | |
| CN102147746B (en) | Dynamic thread pool management system and method | |
| CN108683604A (en) | concurrent access control method, terminal device and medium | |
| CN109298951A (en) | The execution method and terminal device of Celery task | |
| CN108052396A (en) | A kind of resource allocation methods and system | |
| CN108334408B (en) | Code execution method and device, terminal equipment and computer readable storage medium | |
| CN108388507A (en) | Batch script testing method and device | |
| CN110275882A (en) | Method for writing data, system, device and readable storage medium storing program for executing based on HBASE | |
| CN109343972A (en) | Task processing method and terminal device | |
| CN107741882A (en) | The method and device and electronic equipment of distribution task | |
| CN111443999A (en) | Data parallel processing method, actuator, computer device and storage medium | |
| CN110262878A (en) | Timed task processing method, device, equipment and computer readable storage medium | |
| CN112101831A (en) | Goods delivery method, device, medium and electronic equipment | |
| CN109634853A (en) | Data test method, system, electronic device and computer readable storage medium | |
| CN109240802A (en) | request processing method and device | |
| CN108038126A (en) | Method, apparatus, terminal device and storage medium derived from a kind of data | |
| CN109255671A (en) | Processing method, device and the computer readable storage medium of order request | |
| CN110347735A (en) | The data forwarding of configurationization mends processing method, device and readable storage medium storing program for executing extremely | |
| CN107195144B (en) | The method, apparatus and computer readable storage medium of managing payment terminal hardware module | |
| CN104917720B (en) | The method and device of password resetting | |
| CN109857579A (en) | A kind of data processing method and relevant apparatus | |
| US10079777B2 (en) | Apparatus, systems, and methods of allocating heterogeneous resources | |
| CN115718741A (en) | Automatic cleaning method and device for health data, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |