CN106021284A - Data query method, data monitoring method and device - Google Patents
Data query method, data monitoring method and device Download PDFInfo
- Publication number
- CN106021284A CN106021284A CN201610282721.5A CN201610282721A CN106021284A CN 106021284 A CN106021284 A CN 106021284A CN 201610282721 A CN201610282721 A CN 201610282721A CN 106021284 A CN106021284 A CN 106021284A
- Authority
- CN
- China
- Prior art keywords
- data
- memory node
- type
- target
- volume
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention provides a data query method, a data monitoring method and a device. The data monitoring method comprises the steps that a data query request is sent to each storage node, wherein the data query request is used to inquire a total data size of a target data type; data query responses sent by each storage node are received, and each data query response contains a data size of the target data type stored in the corresponding storage node, wherein the data size of the target data type stored in each storage node is obtained by the corresponding storage node according to a data storage record of a target queue corresponding to the target data type; and the total data size of the target data type is obtained according to the data sizes of the target data type stored in each storage node, so that the data size of each type of data can be known in a real time, and a basis can be provided for other subsequent processing courses.
Description
Technical field
The present embodiments relate to data processing field, particularly relate to a kind of data query method, data prison
Survey method and device.
Background technology
Assume following application scenarios: the multiple different service of certain Internet service platform maintenance, including
The data producing difference service carry out storage system maintenance, access maintenance etc..It is to say, every kind of service
Corresponding a kind of data type, every kind of service data of generation in running are input to this service platform
In, this service platform carry out data storage, it addition, certain user there may be uses this stored
The demand of a little data, user accesses data can also be conducted interviews control by this service platform.
At present, a kind of carry out data storage mode be: estimate based on to issuable data volume,
Configure a number of memory node.During carrying out data storage, examine safely etc. based on data
Consider, data are carried out distributed redundancy backup storage and processes.Storing process is in simple terms: will successively
A certain amount of data being input to this service platform carry out burst process according to certain rule, by each
Data fragmentation replicates certain number, thus this data fragmentation and its duplicate are respectively stored into difference
Memory node in.
In certain actual application, the multiple different service due to above-mentioned platform maintenance, be no matter
The access user of the data of access platform storage, or the guardian of platform, may exist and want
Solve the data input condition data storage condition in other words of every kind of service, in order to know that what each serviced makes
By situation, and provide foundation for memory node being carried out load balancing, dilatation etc..Therefore, a kind of
The mode carrying out data monitoring urgently provides.
Summary of the invention
The embodiment of the present invention provides a kind of data monitoring method, device and memory node, in order to realize respectively
The real-time monitoring of the storage condition of categorical data.
The embodiment of the present invention provides a kind of data query method, including:
Receiving data inquiry request, described data inquiry request is for inquiring about the data volume of target data type；
Data storage record according to object queue corresponding to described target data type, determines described target
The data volume of data type；
Transmission data query responds, and comprises the data of described target data type in the response of described data query
Amount.
The embodiment of the present invention provides a kind of data monitoring method, including:
Sending data inquiry request to each memory node, described data inquiry request is used for inquiring about target data
The total amount of data of type；
Receiving the data query response that each memory node sends, it is right that the response of each described data query includes
Answer the data volume of the described target data type that memory node is corresponding；
According to the data volume of the described target data type of storage in each memory node, it is thus achieved that described target
The total amount of data of data type.
The embodiment of the present invention provides a kind of data query arrangement, including:
Receiver module, is used for receiving data inquiry request, and described data inquiry request is used for inquiring about number of targets
Data volume according to type；
Determine module, for the data storage record of the object queue corresponding according to described target data type,
Determine the data volume of described target data type；
Sending module, is used for sending data query response, comprises described target in the response of described data query
The data volume of data type.
The embodiment of the present invention provides a kind of data monitoring device, including:
Sending module, for sending data inquiry request to each memory node, described data inquiry request is used
Total amount of data in inquiry target data type；
Receiver module, for receiving the data query response that each memory node sends, each described data are looked into
Ask the data volume of the described target data type that response includes that corresponding memory node is corresponding；
First determines module, for according to the described target data type of storage in each memory node
Data volume, it is thus achieved that the total amount of data of described target data type.
Data query method, data monitoring method and the device that the embodiment of the present invention provides, when needs are monitored
When knowing the storage data volume of data of Various types of data type, it is used for inquiring about by sending to each memory node
The inquiry request of the data volume of certain or some data types, owing to data are being carried out by each memory node
In storing process, there is the queue of a correspondence in order to data cached for every kind of data type, from queue
During data are stored to the memory space of memory node, form the data storage record of this queue, because of
This, each memory node can data based on the queue corresponding with the data type of required inquiry storage note
Record, it is thus achieved that the data volume of this data type, eventually through the feedback result of accumulative each memory node, it is possible to
Obtain the total amount of data of this certain or some data type, to realize the reality of the data volume to all types of data
Time understand, for other processing procedures follow-up provide foundation.
Accompanying drawing explanation
In order to be illustrated more clearly that the embodiment of the present invention or technical scheme of the prior art, below will be to reality
Execute the required accompanying drawing used in example or description of the prior art to be briefly described, it should be apparent that under,
Accompanying drawing during face describes is some embodiments of the present invention, for those of ordinary skill in the art,
On the premise of not paying creative work, it is also possible to obtain other accompanying drawing according to these accompanying drawings.
Fig. 1 is the flow chart of data query embodiment of the method one of the present invention；
Fig. 2 is the flow chart of data monitoring method embodiment two of the present invention；
Fig. 3 is the flow chart of data monitoring method embodiment three of the present invention；
Fig. 4 is the flow chart of data monitoring method embodiment four of the present invention；
Fig. 5 is the structural representation of data query arrangement embodiment of the present invention；
Fig. 6 is the structural representation of data monitoring device embodiment one of the present invention；
Fig. 7 is the structural representation of data monitoring device embodiment two of the present invention；
Fig. 8 is the structural representation of data monitoring device embodiment three of the present invention.
Detailed description of the invention
For making the purpose of the embodiment of the present invention, technical scheme and advantage clearer, below in conjunction with this
Accompanying drawing in bright embodiment, is clearly and completely described the technical scheme in the embodiment of the present invention,
Obviously, described embodiment is a part of embodiment of the present invention rather than whole embodiments.Based on
Embodiment in the present invention, those of ordinary skill in the art are obtained under not making creative work premise
The every other embodiment obtained, broadly falls into the scope of protection of the invention.
Before specifically introducing the data monitoring method that the present invention provides, first permissible to this data monitoring method
Be applicable to application scenarios simply introduce: be configured with certain cluster multiple memory node (realize
For storage device or server) it is used for carrying out data storage.Wherein it is desired to carry out the data class stored
Type can be multiple different data type, such as the multiple different software service of certain enterprise offer, every kind
The data type of the data corresponding to software service is different, and every kind of software service is produced in running
Raw data all store in certain memory node in above-mentioned cluster, and concrete storing process is not belonging to this
The emphasis of invention, does not repeats.Along with being constantly stored in of all types of data, want to know when user has
During the situation of the data volume that each type of data are current, the data monitoring side that the present invention provides can be used
Method realizes.
The meaning of this data monitoring is partly in order to meet the user providing certain service, or safeguards above-mentioned
The timely understanding of the data volume situation that the user of cluster is corresponding to certain service, on the other hand, also uses for cluster
Family judges whether memory node be enough to support the storage of mass data, reads a kind of decision procedure of offer, with
For memory node load balancing, expand, access etc. provides foundation.
Fig. 1 is the flow chart of data query embodiment of the method one of the present invention, and the data that this embodiment provides are looked into
Inquiry method can be performed by data query arrangement, and this data query arrangement can be arranged on arbitrary memory node
In.As it is shown in figure 1, comprise the steps:
Step 101, reception data inquiry request, this data inquiry request is used for inquiring about target data type
Data volume.
Step 102, data storage record according to object queue corresponding to target data type, determine mesh
The data volume of mark data type.
Step 103, transmission data query response, comprise the number of target data type in data query response
According to amount.
In the present embodiment, when user wants the data volume inquiring about certain categorical data, above-mentioned number can be triggered
According to inquiry request to the transmission of each memory node.When implementing, can at the terminal unit of user or
Each memory node arranges client application, after this client application can be configured as being activated,
Showing the mark of each memory node and the mark of each data type on interface, user can pass through
The operation chosen selects to need the target data type of inquiry.Under default situations, all of depositing can be set
Storage node is the most selected.
It is understood that the number of the above-mentioned target data type of user's selection can be one, it is possible to
To be multiple, when being multiple, it is meant that need inquiry is the data volume of every kind of target data type.
After each memory node in whole memory nodes receives above-mentioned data inquiry request, carry out this
The query processing process of the data volume of the target type data of ground storage:
Firstly the need of explanation a bit: safeguarding there is multiple queue in each memory node, each queue is used
One-to-one relationship is there is, the most namely in the data of a certain type of caching, i.e. queue and data type
Say, in a memory node, the data of number of different types can be stored.Come with any one memory node
Say, after it receives certain data to be stored, its storing process be briefly described into: according to these data
Data type determine the queue that these data are corresponding；These data are stored in this correspondence queue；Based on data
Burst mechanism, when the data fragmentation comprising these data meets burst division condition, needs to store this data
Burst, and, the safety problem caused when breaking down in order to avoid a data fragmentation is permissible
This data fragmentation is replicated several parts, thus this data fragmentation and each duplicate are stored to different storage joints
In point.Wherein, the division condition of data fragmentation such as reaches some, or number for data volume number
Certain threshold size is reached according to amount size.Further, during carrying out above-mentioned storage, this queue is generated
Corresponding data storage record.
Secondly it is worth explanation on the other hand: for certain memory node, can be considered as wherein storing
Two kinds of data, one is main of data, another kind is data duplicates, corresponding to above-mentioned data fragmentation
For concept, one is main of data fragmentation, and another kind is data fragmentation duplicate, certainly, and same data
Main of burst and duplicate are typically not stored in same memory node.Although same memory node stores
Main of data and data duplicate two kinds of data, but the multiple queues safeguarded for this memory node come
Saying, in queue, storage is main of data.
Thus, in the present embodiment, for any one memory node, its target data type obtained
Data volume include main of the data of its locally stored target data type and be stored in other storage joints
Main with these data corresponding data duplicate in point.
It is understood that the concept of above-mentioned main and duplicate is to distinguish two kinds of data and self-defining
Address, meets in the range of other concepts of this meaning are also contained in the present embodiment.
Necessarily associate owing to query script has with storing process, therefore, based on above-mentioned storing process,
When arbitrary memory node inquires about the data volume of above-mentioned target data type, first according to this target data class
Type determines the object queue of correspondence, and then stores record according to the data of object queue, determines target data
The data volume of type.
Optionally, object queue is for storing main of data corresponding to described target data type.Target team
The data storage record of row can record the data fragmentation mark of data fragmentation and answering of data fragmentation
This number, i.e. in carrying out above-mentioned storing process, whenever meeting data fragmentation and dividing condition, generates one
Individual data fragmentation, the data fragmentation mark that distribution is corresponding.
Thus, the data storage record of the above-mentioned object queue corresponding according to target data type, determine mesh
Mark data type data volume, can be accomplished in that according to data fragmentation mark quantity,
Data volume that each data fragmentation is corresponding and duplicate number corresponding to each data fragmentation, determine number of targets
Total amount of data according to type.
In the present embodiment, data volume can be characterized by data amount check or data strip number.
For arbitrary memory node, when it receives data inquiry request, wherein may deposit
Contain the data of a certain amount of target data type.General, this memory node is by a data fragmentation
Main stores in this locality, and its multiple duplicates are deposited in other memory nodes.
Therefore, store according to the data of object queue the data fragmentation mark of record in record number and
The duplicate number that each data fragmentation is corresponding, it may be determined that the target data type that this memory node is corresponding
Data fragmentation number, i.e. assumes that data fragmentation mark quantity is 100, the duplicate that each data fragmentation is corresponding
Quantity is 2, then the data fragmentation of corresponding 300 target data type of this memory node (includes 100
Main and 200 data fragmentation duplicates of individual data fragmentation).Then, corresponding according to each data fragmentation
Data volume, the data amount check comprised in the most each data fragmentation, obtain the total data that this memory node is corresponding
Amount, i.e. total data number.
What deserves to be explained is, the data volume of the target data type that memory node is corresponding, be not limited to be
The data volume of this target data type of storage in this memory node, can be storage in this memory node
In main corresponding data volume of the data fragmentation of this target data type and other memory nodes, storage should
The sum of the data volume corresponding to data fragmentation duplicate of target data type.
It addition, optional, the data storage record of above-mentioned object queue can also record tail of the queue data
Mark.And assume that the data being stored in object queue are sequentially numbered, such as data 1, data 2 ....Phase
Ying Di, the data storage record of the above-mentioned object queue corresponding according to target data type, determine number of targets
According to the data volume of type, the data characterized according to tail of the queue Data Identification can be accomplished in that
Amount and data duplicate number, determine the total amount of data of target data type.
Because each data under normal circumstances, being stored in the object queue of certain memory node are all to be stored
To this memory node, its duplicate is deposited in other memory nodes, therefore, it can based on current mesh
In mark queue, the Data Identification of tail of the queue is known and currently have received how many data, thus according to tail of the queue number
The data volume characterized according to mark and data duplicate number, it is also possible to determine the target that this memory node is corresponding
The total amount of data of data type.
For arbitrary memory node, when the data volume of the target data type that defining self correspondence
Afterwards, to client feedback data inquiry response, to inform this result so that client-side can be based on
The feedback result of each memory node received, obtains the total data of target data type by cumulative calculation
Amount.
It addition, what deserves to be explained is, each queue can have capacity limit, and this capacity limit both may be used
To be time restriction, it is also possible to being that quantity number limits, such as one queue is configured to store 3
The data that it time memory enters, or 50000 data can be stored.When the capacity limit beyond queue
Or when reaching certain capacity threshold, a certain amount of data most starting to be stored in queue can be deleted.
In the present embodiment, when the storage data volume of data needing monitoring to know Various types of data type, logical
Cross the inquiry request sending data volume for inquiring about certain or some data types to each memory node, by
In storing process, the team of a correspondence is had for every kind of data type data are carried out in each memory node
Row are in order to data cached, during the memory space that from queue, data are stored to memory node, and shape
Becoming the data storage record of this queue, therefore, each memory node can be based on the data with required inquiry
The data storage record of the queue that type is corresponding, it is thus achieved that the data volume of this data type, eventually through accumulative
The feedback result of each memory node, it is possible to obtain the total amount of data of this certain or some data type, with reality
Now the real-time of data volume of all types of data is understood, provide foundation for other processing procedures follow-up.
Fig. 2 is the flow chart of data monitoring method embodiment two of the present invention, the data prison that this embodiment provides
Survey method can have a data monitoring device to perform, and it is soft that this data monitoring device can be implemented as client
Part application program, is carried in certain hardware device, and this hardware device can be arbitrary memory node,
Other equipment of memory node can also be different from, but have between this other equipment and each memory node
Data interaction.As in figure 2 it is shown, the method comprises the steps:
Step 201, to each memory node send data inquiry request, this data inquiry request is used for inquiring about
The total amount of data of target data type.
Step 202, receive each memory node send data query response, each data query response in
Comprise the data volume of target data type corresponding to corresponding memory node.
Wherein, the data volume of the target data type that each memory node is corresponding be corresponding memory node according to
The data storage record of the object queue that target data type is corresponding obtains, and object queue is used for storing mesh
Main of the data that mark data type is corresponding.
Step 203, data volume according to target data type corresponding to each memory node, it is thus achieved that target
The total amount of data of data type.
The present embodiment is from client-side, to how inquiring about the total amount of data that obtains certain target data type
The introduction that process is carried out, concrete implementation mode is referred to the description in embodiment illustrated in fig. 1, herein
Repeat no more.
But, what deserves to be explained is, the client inquiry to the total amount of data of either objective data type,
Can be the periodic queries at a time interval for the cycle, this time interval can be according to practical situation
Adaptive setting such that it is able to understand the data volume situation of each type data in time.
In the present embodiment, when the storage data volume of data needing monitoring to know Various types of data type, logical
Cross the inquiry request sending data volume for inquiring about certain or some data types to each memory node, by
In storing process, the team of a correspondence is had for every kind of data type data are carried out in each memory node
Row are in order to data cached, during the memory space that from queue, data are stored to memory node, and shape
Becoming the data storage record of this queue, therefore, each memory node can be based on the data with required inquiry
The data storage record of the queue that type is corresponding, it is thus achieved that the data volume of this data type, eventually through accumulative
The feedback result of each memory node, it is possible to obtain the total amount of data of this certain or some data type, with reality
Now the real-time of data volume of all types of data is understood, provide foundation for other processing procedures follow-up.
Fig. 3 is the flow chart of data monitoring method embodiment three of the present invention, based on embodiment illustrated in fig. 2,
Can inquire about and obtain the data volume that in each memory node, every kind of data type is corresponding, such that it is able to obtain every
Plant the total amount of data that data type is corresponding, i.e. for any data type, storage in whole memory nodes
The total amount of data of this any data type, on this basis, as shown in Figure 3, it is also possible to include walking as follows
Rapid:
Step 301, the total data of each target data type obtained according to inquiry each in preset time period
Amount, determines total amount of data variable quantity in preset time period.
Wherein, the conceptual data amount of the data that this total amount of data variable quantity characterizes total data type changes
Situation.
If step 302 total amount of data variable quantity is more than or equal to preset data amount threshold value, then limit depositing
The access of the data of storage.
In the present embodiment, at a time, can look into respectively to use as shown in Figure 2 in the way of embodiment
Ask the data volume obtaining every kind of data type, thus, add and obtain the total amount of data 1 of total data type.
After a certain interval, inquiry obtains the data volume of every kind of data type the most respectively, adds and obtains complete
The total amount of data 2 of portion's data type.By total amount of data 2 and the difference operation of total amount of data 1, it is possible to
Obtain the variable quantity of total amount of data in during this period of time.
If the total amount of data variable quantity in during this period of time is bigger, data volume the most such as occurs the most significantly
The situation that degree increases, then in order to reduce the processing pressure of memory node, it is ensured that being reliably stored in of data, this
Time can limit the access to data.
Wherein, the data access of user is limited, can be to limit: access the number of users of data, visit
The time span asked, the data type of access, access data volume how many.
In the present embodiment, by the analysis to total amount of data intensity of variation, it is possible to find total amount of data in time
Uprush or the situation of anticlimax.Situation about uprushing based on the total amount of data found, by limiting the visit to data
Ask that the reliable memory ensureing data processes.
Previous embodiment is described, below from storage from the angle of data volume corresponding to each data type
The angle of the storage load of node, present invention also offers examples below:
Fig. 4 is the flow chart of data monitoring method embodiment four of the present invention, as shown in Figure 4, it is also possible to bag
Include following steps:
Step 401, obtain in each memory node the data volume of storage, and determine that the data volume of storage is big
Number in the memory node of predetermined threshold value.
If the number of this memory node of step 402 is more than predetermined number, it is determined that need to increase new depositing
Storage node.
If the number of this memory node of step 403 is less than predetermined number, it is determined that need to carry out storage joint
Load balancing between point.
Not there is between step 402 and step 403 sequential qualified relation.
In the present embodiment, it is many that client can also inquire about the data volume obtaining each memory node currently stored
Few, wherein, in each memory node, the data volume of storage includes main of the above-mentioned data mentioned and data
Duplicate.I.e. in certain memory node, wherein may store and have the data of numerous types of data, every kind of number
According to main of the data that both can include the type data in the data of type, it is also possible to include the type data
Data duplicate.
When after the data volume obtaining each memory node, based on the comparison with predetermined threshold value, it is possible to obtain deposit
The data volume of storage is more than the number of the memory node of predetermined threshold value.If this number is more than predetermined number, say
Bright current most of memory nodes have tended to heavy duty, and storage capacity is not enough, now determines needs
Increase new memory node, with dilatation storage capacity.If this number is less than predetermined number, now illustrate
The load having part memory node is the heaviest, and the load of other major part memory nodes is the heaviest, the most really
Surely need to carry out the load balancing between memory node.Wherein, the implementation of load balancing can use existing
The multiple load-balancing mechanism having realizes, and its purpose mainly avoids data in the memory node of load weight
Storage.
By to the inquiry of the data volume of storage in each memory node, it is possible to based on memory node
Loading condition processes accordingly, it is ensured that the reliability of data storage.
Fig. 5 is the structural representation of data query arrangement embodiment of the present invention, as it is shown in figure 5, these data
Inquiry unit includes: receiver module 11, determine module 12, sending module 13.
Receiver module 11, is used for receiving data inquiry request, and described data inquiry request is used for inquiring about target
The data volume of data type.
Determine module 12, for the data storage note of the object queue corresponding according to described target data type
Record, determines that the data volume of described target data type, described object queue are used for storing described target data
Main of the data that type is corresponding.
Sending module 13, is used for sending data query response, and the response of described data query includes described mesh
The data volume of mark data type.
Optionally, object queue is for storing main of data corresponding to described target data type.Described mesh
Mark record in the data storage record of queue and have the data fragmentation mark of data fragmentation and described data fragmentation
Duplicate number.
Described determine that module 12 includes: first determines unit 121.
First determines unit 121, for the quantity according to data fragmentation mark, each data fragmentation correspondence
Data volume and duplicate number corresponding to each data fragmentation, determine the data of described target data type
Amount.
Optionally, in the data storage record of described object queue, record has tail of the queue Data Identification, is stored in institute
The data stated in object queue are sequentially numbered；Described determine that module 12 includes: second determines unit 122.
Second determines unit 122, for the data volume characterized according to described tail of the queue Data Identification and data
Duplicate number, determines the data volume of described target data type.
Data query arrangement shown in Fig. 5 can perform the method for Fig. 1 and embodiment illustrated in fig. 2, it is achieved
Principle and technique effect, with reference to Fig. 1 and embodiment illustrated in fig. 2, repeat no more.
Fig. 6 is the structural representation of data monitoring device embodiment one of the present invention, as shown in Figure 6, this dress
Put and include: sending module 21, receiver module 22, first determine module 23.
Sending module 21, for sending data inquiry request, described data inquiry request to each memory node
For inquiring about the total amount of data of target data type.
Receiver module 22, for receiving the data query response that each memory node sends, each described data
Inquiry response includes the data volume of the described target data type that corresponding memory node is corresponding.
Wherein, the data volume of the described target data type that each memory node is corresponding is corresponding memory node
Data storage record according to object queue corresponding to described target data type obtains, described target team
Row are for storing main of data corresponding to described target data type.
First determines module 23, for the number of the described target data type corresponding according to each memory node
According to amount, it is thus achieved that the total amount of data of described target data type.
Fig. 6 shown device can perform the method for embodiment illustrated in fig. 2, it is achieved principle and technique effect ginseng
Examine Fig. 1 and embodiment illustrated in fig. 2, repeat no more.
Fig. 7 is the structural representation of data monitoring device embodiment two of the present invention, as it is shown in fig. 7, at figure
On the basis of 6 illustrated embodiments, this device also includes: second determines module 31, access process module 32.
Second determines module 31, for according to each target data class that in preset time period, inquiry obtains every time
The total amount of data of type, determines total amount of data variable quantity in described preset time period.
Access process module 32, if for described total amount of data variable quantity more than or equal to preset data amount threshold
Value, then limit the access to the data stored.
Fig. 7 shown device can perform the method for embodiment illustrated in fig. 3, it is achieved principle and technique effect ginseng
Examine embodiment illustrated in fig. 3, repeat no more.
Fig. 8 is the structural representation of data monitoring device embodiment three of the present invention, as shown in Figure 8, at figure
On the basis of 6 illustrated embodiments, this device also includes: acquisition module the 41, the 3rd determines module the 42, the 4th
Determine module 43.
Acquisition module 41, for obtaining the data volume of storage in each memory node, and determines the number of storage
The number of the memory node of predetermined threshold value it is more than according to amount.
3rd determines module 42, and the number for described memory node is more than predetermined number, it is determined that need
Increase new memory node.
4th determines module 43, if the number for described memory node is less than predetermined number, it is determined that need
Load balancing between memory node to be carried out.
Fig. 8 shown device can perform the method for embodiment illustrated in fig. 4, it is achieved principle and technique effect ginseng
Examine embodiment illustrated in fig. 4, repeat no more.
Device embodiment described above is only schematically, wherein said illustrates as separating component
Unit can be or may not be physically separate, the parts shown as unit can be or
Person may not be physical location, i.e. may be located at a place, or can also be distributed to multiple network
On unit.Some or all of module therein can be selected according to the actual needs to realize the present embodiment
The purpose of scheme.Those of ordinary skill in the art are not in the case of paying performing creative labour, the most permissible
Understand and implement.
Through the above description of the embodiments, those skilled in the art is it can be understood that arrive each reality
The mode of executing can add the mode of required general hardware platform by software and realize, naturally it is also possible to by firmly
Part.Based on such understanding, the portion that prior art is contributed by technique scheme the most in other words
Dividing and can embody with the form of software product, this computer software product can be stored in computer can
Read in storage medium, such as ROM/RAM, magnetic disc, CD etc., including some instructions with so that a meter
Calculate machine (can be personal computer, server, or network equipment etc.) and perform each embodiment
Or the method described in some part of embodiment.
Last it is noted that above example is only in order to illustrate technical scheme, rather than to it
Limit；Although the present invention being described in detail with reference to previous embodiment, the ordinary skill of this area
Personnel it is understood that the technical scheme described in foregoing embodiments still can be modified by it, or
Person carries out equivalent to wherein portion of techniques feature；And these amendments or replacement, do not make corresponding skill
The essence of art scheme departs from the spirit and scope of various embodiments of the present invention technical scheme.
Claims (12)
1. a data query method, it is characterised in that comprise the following steps that
Receiving data inquiry request, described data inquiry request is for inquiring about the data volume of target data type；
Data storage record according to object queue corresponding to described target data type, determines described target
The data volume of data type；
Transmission data query responds, and the response of described data query includes the data of described target data type
Amount.
Method the most according to claim 1, it is characterised in that described object queue is used for storing institute
Stating main of data corresponding to target data type, in the data storage record of described object queue, record has number
Identify and the duplicate number of described data fragmentation according to the data fragmentation of burst；
The data storage record of the described object queue corresponding according to target data type, determines described target
The data volume of data type, including:
Data volume that the quantity identified according to described data fragmentation, each data fragmentation are corresponding and every number
According to the duplicate number that burst is corresponding, determine the data volume of described target data type.
Method the most according to claim 1, it is characterised in that the data storage of described object queue
In record, record has tail of the queue Data Identification, and the data being stored in described object queue are sequentially numbered；
The data storage record of the described object queue corresponding according to target data type, determines described target
The data volume of data type, including:
The data volume characterized according to described tail of the queue Data Identification and data duplicate number, determine described target
The data volume of data type.
4. a data monitoring method, it is characterised in that comprise the following steps that
Sending data inquiry request to each memory node, described data inquiry request is used for inquiring about target data
The total amount of data of type；
Receiving the data query response that each memory node sends, it is right that the response of each described data query includes
Answer the data volume of the described target data type that memory node is corresponding；
Data volume according to described target data type corresponding to each memory node, it is thus achieved that described number of targets
Total amount of data according to type.
Method the most according to claim 4, it is characterised in that described method also includes:
According to the total amount of data of each target data type inquiring about acquisition in preset time period every time, determine institute
State total amount of data variable quantity in preset time period；
If described total amount of data variable quantity is more than or equal to preset data amount threshold value, then limit the number to storage
According to access.
Method the most according to claim 4, it is characterised in that described method also includes:
Obtain the data volume of storage in each memory node；
Determine the data volume number more than the memory node of predetermined threshold value of storage；If described memory node
Number is more than predetermined number, it is determined that need to increase new memory node；
If the number of described memory node is less than predetermined number, it is determined that need to carry out bearing between memory node
Carry equilibrium.
7. a data query arrangement, it is characterised in that including:
Receiver module, is used for receiving data inquiry request, and described data inquiry request is used for inquiring about number of targets
Data volume according to type；
Determine module, for the data storage record of the object queue corresponding according to described target data type,
Determine the data volume of described target data type；
Sending module, is used for sending data query response, and the response of described data query includes described target
The data volume of data type.
Memory node the most according to claim 7, it is characterised in that described object queue is used for depositing
Store up main of data corresponding to described target data type, record in the data storage record of described object queue
There are data fragmentation mark and the duplicate number of described data fragmentation of data fragmentation；
Described determine that module includes:
First determines unit, for the quantity identified according to described data fragmentation, each data fragmentation correspondence
Data volume and duplicate number corresponding to each data fragmentation, determine the data of described target data type
Amount.
Memory node the most according to claim 7, it is characterised in that the data of described object queue
In storage record, record has tail of the queue Data Identification, and the data being stored in described object queue are sequentially numbered；
Described determine that module includes:
Second determines unit, for the data volume characterized according to described tail of the queue Data Identification and data duplicate
Number, determines the data volume of described target data type.
10. a data monitoring device, it is characterised in that including:
Sending module, for sending data inquiry request to each memory node, described data inquiry request is used
Total amount of data in inquiry target data type；
Receiver module, for receiving the data query response that each memory node sends, each described data are looked into
Ask the data volume of the described target data type that response includes that corresponding memory node is corresponding；
First determines module, for according to the number of the described target data type of storage in each memory node
According to amount, it is thus achieved that the total amount of data of described target data type.
11. data monitoring devices according to claim 10, it is characterised in that also include:
Second determines module, for according to each target data type that in preset time period, inquiry obtains every time
Total amount of data, determine total amount of data variable quantity in described preset time period；
Access process module, if for described total amount of data variable quantity more than or equal to preset data amount threshold value,
Then limit the access to the data stored.
12. data monitoring devices according to claim 10, it is characterised in that also include:
Acquisition module, for obtaining the data volume of storage in each memory node, and determines the data of storage
Amount is more than the number of the memory node of predetermined threshold value；
3rd determines module, and the number for described memory node is more than predetermined number, it is determined that need to increase
Add new memory node；
4th determines module, if the number for described memory node is less than predetermined number, it is determined that need
Carry out the load balancing between memory node.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610282721.5A CN106021284A (en) | 2016-04-29 | 2016-04-29 | Data query method, data monitoring method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610282721.5A CN106021284A (en) | 2016-04-29 | 2016-04-29 | Data query method, data monitoring method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN106021284A true CN106021284A (en) | 2016-10-12 |
Family
ID=57081383
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610282721.5A Pending CN106021284A (en) | 2016-04-29 | 2016-04-29 | Data query method, data monitoring method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106021284A (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107153703A (en) * | 2017-05-10 | 2017-09-12 | 华自科技股份有限公司 | Data correlation method for pushing and system |
| CN107357915A (en) * | 2017-07-19 | 2017-11-17 | 郑州云海信息技术有限公司 | A kind of date storage method and system |
| CN108156189A (en) * | 2016-12-02 | 2018-06-12 | 航天星图科技（北京）有限公司 | Redundant data processing method in a kind of multi-node system |
| CN109634986A (en) * | 2018-10-29 | 2019-04-16 | 平安科技（深圳）有限公司 | System monitoring method, device, computer and computer readable storage medium |
| CN111274010A (en) * | 2020-01-14 | 2020-06-12 | Oppo广东移动通信有限公司 | Data control method and device, electronic equipment and storage medium |
| CN112328688A (en) * | 2020-11-09 | 2021-02-05 | 广州虎牙科技有限公司 | Data storage method and device, computer equipment and storage medium |
| CN115952202A (en) * | 2023-02-24 | 2023-04-11 | 北京锐服信科技有限公司 | Query method |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101923569A (en) * | 2010-07-09 | 2010-12-22 | 南京朗坤软件有限公司 | Storage method of structure type data of real-time database |
| CN102158531A (en) * | 2010-02-01 | 2011-08-17 | 北京大学 | Distributed transmission method for query data stream |
| US9110965B1 (en) * | 2013-03-06 | 2015-08-18 | Symantec Corporation | Systems and methods for disaster recovery from binary large objects |
-
2016
- 2016-04-29 CN CN201610282721.5A patent/CN106021284A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102158531A (en) * | 2010-02-01 | 2011-08-17 | 北京大学 | Distributed transmission method for query data stream |
| CN101923569A (en) * | 2010-07-09 | 2010-12-22 | 南京朗坤软件有限公司 | Storage method of structure type data of real-time database |
| US9110965B1 (en) * | 2013-03-06 | 2015-08-18 | Symantec Corporation | Systems and methods for disaster recovery from binary large objects |
Non-Patent Citations (1)
| Title |
|---|
| 徐伟: "流式数据实时查询方法研究", 《中国优秀硕士学位论文全文数据库》 * |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108156189A (en) * | 2016-12-02 | 2018-06-12 | 航天星图科技（北京）有限公司 | Redundant data processing method in a kind of multi-node system |
| CN107153703A (en) * | 2017-05-10 | 2017-09-12 | 华自科技股份有限公司 | Data correlation method for pushing and system |
| CN107153703B (en) * | 2017-05-10 | 2020-05-12 | 华自科技股份有限公司 | Data association pushing method and system |
| CN107357915A (en) * | 2017-07-19 | 2017-11-17 | 郑州云海信息技术有限公司 | A kind of date storage method and system |
| CN109634986A (en) * | 2018-10-29 | 2019-04-16 | 平安科技（深圳）有限公司 | System monitoring method, device, computer and computer readable storage medium |
| CN109634986B (en) * | 2018-10-29 | 2024-03-29 | 平安科技（深圳）有限公司 | System monitoring method, device, computer and computer readable storage medium |
| CN111274010A (en) * | 2020-01-14 | 2020-06-12 | Oppo广东移动通信有限公司 | Data control method and device, electronic equipment and storage medium |
| CN111274010B (en) * | 2020-01-14 | 2024-01-16 | Oppo广东移动通信有限公司 | Data control method, device, electronic equipment and storage medium |
| CN112328688A (en) * | 2020-11-09 | 2021-02-05 | 广州虎牙科技有限公司 | Data storage method and device, computer equipment and storage medium |
| CN112328688B (en) * | 2020-11-09 | 2023-10-13 | 广州虎牙科技有限公司 | Data storage method, device, computer equipment and storage medium |
| CN115952202A (en) * | 2023-02-24 | 2023-04-11 | 北京锐服信科技有限公司 | Query method |
| CN115952202B (en) * | 2023-02-24 | 2023-08-22 | 北京锐服信科技有限公司 | Big data fragment query method based on cache |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106021284A (en) | Data query method, data monitoring method and device | |
| US9948705B2 (en) | Load balanced network file accesses | |
| US9667738B2 (en) | Local data caching for data transfers on a network of computational devices | |
| CN108780386A (en) | A kind of methods, devices and systems of data storage | |
| US8191068B2 (en) | Resource management system, resource information providing method and program | |
| KR101429555B1 (en) | System and method for providing high availability data | |
| US8849756B2 (en) | Selecting data nodes in distributed storage system | |
| US10412188B2 (en) | Data caching, distribution and request consolidation in a local network | |
| CN103347068B (en) | A kind of based on Agent cluster network-caching accelerated method | |
| CN111405019B (en) | Data processing method, data processing device, computer equipment and storage medium | |
| CN106899681A (en) | The method and server of a kind of information pushing | |
| EP3180692A1 (en) | Fault tolerant federation of computing clusters | |
| US11922059B2 (en) | Method and device for distributed data storage | |
| CN103634401A (en) | Data copy storage method and terminal unit, and server unit | |
| CN108696581A (en) | Caching method, device, computer equipment and the storage medium of distributed information | |
| CN104579765A (en) | Disaster tolerance method and device for cluster system | |
| CN109981702A (en) | A kind of file memory method and system | |
| CN110401711A (en) | Data processing method, device, system and storage medium | |
| CN105991769A (en) | Data migration method and device and edge node server | |
| CN109413202A (en) | The ordering system and method for block chain Transaction Information | |
| CN106973091B (en) | Distributed memory data redistribution method and system, and master control server | |
| CN112492022A (en) | Cluster, method, system and storage medium for improving database availability | |
| US11277489B2 (en) | Software application updating in a local network | |
| CN108304555A (en) | Distributed maps data processing method | |
| CN108459926A (en) | Data remote backup method, apparatus and computer-readable medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20161012 |
|
| WD01 | Invention patent application deemed withdrawn after publication |