CN106681995A - Data caching method and data query method and device - Google Patents

Data caching method and data query method and device Download PDF

Info

Publication number
CN106681995A
CN106681995A CN201510746534.3A CN201510746534A CN106681995A CN 106681995 A CN106681995 A CN 106681995A CN 201510746534 A CN201510746534 A CN 201510746534A CN 106681995 A CN106681995 A CN 106681995A
Authority
CN
China
Prior art keywords
data
data element
mark
internal memory
linked list
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201510746534.3A
Other languages
Chinese (zh)
Other versions
CN106681995B (en
Inventor
苏国庆
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Cainiao Smart Logistics Holding Ltd
Original Assignee
Alibaba Group Holding Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Alibaba Group Holding Ltd filed Critical Alibaba Group Holding Ltd
Priority to CN201510746534.3A priority Critical patent/CN106681995B/en
Publication of CN106681995A publication Critical patent/CN106681995A/en
Application granted granted Critical
Publication of CN106681995B publication Critical patent/CN106681995B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/245—Query processing
    • G06F16/2455—Query execution
    • G06F16/24552—Database cache management

Abstract

The invention discloses a data caching method and a data query method and device. The data caching method includes: receiving to-be-cached data; acquiring the identification of each first data element comprised by the to-be-cached data; determining whether an existing data element corresponding to the identification of each first data element is saved in the internal memory of a big data terminal or not; if so, replacing the corresponding existing data element with each first data element; if not, determining whether the memory space of the internal memory reaches a set threshold or not; if the memory space of the internal memory does not reach the set threshold, saving each first data element and the identification of the first data element into the internal memory; if the memory space of the internal memory reaches the set threshold, saving each first data element into the external memory of the big data terminal, and saving the identification and the saving position of the first data element into the internal memory. By the data caching method, problems of related techniques can be avoided effectively.

Description

Data cache method, data query method and device
Technical field
The application is related to networking technology area, more particularly to a kind of data cache method, data query method and device.
Background technology
With developing rapidly for network technology, increasing user needs data to be processed by network acquisition information, website Amount is also increasing, and the big data epoch are already entered now.The service end of website generally include at least one big data terminal, Push server and data base, wherein, different big data terminals may process the different business of user, each big data end Sometimes data interaction can be also carried out between end, that is to say, that each big data terminal may process big with other from user The data acquisition request of data terminal, big data terminal obtains request to the time needed for feedback data to respond from receiving data Time, it is obvious that response time is shorter, the performance of big data terminal is better, and Consumer's Experience is also better.
In order to reduce response time, push server can obtain the full dose data in data base with setting cycle, then again will be every The data-pushing of individual big data terminal gives corresponding big data terminal.According to related data cache method, each big data end Hold and the data that push server is pushed are received with setting cycle, if there is data with existing in the internal memory of big data terminal, will connect The data for receiving replace data with existing, if there is no data with existing in the internal memory of big data terminal, directly by the number for receiving According to being stored in internal memory, after big data terminal receives data acquisition request, can obtain from the data of memory cache needs Data element, and feed back to send data acquisition request user or other big data terminals.
It is to ensure that big data terminal can be several replacing with normal process data acquisition request in above-mentioned data cache method According to during, two number evidences can be preserved in internal memory, because this two number is according to the big data end for being all the preservation in data base Total data element needed for end, data volume is very big, and the process of replacement data can take many memory sources, so as to easy Cause low memory, or even Memory recycle and big data terminal can be caused to delay machine.
The content of the invention
The embodiment of the present application provides a kind of data cache method, data query method and device, to solve correlation technique in deposit The process of replacement data take many memory sources, so as to easily cause low memory, in addition can cause Memory recycle and Big data terminal is delayed the problem of machine.
According to the embodiment of the present application, there is provided a kind of data cache method, apply in big data terminal, including：
Reception treats data cached, for it is described treat it is data cached including each first data element, perform：
Obtain the mark of first data element；
Whether the mark corresponding data with existing unit of first data element is preserved in the internal memory for determining the big data terminal Element；
If it is determined that preserving the data with existing element in the internal memory, then the data with existing element is replaced with into first number According to element；
If it is determined that not preserving the data with existing element in the internal memory, whether the amount of storage for determining the internal memory reaches setting threshold Value；If the amount of storage of the internal memory is not up to the given threshold, first data element and its mark are stored in into institute In stating internal memory；If the amount of storage of the internal memory reaches the given threshold, first data element is stored in described big In the external memory storage of data terminal, and the mark and storage location of first data element are stored in the internal memory.
Specifically, the mark of first data element is obtained, is specifically included：
Search from first data element and select field；
The numerical value of the selected field is obtained, the mark of first data element is obtained.
Specifically, the mark that first data element whether is preserved in the internal memory for determining the big data terminal is corresponding existing Data element, specifically includes：
Obtain the doubly linked list with index preserved in the internal memory；
Determine whether include the mark of first data element in the index of the doubly linked list；
If the index of the doubly linked list includes the mark of first data element, it is determined that preserve in the internal memory described Data with existing element；
If not including the mark of first data element in the index of the doubly linked list, it is determined that do not preserve in the internal memory The data with existing element.
Optionally, determine and preserve in the internal memory after the data with existing element, also include：
Obtain corresponding first node of mark of the first data element described in the doubly linked list；
The data with existing element is obtained from the data field of first node；
Contrast the content that first data element includes whether identical with the content that the data with existing element includes；
If the content that the content that first data element includes includes with the data with existing element is incomplete same, perform The step of data with existing element is replaced with into first data element.
Specifically, first data element and its mark are stored in the internal memory, are specifically included：
The mark of first data element is preserved in the index of the doubly linked list；
The second node corresponding with the mark of first data element is set up in the doubly linked list, and described first is counted It is stored in the data field of the secondary nodal point according to element.
Specifically, first data element is stored in the external memory storage of the big data terminal, and by described first The mark and storage location of data element is stored in the internal memory, is specifically included：
First data element is stored in the file of the external memory storage of the big data terminal；
Obtain storage location of first data element in the external memory storage of the big data terminal；
Set up in the corresponding relation of mark and storage location that the internal memory is preserved include the mark of first data element and The list item of storage location.
Optionally, also include：
The data acquisition request of the mark of the data element of carrying second that receive user or other big data terminals send；
Second data element or second data are inquired about in the internal memory according to the mark of second data element The storage location of element；
If inquiring second data element, second data element is sent to into the user or other big datas Terminal；
If inquiring the storage location of second data element, according to the storage location of second data element from described External memory storage obtains second data element, and is sent to the user or other big data terminals；And, by institute State the second data element to be stored in the internal memory.
Optionally, second data element or described is inquired about in the internal memory according to the mark of second data element The storage location of the second data element, specifically includes：
In the doubly linked list with index for preserving from the internal memory respectively and in the corresponding relation of mark and storage location Search the mark of second data element；
If finding the mark of second data element in the doubly linked list, obtain described from the doubly linked list Corresponding 3rd node of mark of the second data element, and obtain second data element from the data field of the 3rd node Element；
If finding the list item of the mark including second data element in the corresponding relation, obtain in the list item Take the storage location of second data element.
Optionally, after inquiring second data element, also include：
Update the accessed number of times of the second data element described in the 3rd node；And,
The position of each node in the doubly linked list is adjusted according to accessed number of times, and accordingly adjusts each in the doubly linked list Pointer in individual node.
Specifically, second data element is stored in the internal memory, is specifically included：
Whether the amount of storage for judging the internal memory reaches the given threshold；
If the amount of storage of the internal memory reaches the given threshold, it is determined that the accessed least number of times preserved in the internal memory 3rd data element, the 3rd data element is stored in the external memory storage, and is preserved in the corresponding relation The list item of mark and storage location including the 3rd data element；Delete in the index of the doubly linked list preserve it is described The node that the 3rd data element is located described in the mark of the 3rd data element and the doubly linked list；In the doubly linked list The 4th node corresponding with the mark of second data element is set up, second data element is stored in into described Section four In the data field of point, and the mark of second data element is preserved in the index of the doubly linked list；Update described The accessed number of times of the second data element described in four nodes, according to accessed number of times each node in the doubly linked list is adjusted Position, and accordingly adjust the pointer in the doubly linked list in each node；
If the amount of storage of the internal memory is not up to the given threshold, set up in the doubly linked list and second data Corresponding 5th node of mark of element, second data element is stored in the data field of the 5th node, and The mark of second data element is preserved in the index of the doubly linked list；Update the second number described in the 5th node According to the accessed number of times of element, the position of each node in the doubly linked list is adjusted according to accessed number of times, and accordingly adjusted Pointer in the doubly linked list in each node.
According to the embodiment of the present application, a kind of data query method is also provided, applied in big data terminal, including：
The data acquisition request of the mark of the data element of carrying second that receive user or other big data terminals send；
According to the mark of second data element inquire about in the internal memory of the big data terminal second data element or The storage location of second data element；
If inquiring second data element, second data element is sent to into the user or other big datas Terminal；
If inquiring the storage location of second data element, according to the storage location of second data element from described The external memory storage of big data terminal obtains second data element, and is sent to the user or other big data terminals； And, second data element is stored in the internal memory.
Specifically, second data element or described is inquired about in the internal memory according to the mark of second data element The storage location of the second data element, specifically includes：
In the doubly linked list with index for preserving from the internal memory respectively and in the corresponding relation of mark and storage location Search the mark of second data element；
If finding the mark of second data element in the doubly linked list, obtain described from the doubly linked list Corresponding 3rd node of mark of the second data element, and obtain second data element from the data field of the 3rd node Element；
If finding the list item of the mark including second data element in the corresponding relation, obtain in the list item Take the storage location of second data element.
Optionally, after inquiring second data element, also include：
Update the accessed number of times of the second data element described in the 3rd node；And,
The position of each node in the doubly linked list is adjusted according to accessed number of times, and accordingly adjusts each in the doubly linked list Pointer in individual node.
Specifically, second data element is stored in the internal memory, is specifically included：
Whether the amount of storage for judging the internal memory reaches the given threshold；
If the amount of storage of the internal memory reaches the given threshold, it is determined that the accessed least number of times preserved in the internal memory 3rd data element, the 3rd data element is stored in the external memory storage, and is preserved in the corresponding relation The list item of mark and storage location including the 3rd data element；Delete in the index of the doubly linked list preserve it is described The node that the 3rd data element is located described in the mark of the 3rd data element and the doubly linked list；In the doubly linked list The 4th node corresponding with the mark of second data element is set up, second data element is stored in into described Section four In the data field of point, and the mark of second data element is preserved in the index of the doubly linked list；Update described The accessed number of times of the second data element described in four nodes, according to accessed number of times each node in the doubly linked list is adjusted Position, and accordingly adjust the pointer in the doubly linked list in each node；
If the amount of storage of the internal memory is not up to the given threshold, set up in the doubly linked list and second data Corresponding 5th node of mark of element, second data element is stored in the data field of the 5th node, and The mark of second data element is preserved in the index of the doubly linked list；Update the second number described in the 5th node According to the accessed number of times of element, the position of each node in the doubly linked list is adjusted according to accessed number of times, and accordingly adjusted Pointer in the doubly linked list in each node.
According to the embodiment of the present application, a kind of data buffer storage device is also provided, applied in big data terminal, including：
First receiving unit, for receive treat data cached, for it is described treat it is data cached including each first data element, Perform：
Acquiring unit, for obtaining the mark of first data element；
Determining unit, for whether preserving the mark correspondence of first data element in the internal memory for determining the big data terminal Data with existing element；
Buffer unit, for if it is determined that preserve the data with existing element in the internal memory, then replacing the data with existing element It is changed to first data element；If it is determined that not preserving the data with existing element in the internal memory, depositing for the internal memory is determined Whether reserves reach given threshold；If the amount of storage of the internal memory is not up to the given threshold, by first data element Element and its mark are stored in the internal memory；If the amount of storage of the internal memory reaches the given threshold, by the described first number It is stored in the external memory storage of the big data terminal according to element, and by the mark and storage location of first data element In being stored in the internal memory.
Specifically, the acquiring unit, for obtaining the mark of first data element, specifically for：
Search from first data element and select field；
The numerical value of the selected field is obtained, the mark of first data element is obtained.
Specifically, the determining unit, for whether preserving first data element in the internal memory for determining the big data terminal The corresponding data with existing element of mark of element, specifically includes：
Obtain the doubly linked list with index preserved in the internal memory；
Determine whether include the mark of first data element in the index of the doubly linked list；
If the index of the doubly linked list includes the mark of first data element, it is determined that preserve in the internal memory described Data with existing element；
If not including the mark of first data element in the index of the doubly linked list, it is determined that do not preserve in the internal memory The data with existing element.
Optionally, the determining unit, is additionally operable to：
Determine and preserve in the internal memory after the data with existing element, obtain the first data element described in the doubly linked list Corresponding first node of mark；
The data with existing element is obtained from the data field of first node；
Contrast the content that first data element includes whether identical with the content that the data with existing element includes；
If the content that the content that first data element includes includes with the data with existing element is incomplete same, turn to The buffer unit.
Specifically, the buffer unit, for first data element and its mark to be stored in the internal memory, specifically For：
The mark of first data element is preserved in the index of the doubly linked list；
The second node corresponding with the mark of first data element is set up in the doubly linked list, and described first is counted It is stored in the data field of the secondary nodal point according to element.
Specifically, the buffer unit, for first data element to be stored in the external storage of the big data terminal In device, and the mark and storage location of first data element are stored in the internal memory, specifically for：
First data element is stored in the file of the external memory storage of the big data terminal；
Obtain storage location of first data element in the external memory storage of the big data terminal；
Set up in the corresponding relation of mark and storage location that the internal memory is preserved include the mark of first data element and The list item of storage location.
Optionally, also include：
Second receiving unit, the mark of the data element of carrying second sent for receive user or other big data terminals Data acquisition request；
Query unit, for according to the mark of second data element inquire about in the internal memory second data element or The storage location of the second data element described in person；
Transmitting element, if for inquiring second data element, second data element is sent to into the user Or other big data terminals；If inquiring the storage location of second data element, according to second data element Storage location obtain second data element from the external memory storage, and be sent to the user or other big datas Terminal；And, second data element is stored in the internal memory.
Specifically, the query unit, for described to be inquired about in the internal memory according to the mark of second data element The storage location of two data elements or second data element, specifically for：
In the doubly linked list with index for preserving from the internal memory respectively and in the corresponding relation of mark and storage location Search the mark of second data element；
If finding the mark of second data element in the doubly linked list, obtain described from the doubly linked list Corresponding 3rd node of mark of the second data element, and obtain second data element from the data field of the 3rd node Element；
If finding the list item of the mark including second data element in the corresponding relation, obtain in the list item Take the storage location of second data element.
Optionally, the transmitting element, is additionally operable to：
After the query unit inquires second data element, the second data element described in the 3rd node is updated The accessed number of times of element；And,
The position of each node in the doubly linked list is adjusted according to accessed number of times, and accordingly adjusts each in the doubly linked list Pointer in individual node.
Specifically, the transmitting element, for second data element to be stored in the internal memory, specifically for：
Whether the amount of storage for judging the internal memory reaches the given threshold；
If the amount of storage of the internal memory reaches the given threshold, it is determined that the accessed least number of times preserved in the internal memory 3rd data element, the 3rd data element is stored in the external memory storage, and is preserved in the corresponding relation The list item of mark and storage location including the 3rd data element；Delete in the index of the doubly linked list preserve it is described The node that the 3rd data element is located described in the mark of the 3rd data element and the doubly linked list；In the doubly linked list The 4th node corresponding with the mark of second data element is set up, second data element is stored in into described Section four In the data field of point, and the mark of second data element is preserved in the index of the doubly linked list；Update described The accessed number of times of the second data element described in four nodes, according to accessed number of times each node in the doubly linked list is adjusted Position, and accordingly adjust the pointer in the doubly linked list in each node；
If the amount of storage of the internal memory is not up to the given threshold, set up in the doubly linked list and second data Corresponding 5th node of mark of element, second data element is stored in the data field of the 5th node, and The mark of second data element is preserved in the index of the doubly linked list；Update the second number described in the 5th node According to the accessed number of times of element, the position of each node in the doubly linked list is adjusted according to accessed number of times, and accordingly adjusted Pointer in the doubly linked list in each node.
According to the embodiment of the present application, a kind of data query arrangement is also provided, applied in big data terminal, including：
Receiving unit, the data of the mark of the data element of carrying second sent for receive user or other big data terminals Obtain request；
Query unit, for described to be inquired about in the internal memory of the big data terminal according to the mark of second data element The storage location of two data elements or second data element；
Transmitting element, if for inquiring second data element, second data element is sent to into the user Or other big data terminals；If inquiring the storage location of second data element, according to second data element Storage location obtain second data element from the external memory storage of the big data terminal, and be sent to the user or Person other big data terminals；And, second data element is stored in the internal memory.
Specifically, the query unit, for described to be inquired about in the internal memory according to the mark of second data element The storage location of two data elements or second data element, specifically for：
In the doubly linked list with index for preserving from the internal memory respectively and in the corresponding relation of mark and storage location Search the mark of second data element；
If finding the mark of second data element in the doubly linked list, obtain described from the doubly linked list Corresponding 3rd node of mark of the second data element, and obtain second data element from the data field of the 3rd node Element；
If finding the list item of the mark including second data element in the corresponding relation, obtain in the list item Take the storage location of second data element.
Optionally, the transmitting element, is additionally operable to：
After the query unit inquires second data element, the second data element described in the 3rd node is updated The accessed number of times of element；And,
The position of each node in the doubly linked list is adjusted according to accessed number of times, and accordingly adjusts each in the doubly linked list Pointer in individual node.
Specifically, the transmitting element, for second data element to be stored in the internal memory, specifically for：
Whether the amount of storage for judging the internal memory reaches the given threshold；
If the amount of storage of the internal memory reaches the given threshold, it is determined that the accessed least number of times preserved in the internal memory 3rd data element, the 3rd data element is stored in the external memory storage, and is preserved in the corresponding relation The list item of mark and storage location including the 3rd data element；Delete in the index of the doubly linked list preserve it is described The node that the 3rd data element is located described in the mark of the 3rd data element and the doubly linked list；In the doubly linked list The 4th node corresponding with the mark of second data element is set up, second data element is stored in into described Section four In the data field of point, and the mark of second data element is preserved in the index of the doubly linked list；Update described The accessed number of times of the second data element described in four nodes, according to accessed number of times each node in the doubly linked list is adjusted Position, and accordingly adjust the pointer in the doubly linked list in each node；
If the amount of storage of the internal memory is not up to the given threshold, set up in the doubly linked list and second data Corresponding 5th node of mark of element, second data element is stored in the data field of the 5th node, and The mark of second data element is preserved in the index of the doubly linked list；Update the second number described in the 5th node According to the accessed number of times of element, the position of each node in the doubly linked list is adjusted according to accessed number of times, and accordingly adjusted Pointer in the doubly linked list in each node.
The embodiment of the present application provides a kind of data cache method, data query method and device, and reception treats data cached, for It is described treat it is data cached including each first data element, perform：Obtain the mark of first data element；Determine institute State the corresponding data with existing element of mark that first data element whether is preserved in the internal memory of big data terminal；If it is determined that institute State and preserve in internal memory the data with existing element, then the data with existing element is replaced with into first data element；If really The data with existing element is not preserved in the fixed internal memory, whether the amount of storage for determining the internal memory reaches given threshold；If institute The amount of storage for stating internal memory is not up to the given threshold, then first data element and its mark are stored in the internal memory； If the amount of storage of the internal memory reaches the given threshold, first data element is stored in into the big data terminal In external memory storage, and the mark and storage location of first data element are stored in the internal memory.In the program, It is not all to be stored in treating of pushing of push server or data base is data cached in internal memory, but using internal memory and outside The mode that combines of storage treats data cached to preserve, and the amount of storage of internal memory is controlled all the time within given threshold, such that it is able to The process for being prevented effectively from replacement data in correlation technique takes many memory sources, so as to easily cause low memory, or even meeting Memory recycle and big data terminal is caused to delay the problem of machine.
Description of the drawings
Accompanying drawing described herein is used for providing further understanding of the present application, constitutes the part of the application, the application's Schematic description and description does not constitute the improper restriction to the application for explaining the application.In the accompanying drawings：
Fig. 1 is a kind of flow chart of data cache method in the embodiment of the present application；
Fig. 2 is the flow chart of S12 in the embodiment of the present application；
Fig. 3 is the flow chart of S13 in the embodiment of the present application；
Fig. 4 is the structural representation of the doubly linked list with index in the embodiment of the present application；
Fig. 5 is a kind of flow chart of data query method in the embodiment of the present application；
Fig. 6 is a kind of structural representation of data buffer storage device in the embodiment of the present application；
Fig. 7 is the structural representation of another kind of data buffer storage device in the embodiment of the present application；
Fig. 8 is a kind of structural representation of data query arrangement in the embodiment of the present application.
Specific embodiment
In order that technical problems to be solved in this application, technical scheme and beneficial effect are clearer, clear, below in conjunction with Drawings and Examples, are further elaborated to the application.It should be appreciated that specific embodiment described herein is only To explain the application, it is not used to limit the application.
In order to the process for solving replacement data present in correlation technique takes many memory sources, so as to easily cause internal memory not Foot, or even Memory recycle and big data terminal can be caused to delay the problem of machine, the embodiment of the present application provides a kind of data cache method, The method can be, but not limited to apply in big data terminal, the big data terminal can be website service end in it is unique big Any one big data terminal in data terminal, or big data terminal cluster, the flow process of the method as shown in figure 1, Specifically include following steps：
S11：Reception treats data cached, for treat it is data cached including each first data element, perform S12.
Treat that data cached is probably that push server or data base push, push server and data base are introduced separately below Data cached situation is treated in push.
Push server can be pushed in following two situations and treat data cached：
The first situation, push server obtains full dose data and therefrom selects each big data with the period 1 from data base Data needed for terminal, obtain each big data terminal treats data cached, is then pushed to corresponding big data terminal, the One cycle can be set according to actual needs, be can be set as 1 day, 2 days etc., preferably select daily portfolio The fewer period is pushed, such as daily 0 point, 0: 30 grade.
Second situation, push server can also be obtained from data base with second round and updated the data and therefrom to select each big Data terminal is updated the data, and obtain each big data terminal treats data cached, be then pushed to corresponding big data terminal, Second round is less than the period 1, can be set according to actual needs, can be set as 1 hour, 2 hours, 3 hours Etc..Due to push server timing push updating the data for each big data terminal, such that it is able to guarantee big data terminal in What is preserved is newest, most full data, and then guarantees that big data terminal can reduce industry with real-time processing from the business of user Business processes time delay.
Data base can push in situations below and treat data cached, when big data terminal initialization, note to push server first Volume, then push server will select in the full dose data obtained from data base data-pushing needed for the big data terminal to The big data terminal, if push server is promoted with the first setting cycle to each big data terminal treats data cached, that is, Say push server push be the set time before data, in order to ensure getting newest data, big data terminal can With voluntarily, to database request current time to the data between the set time, data base pushes away the data of big data terminal request Give the big data terminal.For example, if push server in daily 0 point timing to big data terminal propelling data, and when current Quarter is 13 points, then the data between 0 point to 13 points of today need big data terminal to database request, and data base is by the present Data-pushing between it 0 point to 13 points gives big data terminal.
S12：Obtain the mark of the first data element.
S13：Determine the corresponding data with existing element of mark that the first data element whether is preserved in the internal memory of big data terminal, if Determine preservation data with existing element in internal memory, perform S14；If it is determined that not preserving data with existing element in internal memory, S15 is performed.
The data volume that one data element includes may be than larger, it is determined that whether preserving first in the internal memory of big data terminal During data element, the mark of the first data element can be obtained, it is then determined that whether preserving first in the internal memory of big data terminal The corresponding data with existing element of mark of data element, due to the mark data amount of the first data element it is very little, such that it is able to Greatly reduce the workload of big data terminal, lift data buffer storage efficiency.
S14：Data with existing element is replaced with into the first data element.
If in internal memory preserve data with existing element, in order to ensure in internal memory preserve be newest data element, can directly by Data with existing element replaces with the first data element.
S15：Whether the amount of storage for determining internal memory reaches given threshold, if the amount of storage of internal memory is not up to given threshold, performs S16； If the amount of storage of internal memory reaches given threshold, S17 is performed.
If not preserving data with existing element in internal memory, it needs to be determined that the first data element is stored in also being stored in internal memory In the external memory storage of big data terminal, specifically can determine whether internal memory also has memory space, that is, judge the amount of storage of internal memory Whether given threshold is reached, and given threshold can be set according to actual needs.
S16：First data element and its mark are stored in internal memory.
S17：First data element is stored in the external memory storage of big data terminal, and by the mark of the first data element and Storage location is stored in internal memory.
If the amount of storage of internal memory reaches given threshold, need that the first data element is stored in external memory storage.Wherein, The mark and storage location of the first data element can be stored in internal memory using key (Key)/value (Vlaue) in the form of, Key is the mark of the first data element, and Value is storage location of first data element in external memory storage.In a reality In applying mode, the storage location can be specifically the side-play amount of external memory storage.In another embodiment, the storage position It can be specifically discrete storage location in external memory storage to put.
It is not to be all stored in treating of pushing of push server or data base is data cached in internal memory in the program, but Preserve by the way of internal memory and external storage combine and treat data cached, the amount of storage of internal memory is controlled all the time in given threshold Within, such that it is able to be prevented effectively from correlation technique in the process of replacement data take many memory sources, so as in easily causing Deposit deficiency, or even Memory recycle and big data terminal can be caused to delay the problem of machine.
Each step in above-mentioned data cache method is described in detail below.
Specifically, the mark of the data element of acquisition one in above-mentioned S12 realizes process as shown in Fig. 2 specifically including as follows Step：
S121：Search from the first data element and select field.
S122：The numerical value for selecting field is obtained, the mark of the first data element is obtained.
The unique field for much distinguishing with other data elements is generally included in first data element, one can be therefrom selected Individual or multiple fields obtain selecting field, using the numerical value of the selected field as the first data element mark, this is designated the The globally unique indications of one data element, can include numeral, letter, character etc. form, it is also possible to including numeral, Letter, character etc. various forms of combinations.
Specifically, the mark that the first data element whether is preserved in the internal memory that big data terminal is determined in above-mentioned S13 is corresponding Have data element realizes process as shown in figure 3, specifically including following steps：
S131：Obtain the doubly linked list with index preserved in internal memory.
The storage organization of data in EMS memory element can take various forms, for example doubly linked list, data form etc., below Illustrate by taking the doubly linked list as shown in Figure 4 with index as an example, wherein：
Meter pointer (Head):First node pointer of chained list, can find first knot of chained list from this meter pointer Point；
Front node pointer (Pre)：A upper node of current node is pointed to, if not going up a node, shows current knot Point is first node；
Postjunction pointer (Next)：The next node of current node is pointed to, without next one node, shows current knot Point is exactly last node；
Data field (data)：The data entity of current node, can be business datum etc.；
Table caudal knot point (Tail)：Last node pointer of chained list, can from this table caudal knot point find chained list last Individual node.
The above-mentioned doubly linked list with index using Key/Vlaue pair in the form of, wherein, data element is designated Key, several The node being located according to element is Value.
S132：Determine and whether include in the index of doubly linked list the mark of the first data element, if wrapping in the index of doubly linked list The mark of the first data element is included, S133 is performed；If not holding including the mark of the first data element in the index of doubly linked list Row S134.
S133：Determine preservation data with existing element in internal memory.
S134：Determine and do not preserve in internal memory data with existing element.
Due in the doubly linked list with index the mark of data element be with the node at data element place it is one-to-one, Whether is preserved in therefore, it can whether be preserved in the index according to doubly linked list the mark of the first data element to determine internal memory One data element, determines the first data element whether is preserved in internal memory such that it is able to quick, improves data buffer storage efficiency.
Optionally, after it is determined that preserving data with existing element in internal memory, before performing S16, also include：
Obtain corresponding first node of mark of the first data element in doubly linked list；
Data with existing element is obtained from the data field of the first node；
Contrast the content that the first data element includes whether identical with the content that data with existing element includes；
If the content that the content that the first data element includes includes with data with existing element is incomplete same, S16 is performed.
If preserving data with existing element in internal memory, may further determine that whether data with existing element is identical with the first data element, If the first data element is different from data with existing element, that is to say, that content of first data element relative to data with existing element There is certain renewal, because mark is without change, it is only necessary to which data with existing element is replaced with into the first data element； If the first data element is identical with data with existing element, then can just ignore the first data element, continuing with next number Judged according to element.
Replace at one in embodiment, it is also possible to which the content that omission includes the first data element includes with data with existing element Content the step of be compared, directly data with existing element is replaced with into the first data element.That is, no matter data with existing unit How is the content of element, and it is newest value that the first acquired data element is given tacit consent to all the time, thus performs replacement operation.
Specifically, when the storage organization in internal memory is using the doubly linked list with index, by data with existing in above-mentioned S16 Element replaces with the first data element, specifically includes：Delete the data with existing element preserved in the data field of the first node； First data element is stored in the data field of the first node.
When replacing data with existing element, actually by the data word in corresponding first node of mark of the first data element The data element preserved in section replaces with the first data element.
Correspondingly, the first data element and its mark are stored in internal memory in above-mentioned S17, are specifically included：In doubly linked list Index in preserve the mark of the first data element；Corresponding with the mark of the first data element second is set up in doubly linked list Node, and the first data element is stored in the data field of secondary nodal point.
The first data element and its mark are preserved in internal memory, seeks to set up Key/Value in the doubly linked list with index Right, Key is the mark of the first data element, and Value is the second node for including the first data element.
Correspondingly, the first data element is stored in the external memory storage of big data terminal in above-mentioned S17, and by first The mark and storage location of data element is stored in internal memory, is specifically included：First data element is stored in into big data terminal External memory storage file in；Obtain storage location of first data element in the external memory storage of big data terminal；Including Deposit to be set up in the mark of preservation and the corresponding relation of storage location and include the mark of the first data element and the list item of storage location.
External memory storage can be set according to actual needs, preferably can be adopted with high digital independent/writing speed Mass-memory unit, such as, solid state hard disc (Solid State Drives, SSD), it would however also be possible to employ common great Rong Amount storage device.Data element in external memory storage can be preserved in one file, can also be deposited using others certainly Storage mode, repeats no more here.
Data cache method is described above, big data terminal can also be looked into after caching to data from the data of caching The data element of needs, the embodiment of the present application is looked for also to provide a kind of data query method, the method can be, but not limited to apply In big data terminal, the flow process of the method is as shown in figure 5, comprise the steps：
S51：The data acquisition request of the mark of the data element of carrying second that receive user or other big data terminals send.
When user or other big data terminals need to obtain data element, these data elements can be defined as the second data Element, can send the data acquisition request of the mark for carrying the second data element to big data terminal.
S52：Second data element or the second data are inquired about in the internal memory of big data terminal according to the mark of the second data element The storage location of element, if inquiring the second data element, performs S53；If inquiring the storage location of the second data element, Perform S54.
Due to the storage location for preserving data element or data element in the internal memory of big data terminal, and the inquiry in internal memory Speed is very fast, therefore, it can directly search the storage location of the second data element or the second data element in internal memory.
Specifically, can respectively from internal memory preserve with index doubly linked list in and identify it is right with storage location The mark of the second data element of middle lookup should be related to；If finding the mark of the second data element in doubly linked list, from double Corresponding 3rd node of mark of the second data element is obtained in chained list, and second is obtained from the data field of the 3rd node and counted According to element；If finding the list item of the mark including the second data element in corresponding relation, second is obtained in the table entry and is counted According to the storage location of element.
S53：Second data element is sent to into user or other big data terminals.
S54：Second data element is obtained from the external memory storage of big data terminal according to the storage location of the second data element, and It is sent to user or other big data terminals；And, the second data element is stored in internal memory.
In the program, because the data element in big data terminal is by the way of internal memory adds external storage to combine, and preserve The mark and storage location of the data element in external memory storage is stored in internal memory, correspondingly, is inquiring about the second data element When plain, if the second data element is stored in internal memory, can be with the data element of direct access second, if the second data element is preserved In external memory storage, the storage location of the second data element can be inquired in internal memory, therefore, even if the second data element Element is stored in external memory storage, it is also possible to the storage location of the data element of quick obtaining second, and then obtains the second data element Element, so as to reduce the time delay of data query, it is ensured that the real-time of data query, improves Consumer's Experience.
Each step of above-mentioned data query method is described in detail below.
Optionally, after inquiring the second data element, also include：
Update the accessed number of times of the second data element in the 3rd node；And,
The position of each node in doubly linked list is adjusted according to accessed number of times, and accordingly in adjustment doubly linked list in each node Pointer.
If the storage organization of internal memory is the doubly linked list of tape index, enumerator can be increased in each node of doubly linked list (counter) pointer, for recording the accessed number of times for corresponding to data element in node.Replace at one in embodiment, If the node in doubly linked list is sorted according to the accessed number of times size of data element, the second number in the 3rd node is updated After according to the accessed number of times of element, in addition it is also necessary to adjust the position of the second data element.I.e., for example, the second data element For A, two in doubly linked list data element B and C, then after the accessed number of times of A has been updated, from the node that A is located Node of the accessed number of times of beginning Look-ahead more than A, it is assumed that find the accessed number of times of B more than A, C's is accessed secondary Number is less than A, then after the node that is located to B of knot refinement A being located, before the node at C places, and adjust A, B, Pointer in the node of C places.
Optionally, after the second data element is stored in internal memory in above-mentioned S54, in order to avoid repeating to preserve, can be with The second data element preserved in the storage location and external memory storage of deleting the second data element preserved in internal memory, concrete mistake Cheng Wei：Deleting from corresponding relation includes the mark of the second data element and its list item of storage location；And, deposit from outside The second data element is deleted in reservoir.
Specifically, the second data element is stored in internal memory in above-mentioned S54 realizes process, specifically includes following steps：
Whether the first step, the amount of storage for judging internal memory reaches given threshold, if the amount of storage of internal memory reaches given threshold, performs Second step；If the amount of storage of internal memory is not up to given threshold, the 3rd step is performed.
Second step, determines the 3rd data element of the accessed least number of times preserved in internal memory, and the 3rd data element is stored in In external memory storage, and preservation includes the mark of the 3rd data element and the list item of storage location in corresponding relation；Delete double The node that the 3rd data element is located in the mark and doubly linked list of the 3rd data element preserved in the index of chained list；Double The 4th node corresponding with the mark of the second data element is set up in chained list, the second data element is stored in into fourth node In data field, and the mark of the second data element is preserved in the index of doubly linked list；Update the second data in the 4th node The accessed number of times of element, according to accessed number of times the position of each node in doubly linked list is adjusted, and accordingly adjusts Two-way Chain Pointer in table in each node.
Due to the accessed least number of times of the 3rd data element, the probability very little that the 3rd data element is reused is illustrated, Therefore, it can save it in external memory storage, leave the space of internal memory for accessed data element often.One The second data element is A in individual specific embodiment, if the amount of storage of internal memory reaches given threshold, is deleting the 3rd data element Afterwards, then from the afterbody Look-ahead of doubly linked list, once finding the accessed number of times of data element B more than 1, then it is located in B Node after set up and include the node of A, if there is no B, then will include the node of A in the foundation of the afterbody of doubly linked list, The pointer in the 4th node neighborhood of nodes that A is located is adjusted simultaneously, and updates the accessed number of times in the 4th node for 1.
3rd step, sets up the 5th node corresponding with the mark of the second data element, by the second data element in doubly linked list In being stored in the data field of the 5th node, and the mark of the second data element is preserved in the index of doubly linked list；Update the The accessed number of times of the second data element in five nodes, according to accessed number of times the position of each node in doubly linked list is adjusted, And accordingly adjust the pointer in doubly linked list in each node.
In a detailed embodiment, the second data element is A, if the amount of storage of internal memory is not up to given threshold, directly The 5th node for including A is set up in the afterbody of doubly linked list, while adjusting the pointer in the 5th node, and the 5th node is updated In accessed number of times be 1.
It should be noted that above data cache method and data query method are described respectively, but for big data terminal For, both can respectively perform above-mentioned data cache method and data query method, it is also possible to while performing above-mentioned data buffer storage Method and data query method.
Based on same inventive concept, the embodiment of the present application also provides a kind of data buffer storage device, the device with it is as shown in Figure 1 Data cache method correspondence, the structure of the device as shown in 6 figures, including
First receiving unit 61, for receive treat data cached, for treat it is data cached including each first data element, Perform：
Acquiring unit 62, for obtaining the mark of the first data element；
Determining unit 63, the mark for whether preserving the first data element in the internal memory for determining big data terminal is corresponding existing Data element；
Buffer unit 64, for if it is determined that in internal memory preserve data with existing element, then data with existing element is replaced with into the first number According to element；If it is determined that not preserving data with existing element in internal memory, whether the amount of storage for determining internal memory reaches given threshold；If interior The amount of storage deposited is not up to given threshold, then the first data element and its mark are stored in internal memory；If the amount of storage of internal memory Given threshold is reached, then the first data element is stored in the external memory storage of big data terminal, and by the first data element Mark and storage location be stored in internal memory.
It is not to be all stored in treating of pushing of push server or data base is data cached in internal memory in the program, but Preserve by the way of internal memory and external storage combine and treat data cached, the amount of storage of internal memory is controlled all the time in given threshold Within, such that it is able to be prevented effectively from correlation technique in the process of replacement data take many memory sources, so as in easily causing Deposit deficiency, or even Memory recycle and big data terminal can be caused to delay the problem of machine.
Specifically, acquiring unit 62, for obtaining the mark of the first data element, specifically for：
Search from the first data element and select field；
The numerical value for selecting field is obtained, the mark of the first data element is obtained.
Specifically, determining unit 63, the mark for whether preserving the first data element in the internal memory for determining big data terminal is right The data with existing element answered, specifically includes：
Obtain the doubly linked list with index preserved in internal memory；
Determine whether include the mark of the first data element in the index of doubly linked list；
If the index of doubly linked list includes the mark of the first data element, it is determined that preserve data with existing element in internal memory；
If not including the mark of the first data element in the index of doubly linked list, it is determined that do not preserve data with existing element in internal memory.
Optionally, determining unit 63, are additionally operable to：
Determine and the mark corresponding first that the first data element in doubly linked list is obtained after data with existing element is preserved in internal memory Node；
Data with existing element is obtained from the data field of the first node；
Contrast the content that the first data element includes whether identical with the content that data with existing element includes；
If the content that the content that the first data element includes includes with data with existing element is incomplete same, buffer unit is turned to.
Specifically, buffer unit 64, for the first data element and its mark to be stored in internal memory, specifically for：
The mark of the first data element is preserved in the index of doubly linked list；
The second node corresponding with the mark of the first data element is set up in doubly linked list, and the first data element is stored in In the data field of secondary nodal point.
Specifically, buffer unit 64, for the first data element to be stored in the external memory storage of big data terminal, and will The mark and storage location of the first data element is stored in internal memory, specifically for：
First data element is stored in the file of the external memory storage of big data terminal；
Obtain storage location of first data element in the external memory storage of big data terminal；
The mark and storage location for including the first data element is set up in corresponding relation of the mark that internal memory is preserved with storage location List item.
Also provide another kind of data buffer storage device according to the embodiment of the present application, the structure of the device as shown in fig. 7, the device with Device identical unit as shown in Figure 6 omits not table, on the basis of device as shown in Figure 6, also includes：
Second receiving unit 65, the mark of the data element of carrying second sent for receive user or other big data terminals Data acquisition request；
Query unit 66, for the second data element or the second data to be inquired about in internal memory according to the mark of the second data element The storage location of element；
Transmitting element 67, if for inquiring the second data element, the second data element being sent to into user or other being big Data terminal；If inquiring the storage location of the second data element, deposited from outside according to the storage location of the second data element Reservoir obtains the second data element, and is sent to user or other big data terminals；And, the second data element is preserved In internal memory.
Specifically, query unit 66, for according to the mark of the second data element inquire about in internal memory the second data element or The storage location of the second data element, specifically for：
Search in the doubly linked list with index for preserving from internal memory respectively and in the corresponding relation of mark and storage location The mark of the second data element；
If finding the mark of the second data element in doubly linked list, the mark of the second data element is obtained from doubly linked list Know corresponding 3rd node, and the second data element is obtained from the data field of the 3rd node；
If finding the list item of the mark including the second data element in corresponding relation, the second data element is obtained in the table entry The storage location of element.
Optionally, transmitting element 67, are additionally operable to：
After query unit 66 inquires the second data element, the accessed number of times of the second data element in the 3rd node is updated； And,
The position of each node in doubly linked list is adjusted according to accessed number of times, and accordingly in adjustment doubly linked list in each node Pointer.
Specifically, transmitting element 67, for the second data element to be stored in internal memory, specifically for：
Whether the amount of storage for judging internal memory reaches given threshold；
If the amount of storage of internal memory reaches given threshold, it is determined that the 3rd data element of the accessed least number of times preserved in internal memory, 3rd data element is stored in external memory storage, and preservation includes the mark of the 3rd data element and deposits in corresponding relation The list item that storage space is put；Delete the 3rd data in the mark and doubly linked list of the 3rd data element preserved in the index of doubly linked list The node that element is located；The 4th node corresponding with the mark of the second data element is set up in doubly linked list, by the second data Element is stored in the data field of fourth node, and the mark of the second data element is preserved in the index of doubly linked list；More The accessed number of times of the second data element in new 4th node, according to accessed number of times the position of each node in doubly linked list is adjusted Put, and accordingly adjust the pointer in doubly linked list in each node；
If the amount of storage of internal memory is not up to given threshold, set up corresponding with the mark of the second data element in doubly linked list 5th node, the second data element is stored in the data field of the 5th node, and preserves in the index of doubly linked list The mark of two data elements；The accessed number of times of the second data element in the 5th node is updated, it is double according to the adjustment of accessed number of times The position of each node in chained list, and accordingly adjust the pointer in doubly linked list in each node.
Based on same inventive concept, the embodiment of the present application also provides a kind of data query arrangement, the device with it is as shown in Figure 5 Data query method correspondence, the structure of the device as shown in figure 8, including
Receiving unit 81, the number of the mark of the data element of carrying second sent for receive user or other big data terminals Ask according to obtaining；
Query unit 82, for the second data element to be inquired about in the internal memory of big data terminal according to the mark of the second data element Or second data element storage location；
Transmitting element 83, if for inquiring the second data element, the second data element being sent to into user or other being big Data terminal；If inquiring the storage location of the second data element, according to the storage location of the second data element from big data The external memory storage of terminal obtains the second data element, and is sent to user or other big data terminals；And, by second Data element is stored in internal memory.
In the program, because the data element in big data terminal is by the way of internal memory adds external storage to combine, and preserve The mark and storage location of the data element in external memory storage is stored in internal memory, correspondingly, is inquiring about the second data element When plain, if the second data element is stored in internal memory, can be with the data element of direct access second, if the second data element is preserved In external memory storage, the storage location of the second data element can be inquired in internal memory, therefore, even if the second data element Element is stored in external memory storage, it is also possible to the storage location of the data element of quick obtaining second, and then obtains the second data element Element, so as to reduce the time delay of data query, it is ensured that the real-time of data query, improves Consumer's Experience.
Specifically, query unit 82, for according to the mark of the second data element inquire about in internal memory the second data element or The storage location of the second data element, specifically for：
Search in the doubly linked list with index for preserving from internal memory respectively and in the corresponding relation of mark and storage location The mark of the second data element；
If finding the mark of the second data element in doubly linked list, the mark of the second data element is obtained from doubly linked list Know corresponding 3rd node, and the second data element is obtained from the data field of the 3rd node；
If finding the list item of the mark including the second data element in corresponding relation, the second data element is obtained in the table entry The storage location of element.
Optionally, transmitting element 83, are additionally operable to：
After query unit 82 inquires the second data element, the accessed number of times of the second data element in the 3rd node is updated； And,
The position of each node in doubly linked list is adjusted according to accessed number of times, and accordingly in adjustment doubly linked list in each node Pointer.
Specifically, transmitting element 83, for the second data element to be stored in internal memory, specifically for：
Whether the amount of storage for judging internal memory reaches given threshold；
If the amount of storage of internal memory reaches given threshold, it is determined that the 3rd data element of the accessed least number of times preserved in internal memory, 3rd data element is stored in external memory storage, and preservation includes the mark of the 3rd data element and deposits in corresponding relation The list item that storage space is put；Delete the 3rd data in the mark and doubly linked list of the 3rd data element preserved in the index of doubly linked list The node that element is located；The 4th node corresponding with the mark of the second data element is set up in doubly linked list, by the second data Element is stored in the data field of fourth node, and the mark of the second data element is preserved in the index of doubly linked list；More The accessed number of times of the second data element in new 4th node, according to accessed number of times the position of each node in doubly linked list is adjusted Put, and accordingly adjust the pointer in doubly linked list in each node；
If the amount of storage of internal memory is not up to given threshold, set up corresponding with the mark of the second data element in doubly linked list 5th node, the second data element is stored in the data field of the 5th node, and preserves in the index of doubly linked list The mark of two data elements；The accessed number of times of the second data element in the 5th node is updated, it is double according to the adjustment of accessed number of times The position of each node in chained list, and accordingly adjust the pointer in doubly linked list in each node.
Described above illustrates and describes the preferred embodiment of the application, but as previously mentioned, it should be understood that the application not limits to In form disclosed herein, be not to be taken as the exclusion to other embodiment, and can be used for various other combinations, modification and Environment, and can be changed by the technology or knowledge of above-mentioned teaching or association area in invention contemplated scope described herein It is dynamic.And change that those skilled in the art are carried out and change be without departing from spirit and scope, then all should be appended by the application In scope of the claims.

Claims (28)

1. a kind of data cache method, applies in big data terminal, it is characterised in that include：
Reception treats data cached, for it is described treat it is data cached including each first data element, perform：
Obtain the mark of first data element；
Whether the mark corresponding data with existing unit of first data element is preserved in the internal memory for determining the big data terminal Element；
If it is determined that preserving the data with existing element in the internal memory, then the data with existing element is replaced with into first number According to element；
If it is determined that not preserving the data with existing element in the internal memory, whether the amount of storage for determining the internal memory reaches setting threshold Value；If the amount of storage of the internal memory is not up to the given threshold, first data element and its mark are stored in into institute In stating internal memory；If the amount of storage of the internal memory reaches the given threshold, first data element is stored in described big In the external memory storage of data terminal, and the mark and storage location of first data element are stored in the internal memory.
2. the method for claim 1, it is characterised in that obtain the mark of first data element, specifically include：
Search from first data element and select field；
The numerical value of the selected field is obtained, the mark of first data element is obtained.
3. the method for claim 1, it is characterised in that whether preserve institute in the internal memory for determining the big data terminal The corresponding data with existing element of mark of the first data element is stated, is specifically included：
Obtain the doubly linked list with index preserved in the internal memory；
Determine whether include the mark of first data element in the index of the doubly linked list；
If the index of the doubly linked list includes the mark of first data element, it is determined that preserve in the internal memory described Data with existing element；
If not including the mark of first data element in the index of the doubly linked list, it is determined that do not preserve in the internal memory The data with existing element.
4. method as claimed in claim 3, it is characterised in that determine preserve in the internal memory data with existing element it Afterwards, also include：
Obtain corresponding first node of mark of the first data element described in the doubly linked list；
The data with existing element is obtained from the data field of first node；
Contrast the content that first data element includes whether identical with the content that the data with existing element includes；
If the content that the content that first data element includes includes with the data with existing element is incomplete same, perform The step of data with existing element is replaced with into first data element.
5. method as claimed in claim 4, it is characterised in that be stored in first data element and its mark described In internal memory, specifically include：
The mark of first data element is preserved in the index of the doubly linked list；
The second node corresponding with the mark of first data element is set up in the doubly linked list, and described first is counted It is stored in the data field of the secondary nodal point according to element.
6. method as claimed in claim 4, it is characterised in that first data element is stored in into the big data end In the external memory storage at end, and the mark and storage location of first data element are stored in the internal memory, concrete bag Include：
First data element is stored in the file of the external memory storage of the big data terminal；
Obtain storage location of first data element in the external memory storage of the big data terminal；
Set up in the corresponding relation of mark and storage location that the internal memory is preserved include the mark of first data element and The list item of storage location.
7. the method as described in claim 1-6 is arbitrary, it is characterised in that also include：
The data acquisition request of the mark of the data element of carrying second that receive user or other big data terminals send；
Second data element or second data are inquired about in the internal memory according to the mark of second data element The storage location of element；
If inquiring second data element, second data element is sent to into the user or other big datas Terminal；
If inquiring the storage location of second data element, according to the storage location of second data element from described External memory storage obtains second data element, and is sent to the user or other big data terminals；And, by institute State the second data element to be stored in the internal memory.
8. method as claimed in claim 7, it is characterised in that according to the mark of second data element in the internal memory The middle storage location for inquiring about second data element or second data element, specifically includes：
In the doubly linked list with index for preserving from the internal memory respectively and in the corresponding relation of mark and storage location Search the mark of second data element；
If finding the mark of second data element in the doubly linked list, obtain described from the doubly linked list Corresponding 3rd node of mark of the second data element, and obtain second data element from the data field of the 3rd node Element；
If finding the list item of the mark including second data element in the corresponding relation, obtain in the list item Take the storage location of second data element.
9. method as claimed in claim 8, it is characterised in that after inquiring second data element, also include：
Update the accessed number of times of the second data element described in the 3rd node；And,
The position of each node in the doubly linked list is adjusted according to accessed number of times, and accordingly adjusts each in the doubly linked list Pointer in individual node.
10. method as claimed in claim 8, it is characterised in that second data element is stored in the internal memory, Specifically include：
Whether the amount of storage for judging the internal memory reaches the given threshold；
If the amount of storage of the internal memory reaches the given threshold, it is determined that the accessed least number of times preserved in the internal memory 3rd data element, the 3rd data element is stored in the external memory storage, and is preserved in the corresponding relation The list item of mark and storage location including the 3rd data element；Delete in the index of the doubly linked list preserve it is described The node that the 3rd data element is located described in the mark of the 3rd data element and the doubly linked list；In the doubly linked list The 4th node corresponding with the mark of second data element is set up, second data element is stored in into described Section four In the data field of point, and the mark of second data element is preserved in the index of the doubly linked list；Update described The accessed number of times of the second data element described in four nodes, according to accessed number of times each node in the doubly linked list is adjusted Position, and accordingly adjust the pointer in the doubly linked list in each node；
If the amount of storage of the internal memory is not up to the given threshold, set up in the doubly linked list and second data Corresponding 5th node of mark of element, second data element is stored in the data field of the 5th node, and The mark of second data element is preserved in the index of the doubly linked list；Update the second number described in the 5th node According to the accessed number of times of element, the position of each node in the doubly linked list is adjusted according to accessed number of times, and accordingly adjusted Pointer in the doubly linked list in each node.
A kind of 11. data query methods, apply in big data terminal, it is characterised in that include：
The data acquisition request of the mark of the data element of carrying second that receive user or other big data terminals send；
According to the mark of second data element inquire about in the internal memory of the big data terminal second data element or The storage location of second data element；
If inquiring second data element, second data element is sent to into the user or other big datas Terminal；
If inquiring the storage location of second data element, according to the storage location of second data element from described The external memory storage of big data terminal obtains second data element, and is sent to the user or other big data terminals； And, second data element is stored in the internal memory.
12. methods as claimed in claim 11, it is characterised in that according to the mark of second data element described interior The storage location of middle inquiry second data element or second data element is deposited, is specifically included：
In the doubly linked list with index for preserving from the internal memory respectively and in the corresponding relation of mark and storage location Search the mark of second data element；
If finding the mark of second data element in the doubly linked list, obtain described from the doubly linked list Corresponding 3rd node of mark of the second data element, and obtain second data element from the data field of the 3rd node Element；
If finding the list item of the mark including second data element in the corresponding relation, obtain in the list item Take the storage location of second data element.
13. methods as claimed in claim 11, it is characterised in that after inquiring second data element, also include：
Update the accessed number of times of the second data element described in the 3rd node；And,
The position of each node in the doubly linked list is adjusted according to accessed number of times, and accordingly adjusts each in the doubly linked list Pointer in individual node.
14. methods as described in claim 11-13 is arbitrary, it is characterised in that second data element is stored in into institute In stating internal memory, specifically include：
Whether the amount of storage for judging the internal memory reaches the given threshold；
If the amount of storage of the internal memory reaches the given threshold, it is determined that the accessed least number of times preserved in the internal memory 3rd data element, the 3rd data element is stored in the external memory storage, and is preserved in the corresponding relation The list item of mark and storage location including the 3rd data element；Delete in the index of the doubly linked list preserve it is described The node that the 3rd data element is located described in the mark of the 3rd data element and the doubly linked list；In the doubly linked list The 4th node corresponding with the mark of second data element is set up, second data element is stored in into described Section four In the data field of point, and the mark of second data element is preserved in the index of the doubly linked list；Update described The accessed number of times of the second data element described in four nodes, according to accessed number of times each node in the doubly linked list is adjusted Position, and accordingly adjust the pointer in the doubly linked list in each node；
If the amount of storage of the internal memory is not up to the given threshold, set up in the doubly linked list and second data Corresponding 5th node of mark of element, second data element is stored in the data field of the 5th node, and The mark of second data element is preserved in the index of the doubly linked list；Update the second number described in the 5th node According to the accessed number of times of element, the position of each node in the doubly linked list is adjusted according to accessed number of times, and accordingly adjusted Pointer in the doubly linked list in each node.
A kind of 15. data buffer storage devices, apply in big data terminal, it is characterised in that include：
First receiving unit, for receive treat data cached, for it is described treat it is data cached including each first data element, Perform：
Acquiring unit, for obtaining the mark of first data element；
Determining unit, for whether preserving the mark correspondence of first data element in the internal memory for determining the big data terminal Data with existing element；
Buffer unit, for if it is determined that preserve the data with existing element in the internal memory, then replacing the data with existing element It is changed to first data element；If it is determined that not preserving the data with existing element in the internal memory, depositing for the internal memory is determined Whether reserves reach given threshold；If the amount of storage of the internal memory is not up to the given threshold, by first data element Element and its mark are stored in the internal memory；If the amount of storage of the internal memory reaches the given threshold, by the described first number It is stored in the external memory storage of the big data terminal according to element, and by the mark and storage location of first data element In being stored in the internal memory.
16. devices as claimed in claim 15, it is characterised in that the acquiring unit, for obtaining first data The mark of element, specifically for：
Search from first data element and select field；
The numerical value of the selected field is obtained, the mark of first data element is obtained.
17. devices as claimed in claim 15, it is characterised in that the determining unit, for determining the big data end The corresponding data with existing element of mark of first data element whether is preserved in the internal memory at end, is specifically included：
Obtain the doubly linked list with index preserved in the internal memory；
Determine whether include the mark of first data element in the index of the doubly linked list；
If the index of the doubly linked list includes the mark of first data element, it is determined that preserve in the internal memory described Data with existing element；
If not including the mark of first data element in the index of the doubly linked list, it is determined that do not preserve in the internal memory The data with existing element.
18. devices as claimed in claim 17, it is characterised in that the determining unit, are additionally operable to：
Determine and preserve in the internal memory after the data with existing element, obtain the first data element described in the doubly linked list Corresponding first node of mark；
The data with existing element is obtained from the data field of first node；
Contrast the content that first data element includes whether identical with the content that the data with existing element includes；
If the content that the content that first data element includes includes with the data with existing element is incomplete same, turn to The buffer unit.
19. devices as claimed in claim 18, it is characterised in that the buffer unit, for by first data element Element and its mark are stored in the internal memory, specifically for：
The mark of first data element is preserved in the index of the doubly linked list；
The second node corresponding with the mark of first data element is set up in the doubly linked list, and described first is counted It is stored in the data field of the secondary nodal point according to element.
20. devices as claimed in claim 18, it is characterised in that the buffer unit, for by first data element Element is stored in the external memory storage of the big data terminal, and the mark of first data element and storage location are preserved In the internal memory, specifically for：
First data element is stored in the file of the external memory storage of the big data terminal；
Obtain storage location of first data element in the external memory storage of the big data terminal；
Set up in the corresponding relation of mark and storage location that the internal memory is preserved include the mark of first data element and The list item of storage location.
21. devices as described in claim 15-20 is arbitrary, it is characterised in that also include：
Second receiving unit, the mark of the data element of carrying second sent for receive user or other big data terminals Data acquisition request；
Query unit, for according to the mark of second data element inquire about in the internal memory second data element or The storage location of the second data element described in person；
Transmitting element, if for inquiring second data element, second data element is sent to into the user Or other big data terminals；If inquiring the storage location of second data element, according to second data element Storage location obtain second data element from the external memory storage, and be sent to the user or other big datas Terminal；And, second data element is stored in the internal memory.
22. devices as claimed in claim 21, it is characterised in that the query unit, for according to second data The mark of element inquires about the storage location of second data element or second data element in the internal memory, specifically For：
In the doubly linked list with index for preserving from the internal memory respectively and in the corresponding relation of mark and storage location Search the mark of second data element；
If finding the mark of second data element in the doubly linked list, obtain described from the doubly linked list Corresponding 3rd node of mark of the second data element, and obtain second data element from the data field of the 3rd node Element；
If finding the list item of the mark including second data element in the corresponding relation, obtain in the list item Take the storage location of second data element.
23. devices as claimed in claim 22, it is characterised in that the transmitting element, are additionally operable to：
After the query unit inquires second data element, the second data element described in the 3rd node is updated The accessed number of times of element；And,
The position of each node in the doubly linked list is adjusted according to accessed number of times, and accordingly adjusts each in the doubly linked list Pointer in individual node.
24. devices as claimed in claim 22, it is characterised in that the transmitting element, for by second data element Element is stored in the internal memory, specifically for：
Whether the amount of storage for judging the internal memory reaches the given threshold；
If the amount of storage of the internal memory reaches the given threshold, it is determined that the accessed least number of times preserved in the internal memory 3rd data element, the 3rd data element is stored in the external memory storage, and is preserved in the corresponding relation The list item of mark and storage location including the 3rd data element；Delete in the index of the doubly linked list preserve it is described The node that the 3rd data element is located described in the mark of the 3rd data element and the doubly linked list；In the doubly linked list The 4th node corresponding with the mark of second data element is set up, second data element is stored in into described Section four In the data field of point, and the mark of second data element is preserved in the index of the doubly linked list；Update described The accessed number of times of the second data element described in four nodes, according to accessed number of times each node in the doubly linked list is adjusted Position, and accordingly adjust the pointer in the doubly linked list in each node；
If the amount of storage of the internal memory is not up to the given threshold, set up in the doubly linked list and second data Corresponding 5th node of mark of element, second data element is stored in the data field of the 5th node, and The mark of second data element is preserved in the index of the doubly linked list；Update the second number described in the 5th node According to the accessed number of times of element, the position of each node in the doubly linked list is adjusted according to accessed number of times, and accordingly adjusted Pointer in the doubly linked list in each node.
A kind of 25. data query arrangements, apply in big data terminal, it is characterised in that include：
Receiving unit, the data of the mark of the data element of carrying second sent for receive user or other big data terminals Obtain request；
Query unit, for described to be inquired about in the internal memory of the big data terminal according to the mark of second data element The storage location of two data elements or second data element；
Transmitting element, if for inquiring second data element, second data element is sent to into the user Or other big data terminals；If inquiring the storage location of second data element, according to second data element Storage location obtain second data element from the external memory storage of the big data terminal, and be sent to the user or Person other big data terminals；And, second data element is stored in the internal memory.
26. devices as claimed in claim 25, it is characterised in that the query unit, for according to second data The mark of element inquires about the storage location of second data element or second data element in the internal memory, specifically For：
In the doubly linked list with index for preserving from the internal memory respectively and in the corresponding relation of mark and storage location Search the mark of second data element；
If finding the mark of second data element in the doubly linked list, obtain described from the doubly linked list Corresponding 3rd node of mark of the second data element, and obtain second data element from the data field of the 3rd node Element；
If finding the list item of the mark including second data element in the corresponding relation, obtain in the list item Take the storage location of second data element.
27. devices as claimed in claim 25, it is characterised in that the transmitting element, are additionally operable to：
After the query unit inquires second data element, the second data element described in the 3rd node is updated The accessed number of times of element；And,
The position of each node in the doubly linked list is adjusted according to accessed number of times, and accordingly adjusts each in the doubly linked list Pointer in individual node.
28. devices as described in claim 25-27 is arbitrary, it is characterised in that the transmitting element, for by described Two data elements are stored in the internal memory, specifically for：
Whether the amount of storage for judging the internal memory reaches the given threshold；
If the amount of storage of the internal memory reaches the given threshold, it is determined that the accessed least number of times preserved in the internal memory 3rd data element, the 3rd data element is stored in the external memory storage, and is preserved in the corresponding relation The list item of mark and storage location including the 3rd data element；Delete in the index of the doubly linked list preserve it is described The node that the 3rd data element is located described in the mark of the 3rd data element and the doubly linked list；In the doubly linked list The 4th node corresponding with the mark of second data element is set up, second data element is stored in into described Section four In the data field of point, and the mark of second data element is preserved in the index of the doubly linked list；Update described The accessed number of times of the second data element described in four nodes, according to accessed number of times each node in the doubly linked list is adjusted Position, and accordingly adjust the pointer in the doubly linked list in each node；
If the amount of storage of the internal memory is not up to the given threshold, set up in the doubly linked list and second data Corresponding 5th node of mark of element, second data element is stored in the data field of the 5th node, and The mark of second data element is preserved in the index of the doubly linked list；Update the second number described in the 5th node According to the accessed number of times of element, the position of each node in the doubly linked list is adjusted according to accessed number of times, and accordingly adjusted Pointer in the doubly linked list in each node.
CN201510746534.3A 2015-11-05 2015-11-05 Data caching method, data query method and device Active CN106681995B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201510746534.3A CN106681995B (en) 2015-11-05 2015-11-05 Data caching method, data query method and device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201510746534.3A CN106681995B (en) 2015-11-05 2015-11-05 Data caching method, data query method and device

Publications (2)

Publication Number Publication Date
CN106681995A true CN106681995A (en) 2017-05-17
CN106681995B CN106681995B (en) 2020-08-18

Family

ID=58857483

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201510746534.3A Active CN106681995B (en) 2015-11-05 2015-11-05 Data caching method, data query method and device

Country Status (1)

Country Link
CN (1) CN106681995B (en)

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108647280A (en) * 2018-05-03 2018-10-12 北京云中融信网络科技有限公司 A kind of method and apparatus of memory communicating information
CN109977122A (en) * 2019-04-01 2019-07-05 西安电子科技大学 Table objects search method, device, system, computer equipment and storage medium
CN109977033A (en) * 2017-12-28 2019-07-05 北京顺智信科技有限公司 Store the method and device of telecom number
CN111209304A (en) * 2019-12-30 2020-05-29 华为技术有限公司 Data processing method, device and system
CN113326141A (en) * 2021-07-19 2021-08-31 广东卓启云链科技有限公司 Memory management method and device for block chain nodes
CN115412893A (en) * 2022-10-19 2022-11-29 成都锐成芯微科技股份有限公司 Low-power-consumption Bluetooth attribute access method and low-power-consumption Bluetooth system

Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5241672A (en) * 1991-04-01 1993-08-31 Xerox Corporation System using the storage level of file updates in nonvolatile memory to trigger saving of RAM to disk and using the file updates to reboot after crash
CN101169628A (en) * 2007-11-14 2008-04-30 中控科技集团有限公司 Data storage method and device
US20110138299A1 (en) * 2009-12-04 2011-06-09 Nokia Corporation Method and apparatus for suggesting data for transfer
CN103106153A (en) * 2013-02-20 2013-05-15 哈尔滨工业大学 Web cache replacement method based on access density
CN103345469A (en) * 2013-05-24 2013-10-09 联动优势科技有限公司 Storing and inquiring method and device of number sets
CN103888377A (en) * 2014-03-28 2014-06-25 华为技术有限公司 Message cache method and device
CN104077282A (en) * 2013-03-25 2014-10-01 华为技术有限公司 Method and device for processing data
CN104238963A (en) * 2014-09-30 2014-12-24 华为技术有限公司 Data storage method, device and system
CN104346345A (en) * 2013-07-24 2015-02-11 中兴通讯股份有限公司 Data storage method and device
CN104380261A (en) * 2012-05-16 2015-02-25 苹果公司 Locally backed cloud-based storage

Patent Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5241672A (en) * 1991-04-01 1993-08-31 Xerox Corporation System using the storage level of file updates in nonvolatile memory to trigger saving of RAM to disk and using the file updates to reboot after crash
CN101169628A (en) * 2007-11-14 2008-04-30 中控科技集团有限公司 Data storage method and device
US20110138299A1 (en) * 2009-12-04 2011-06-09 Nokia Corporation Method and apparatus for suggesting data for transfer
CN104380261A (en) * 2012-05-16 2015-02-25 苹果公司 Locally backed cloud-based storage
CN103106153A (en) * 2013-02-20 2013-05-15 哈尔滨工业大学 Web cache replacement method based on access density
CN104077282A (en) * 2013-03-25 2014-10-01 华为技术有限公司 Method and device for processing data
CN103345469A (en) * 2013-05-24 2013-10-09 联动优势科技有限公司 Storing and inquiring method and device of number sets
CN104346345A (en) * 2013-07-24 2015-02-11 中兴通讯股份有限公司 Data storage method and device
CN103888377A (en) * 2014-03-28 2014-06-25 华为技术有限公司 Message cache method and device
CN104238963A (en) * 2014-09-30 2014-12-24 华为技术有限公司 Data storage method, device and system

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
谢柏青 等: "《算法与数据结构》", 31 July 2001, 高等教育出版社 *

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109977033A (en) * 2017-12-28 2019-07-05 北京顺智信科技有限公司 Store the method and device of telecom number
CN108647280A (en) * 2018-05-03 2018-10-12 北京云中融信网络科技有限公司 A kind of method and apparatus of memory communicating information
CN109977122A (en) * 2019-04-01 2019-07-05 西安电子科技大学 Table objects search method, device, system, computer equipment and storage medium
CN111209304A (en) * 2019-12-30 2020-05-29 华为技术有限公司 Data processing method, device and system
CN113326141A (en) * 2021-07-19 2021-08-31 广东卓启云链科技有限公司 Memory management method and device for block chain nodes
CN115412893A (en) * 2022-10-19 2022-11-29 成都锐成芯微科技股份有限公司 Low-power-consumption Bluetooth attribute access method and low-power-consumption Bluetooth system
CN115412893B (en) * 2022-10-19 2023-04-21 成都锐成芯微科技股份有限公司 Low-power-consumption Bluetooth attribute access method and low-power-consumption Bluetooth system

Also Published As

Publication number Publication date
CN106681995B (en) 2020-08-18

Similar Documents

Publication Publication Date Title
CN106681995A (en) Data caching method and data query method and device
US10719495B2 (en) Stream selection for multi-stream storage devices
US20210056087A1 (en) Method and apparatus for providing efficient indexing and computer program included in computer readable medium therefor
CN107491523B (en) Method and device for storing data object
KR20200053512A (en) KVS tree database
CN108255958A (en) Data query method, apparatus and storage medium
US20050203901A1 (en) Searching a range in a set of values in a network with distributed storage entities
US20200257732A1 (en) Systems and methods of managing an index
CN106464669B (en) Intelligent file prefetching based on access patterns
US20180203829A1 (en) Automatic updated document prompt method
US8650224B2 (en) Batching content management operations to facilitate efficient database interactions
US20130346540A1 (en) Storing and Moving Data in a Distributed Storage System
CN104268159A (en) Real-time data warehouse data pre-storage and pre-access method based on dynamic mirrors
CN106991102A (en) The processing method and processing system of key-value pair in inverted index
US11210281B2 (en) Technique for log records management in database management system
CN106775498A (en) A kind of data cached synchronous method and system
US10241963B2 (en) Hash-based synchronization of geospatial vector features
KR20160100226A (en) Method and device for constructing on-line real-time updating of massive audio fingerprint database
CN108647266A (en) A kind of isomeric data is quickly distributed storage, exchange method
CN108153776A (en) Data query method and device
CN108416609A (en) Advertisement gray scale put-on method and device
US20200159722A1 (en) Presenting updated data using persisting views
US9104689B2 (en) Method for synchronizing documents for disconnected operation
CN110727690A (en) Data updating method
US11080239B2 (en) Key value store using generation markers

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
TA01 Transfer of patent application right

Effective date of registration: 20180410

Address after: Four story 847 mailbox of the capital mansion of Cayman Islands, Cayman Islands, Cayman

Applicant after: CAINIAO SMART LOGISTICS HOLDING Ltd.

Address before: Grand Cayman, Cayman Islands

Applicant before: ALIBABA GROUP HOLDING Ltd.

TA01 Transfer of patent application right
GR01 Patent grant
GR01 Patent grant