CN109918380A - Data processing method, device, storage medium and electronic equipment - Google Patents
Data processing method, device, storage medium and electronic equipment Download PDFInfo
- Publication number
- CN109918380A CN109918380A CN201910175898.9A CN201910175898A CN109918380A CN 109918380 A CN109918380 A CN 109918380A CN 201910175898 A CN201910175898 A CN 201910175898A CN 109918380 A CN109918380 A CN 109918380A
- Authority
- CN
- China
- Prior art keywords
- data
- storage unit
- serial number
- index data
- index
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Abstract
The invention discloses a kind of data processing method, device, storage medium and electronic equipments, wherein this method comprises: receiving data read command, wherein mapping techniques are stored in the buffer area being made of multiple storage units data based on memory；Determine that the corresponding index data of data, index data include level-one index data and secondary index data according to data read command；The serial number information of the affiliated storage unit of data is determined according to level-one index data；Determine the location information of data, according to the serial number information of affiliated storage unit and secondary index data to read data.By the invention it is possible to which high efficiency needed for meeting Replicator process is read and write.
Description
Technical field
The present invention, which discloses, is related to data processing field, and in particular to a kind of data processing method, device, storage medium and electricity
Sub- equipment.
Background technique
In the duplication of MySQL data, since binlog (log) time of file on MySQL server is shorter, magnetic
When disk space deficiency, binlog file can be disposed of.
Currently, storage journal file relies primarily on the caching system of such as Redis cluster, Redis is open source software, can be with
Support very big data capacity, and Redis itself also supports data persistence function.
Then, Redis has following defects that 1) (reproducer is responsible for extraction in the Replicator of MySQL data duplication
With caching MySQL binlog log recording) in component, when there are super large buffer size scene, need according to certain data ID
(Identity, identity number) inquires next data, and Redis does not support such operation；2) Replicator is needed
Fixed cache size is set, if spatial cache is full, the newest data being written into can squeeze out oldest data；And
Redis deletes expired data according to TTL (Time to Live, the remaining life span) mechanism of KEY (key), between the two
There is biggish difference, has large effect to the usage scenario of business.
Summary of the invention
In view of this, the embodiment of the present invention provides a kind of data processing method, device, storage medium and electronic equipment, with
It solves not bettering meet due to relying on caused by Redis cluster-based caching system carrys out storing data in the prior art
The problem of Replicator process demand.
According to a first aspect of the embodiments of the present invention, a kind of data processing method is provided, is read this method comprises: receiving data
Instruction fetch, wherein mapping techniques are stored in the buffer area being made of multiple storage units data based on memory；It is read according to data
Instruction fetch determines that the corresponding index data of data, index data include level-one index data and secondary index data；According to level-one
Index data determines the serial number information of the affiliated storage unit of data；According to the serial number information of affiliated storage unit and secondary index number
According to the location information for determining data, to read data.
According to a second aspect of the embodiments of the present invention, a kind of data processing equipment is provided, which includes: command reception list
Member reads instruction for receiving data, wherein data based on memory mapping techniques be stored in be made of multiple storage units it is slow
It deposits in area；Index data determination unit, for determining the corresponding index data of data, index data packet according to data read command
Include level-one index data and secondary index data；Serial number determination unit is deposited belonging to data for being determined according to level-one index data
The serial number information of storage unit；Data-reading unit, for true according to the serial number information and secondary index data of affiliated storage unit
The location information of fixed number evidence, to read data.
According to a third aspect of the embodiments of the present invention, a kind of computer readable storage medium is provided, stores computer thereon
Program instruction, wherein the computer program instructions realize method as described in relation to the first aspect when being executed by processor.
According to a fourth aspect of the embodiments of the present invention, a kind of electronic equipment, including memory and processor are provided, wherein
The memory is for storing one or more computer program instructions, wherein one or more computer program instructions
It is executed by the processor to realize method as described in relation to the first aspect.
The embodiment of the present invention by being stored in data in the buffer area being made of multiple storage units based on MMAP technology,
When receiving data read command, it is first determined the index data of the data determines the tool of data according to index data later
Body storage location simultaneously reads data, to carry out subsequent processing to data, compared with the prior art in dependence Redis cluster it is slow
Deposit system carrys out storing data, and the embodiment of the present invention is due to being based on MMAP technology, height needed for can satisfy Replicator process
Efficiency read-write, and by setting storage unit quantity and each storage unit capacity, can satisfy Replicator into
The super large caching mechanism of Cheng Suoxu.
Detailed description of the invention
By referring to the drawings to the description of the embodiment of the present invention, the above and other purposes of the present invention, feature and
Advantage will be apparent from, in the accompanying drawings:
Fig. 1 is the flow chart of data processing method according to an embodiment of the present invention；
Fig. 2 is the storage organization schematic diagram of data according to an embodiment of the present invention and index data；
Fig. 3 is the structural block diagram of data processing equipment according to an embodiment of the present invention；
Fig. 4 is the detailed block diagram of data processing equipment according to an embodiment of the present invention；
Fig. 5 is the structural block diagram of MMU memory management unit 309 according to an embodiment of the present invention；
Fig. 6 is the structural block diagram of data-reading unit 304 according to an embodiment of the present invention；
Fig. 7 is the application scenario diagram of data processing equipment according to an embodiment of the present invention；
Fig. 8 is the schematic diagram of electronic equipment according to an embodiment of the present invention.
Specific embodiment
Below based on embodiment, present invention is described, but the present invention is not restricted to these embodiments.Under
Text is detailed to describe some specific detail sections in datail description of the invention.Do not have for a person skilled in the art
The present invention can also be understood completely in the description of these detail sections.In order to avoid obscuring essence of the invention, well known method, mistake
There is no narrations in detail for journey, process, element and circuit.
In addition, it should be understood by one skilled in the art that provided herein attached drawing be provided to explanation purpose, and
What attached drawing was not necessarily drawn to scale.
Unless the context clearly requires otherwise, "include", "comprise" otherwise throughout the specification and claims etc. are similar
Word should be construed as the meaning for including rather than exclusive or exhaustive meaning；That is, be " including but not limited to " contains
Justice.
In the description of the present invention, it is to be understood that, term " first ", " second " etc. are used for description purposes only, without
It can be interpreted as indication or suggestion relative importance.In addition, in the description of the present invention, unless otherwise indicated, the meaning of " multiple "
It is two or more.
Carry out storing data due to relying on Redis cluster-based caching system in the prior art to result in and can not better meet
The demand of Replicator process, therefore, Repliactor need to design a set of super large caching mechanism based in JAVA process,
And it is data cached to support persistent storage, can restore data cached after Replicator process is restarted because of failure.
Based on this, the embodiment of the invention provides a kind of data processing method, Fig. 1 is the process of the data processing method
Figure, as shown in Figure 1, this method comprises:
Step 101, data read command is received, wherein data map (MMAP) technology based on memory and are stored in by multiple
In the buffer area that storage unit is constituted；
Step 102, determine that the corresponding index data of data, index data include level-one index number according to data read command
According to secondary index data；
Step 103, the serial number information of the affiliated storage unit of data is determined according to level-one index data；
Step 104, the location information of data is determined according to the serial number information of affiliated storage unit and secondary index data, with
Read the data.
By being stored in data in the buffer area being made of multiple storage units based on MMAP technology, data are being received
When reading instruction, it is first determined the index data of the data determines the specific storage location of data simultaneously according to index data later
Read data, to carry out subsequent processing to data, compared with the prior art in dependence Redis cluster-based caching system store
Data, the embodiment of the present invention are read and write due to being based on MMAP technology, high efficiency needed for can satisfy Replicator process, and
By the way that the quantity of storage unit and the capacity of each storage unit is arranged, super large needed for can satisfy Replicator process is slow
Deposit mechanism.
In embodiments of the present invention, in construction buffer area, serial number can be set gradually for multiple storage units；It is being written
When data, data can be sequentially written in storage unit according to storage unit serial number；And for the number of each storage unit is written
According to setting gradually data number.It, will be in the storage unit of presently written data cover serial number 1 when all storage units are fully written
Data.
Above-mentioned level-one index data may include: data in the serial number information of affiliated storage unit, in affiliated storage unit
Positional shift information and the location information in secondary index data.
Above-mentioned secondary index data include: data number information of the data in affiliated storage unit.
In actual operation, level-one index data can be stored in heap memory (heap) in the form of skip list；Secondary index
Data can be stored in out-pile memory with array form.Index data is stored in this way, can save more memory headroom.
At step 104, array two first can be passed through according to the serial number information of affiliated storage unit and secondary index data
The position for dividing lookup algorithm to determine data in the position range of affiliated storage unit；Number is read on determining position later
According to.Here array binary chop algorithm is the normal lookup algorithm of computer field, can be selected according to the actual situation specific
Binary chop algorithm.
Preferably, the data in the embodiment of the present invention and index data can be managed based on MMAP technology.Using
MMAP technology manages data and index data, can be not take up heap, can also direct Map Disk file, not by physics
Capacity limit is deposited, therefore, less physical memory can be consumed.In addition, MMAP is using out-pile memory, it is unrelated with heap,
It does not require that the very big space heap is arranged, also, out-pile memory does not return JVAGC (Java Virtual Machine Garbage
Collection, Java Virtual Machine garbage reclamation) recycling, therefore even if out-pile memory is very big, additionally GC will not be caused to bear.
Since reading and writing data actually occurs in memory, the data in memory are to be responsible for asynchronous and synchronous to file by kernel
In, therefore the reading and writing data performance that entirely caches still rank based on memory.MMAP technology establishes memory mapping and does not generate reality
The data copy on border, therefore under most of scenes, only one storage unit is being written or read data, other storages are single
Member is an inactive state, and the storage unit File Mapping of an inactive state, what is theoretically consumed is the virtual memory of process, is not
Real physical memory occupies resource very little.
In the specific implementation process, by taking JAVA as an example, Circular buffer data can be constructed using MMAP file mapping technology
(ring buffer data) structure can map 1024 files in the buffer area (buffer), and each file size is
1GB obtains the aggregate capacity of 1024*1GB=1TB, and each mapped file here is a storage unit, storage here
Unit is properly termed as slot (slot), that is to say, that 1024 slot are built into a Circular buffer area, and are this 1024 slot
Set gradually serial number.
When writing data into buffer, data are sequentially written in slot according to slot serial number；When all slots are all fully written number
According to when, by new data cover slot serial number 1 in data, so that new data is written.Specifically, when data are written, exist first
The tail portion write-in data of current slot (initial write operation is since the slot of serial number 1) connect after a slot is fully written
Data are written on next slot, when the last one slot writes it is full after, number is written since the 1st slot again
Data before being covered according to, data being newly written.
The Circular buffer structure of above-mentioned MMAP file mapping technology building, possesses super large caching mechanism, it is data cached can be with
It supports persistent storage, can restore data cached after Replicator process is restarted because of failure.That is, passing through the annular
Buffer structure, can before for a long time time replicate data can also continue from breakpoint after prolonged duplication is interrupted
Duplication；Replicator only needs to extract a log from MySQL, so that it may for many Applier (be responsible for from
Data are obtained in Replicator component, are then write data into the target database of duplication) it uses, realize primary pumping
It takes and is used for multiple times；In addition, the reading and writing data cached based on memory, has very high readwrite performance.
In embodiments of the present invention, every data by key-value (key-value) to forming, the key value of every data
With write sequence monotonic increase, when needing to read data from buffer, the key of data is first judged according to the first index data
In which slot, the Position Approximate range in slot and the location information in secondary index data, later according to second level
Index data determines the specific location of data and reads data.
Specifically, level-one index data can be indicated with following structure type:
|slot idx(2byte)|slot position(4byte)|index record idx(4byte)|…|
Wherein, slot idx is indicated: the serial number of (place) slot belonging to data, such as: idx 100 indicates serial number
100 slot；
Slot position is indicated: positional shift (offset) of the data in slot, such as: position 200,
Indicate the position of the slot of serial number 100 started at 200 byte of head；
Index record idx is indicated: the serial number in secondary index data (that is, list slot global index).
Secondary index data can be indicated with following structure type:
|index record count(4byte)|[index record]|[index record]|…|
Wherein, index record count is indicated: index data item number；
Index record is indicated: index data, the index data identify the more specific location information of data.
In embodiments of the present invention, the index data between slot and slot is mutually isolated, and corresponding one of each slot is solely
From index space.
Fig. 2 is the storage organization schematic diagram of data according to an embodiment of the present invention and index data, as shown in Fig. 2, data
It is stored in each slot (slot) in Circular buffer area, in the buffer area, slot serial number is incremented by upwards by 1, each slot's
Capacity is 1GB, when slot capacity available space is not enough to store a data (or being known as recording), has determined current slot
It writes completely, data manipulation is written later can automatically switch in Next Sequence (being incremented by 1) slot, and the data in each slot pass through
MMAP Map Disk file is to memory.
Specifically, by key-value (key-value) to forming, slot structure can pass through every data in slot
As under type indicates:
|last position(4byte)|key length(1byte)+key+data length(4byte)+data+
checksum|…|
Wherein, last position is indicated: current slot's writes offset (write offset)；The last
Position is located in 4 byte of head of each slot content；
Key length and key are indicated: the key of every data, key are using length and caching (length+buffer) knot
Structure stores；
Data length and data are indicated: the value (that is, particular content) of every data, value use length+
Buffer structure stores；
Checksum is indicated: the checksum value of data, for being verified when reading data.
In actual operation, index data can also consume biggish memory headroom, and therefore, index data cannot be stored all
In heap, it is contemplated that out-pile memory realizes that the complexity of self-defining data structure and algorithm, the embodiment of the present invention use heap
Mode of the skip list in conjunction with out-pile array realizes the index of entire buffer.
In schematic diagram shown in Fig. 2, segmented index is the first above-mentioned index data, and heap is stored in the form of skip list
In memory (heap)；Slot-n full dose index is the second above-mentioned index data, is stored in out-pile in the form of array (array)
In memory (such as direct buffer).
1024 slot serial number informations are stored by skip list structure in heap, in single slot at interval of 1000 datas
Location information and the location information in out-pile array.In this way, can quickly judge target by skip list when searching data
The target key of data belongs to which slot, the Position Approximate range in slot and the location information in out-pile array, later
Data Position is quickly determined in buffer by binary chop algorithm according to the location information in out-pile array and reads number
According to.
For example, receiving the reading instruction of data A, the key value of data A is 100, and inquiry should first in heap skip list
Target key and its corresponding slot serial number information and the location information in out-pile array, such as find the position of target key
It is the 2000, location information in out-pile data is 1501 that confidence breath, which is positional shift information in slot=12, slot, that
, so that it may it is located in the slot of serial number 12, locking positional shift in slot 12 is 1000~2000 this one piece of data,
The data that target key belongs to data number 1501 are searched by binary chop algorithm, specific data A can be navigated to later simultaneously
Read data A.
The embodiment of the present invention also provides a kind of data processing equipment, and Fig. 3 is the structural block diagram of the device, as shown in figure 3, should
Device includes: instruction receiving unit 301, index data determination unit 302, serial number determination unit 303 and data-reading unit
304, in which:
Instruction receiving unit 301 reads instruction, wherein mapping techniques are stored in data based on memory for receiving data
In the buffer area be made of multiple storage units；
Index data determination unit 302, for determining the corresponding index data of data, index number according to data read command
According to including level-one index data and secondary index data；
Serial number determination unit 303, for determining the serial number information of the affiliated storage unit of data according to level-one index data；
Data-reading unit 304, for determining data according to the serial number information and secondary index data of affiliated storage unit
Location information, to read data.
By being stored in data in the buffer area being made of multiple storage units based on MMAP technology, in command reception list
When member 301 receives data read command, index data determination unit 302 determines the index data of the data, and serial number is true later
Order member 303 determines the storage location of data according to index data and reads data by data-reading unit 304, reads data
Subsequent processing can be carried out to data later, compared with the prior art in dependence Redis cluster-based caching system store number
According to the embodiment of the present invention is led to due to being based on MMAP technology, the read-write of high efficiency needed for can satisfy Replicator process
Cross the quantity of setting storage unit and the capacity of each storage unit, the caching of super large needed for can satisfy Replicator process
Mechanism.
As shown in figure 4, the device can also include: serial number setting unit 305, writing unit 306 and number setting unit
307, in which:
Serial number setting unit 305, for setting gradually serial number for multiple storage units；
Writing unit 306, for data to be sequentially written in storage unit according to storage unit serial number；
Number setting unit 307, for setting gradually data number for the data of each storage unit of write-in.
Specifically, in construction buffer area, serial number setting unit 305 is that multiple storage units set gradually serial number；It is writing
When entering data, data can be sequentially written in storage unit by writing unit 306 according to storage unit serial number；Number setting unit
307 be that the data of each storage unit are written to set gradually data number.When all storage units are fully written, writing unit 306
It can be by the data in the storage unit of presently written data cover serial number 1.
In embodiments of the present invention, level-one index data may include: data affiliated storage unit serial number information,
The positional shift information of affiliated storage unit and the location information in secondary index data.Secondary index data can wrap
It includes: data number information of the data in affiliated storage unit.
The structure of specific level-one index data and secondary index data may refer to the description in above-described embodiment, herein
It repeats no more.
Preferably, above-mentioned apparatus further include: Data Management Unit 308, for managing data and rope based on MMAP technology
Argument evidence.Data and index data are managed using MMAP technology by Data Management Unit 308, heap can be not take up,
Can directly Map Disk file not by physical memory capacity limit therefore, less physical memory can be consumed.In addition,
MMAP additionally will not cause GC to bear using out-pile memory.
Also, MMAP technology establishes memory mapping and does not generate actual data copy, therefore under most of scenes, only
One storage unit is being written or read data, other storage units are an inactive state, the storage unit of an inactive state
File Mapping, it is not real physical memory that is theoretically consumed, which is the virtual memory of process, occupies resource very little.
Preferably, above-mentioned apparatus further include: MMU memory management unit 309, for storing index data in a predefined manner.Specifically
Ground, as shown in figure 5, MMU memory management unit 309 specifically includes: the first memory management module 3091 and the second memory management module
3092, in which:
First memory management module 3091, for level-one index data to be stored in heap memory in the form of skip list；
Second memory management module 3092, for being stored in secondary index data in out-pile memory with array form.
The first index is stored in different ways by the first memory management module 3091 and the second memory management module 3092
Data and the second index data, can save more memory headroom.
As shown in fig. 6, data-reading unit 304 includes: Data Position determining module 3041 and data read module 3042,
Wherein:
Data Position determining module 3041, for being passed through according to the serial number information and secondary index data of affiliated storage unit
Binary chop algorithm determines the location information of data；
Data read module 3042, for reading data on determining position.
Here array binary chop algorithm is the normal lookup algorithm of computer field, can be selected according to the actual situation
Specific binary chop algorithm.
Fig. 7 is the application scenario diagram of data processing equipment according to an embodiment of the present invention, as shown in fig. 7, command reception list
Member 301 receives data read command A, and for reading data a, index data determination unit 302 determines the index number of the data
According to serial number determination unit 303 determines the storage location of data according to index data and read by data-reading unit 304 later
Data a, subsequent processing can be carried out to data later by reading data.
Fig. 8 is the schematic diagram of the electronic equipment of the embodiment of the present invention.Electronic equipment shown in Fig. 8 is general data processing dress
It sets comprising general computer hardware structure includes at least processor 801 and memory 802.Processor 801 and storage
Device 802 is connected by bus 803.Memory 802 is suitable for one or more instruction or program that storage processor 801 can be performed.
One or more instruction or program are executed by processor 801 to realize following steps:
Receive data read command, wherein mapping techniques are stored in and are made of multiple storage units data based on memory
In buffer area；
Determine that the corresponding index data of the data, the index data include level-one rope according to the data read command
Argument evidence and secondary index data；
The serial number information of the affiliated storage unit of the data is determined according to the level-one index data；
The position letter of the data is determined according to the serial number information of the affiliated storage unit and the secondary index data
Breath, to read the data.
Above-mentioned steps further include: set gradually serial number for the multiple storage unit；According to storage unit serial number by data
It is sequentially written in storage unit；Data number is set gradually for the data of each storage unit are written.
Above-mentioned steps further include: be fully written in response to all storage units, by depositing for presently written data cover serial number 1
Data in storage unit.
The position letter of the data is determined according to the serial number information of the affiliated storage unit and the secondary index data
Breath includes: according to the serial number information of the affiliated storage unit and the secondary index data by two to read the data
The location information for dividing lookup algorithm to determine the data；The data are read in determining position.
The index data is stored in the following way: the level-one index data is stored in heap memory in the form of skip list
In；The secondary index data are stored in out-pile memory with array form.
The level-one index data include: the data in the serial number information of the affiliated storage unit, described affiliated
The positional shift information of storage unit and the location information in the secondary index data.
The secondary index data include: data number information of the data in the affiliated storage unit.
Above-mentioned steps further include: the data and the index data are managed based on the memory mapping technique.
The buffer area is Circular buffer area.
Above-mentioned processor 801 can be independent microprocessor, be also possible to one or more microprocessor set.By
This, the order that processor 801 is stored by executing memory 802, thereby executing the side of embodiment present invention as described above
Method process realizes the processing for data and the control for other devices.Bus 803 links together above-mentioned multiple components,
Said modules are connected to display controller 804 and display device and input/output (I/O) device 805 simultaneously.Input/defeated
Out (I/O) device 805 can be mouse, keyboard, modem, network interface, touch-control input device, body-sensing input unit,
Printer and other devices well known in the art.Typically, input/output (I/O) device 805 passes through input/output (I/O)
Controller 806 is connected with system.
Wherein, memory 802 can store component software, such as operating system, communication module, interactive module and application
Program.Above-described each module and application program are both corresponded to complete one or more functions and be retouched in inventive embodiments
One group of executable program instructions of the method stated.
In conclusion the embodiment of the present invention, which is based on MMAP technology, constructs large capacity cache structure, by storing data
In the large capacity cache area being made of multiple storage units, when receiving data read command, by determining the data
Index data searches the specific storage location of data and reads data, and to carry out subsequent processing to data, the present invention is implemented
Example is due to being based on MMAP technology, the read-write of high efficiency needed for can satisfy Replicator process, and it is single to pass through setting storage
The quantity of member and the capacity of each storage unit, super large caching mechanism needed for can satisfy Replicator process.
It is above-mentioned according to the method for the embodiment of the present invention, the flow chart and/or frame of equipment (system) and computer program product
Figure describes various aspects of the invention.It should be understood that each of flowchart and or block diagram piece and flow chart legend and/or frame
The combination of block in figure can be realized by computer program instructions.These computer program instructions can be provided to general meter
The processor of calculation machine, special purpose computer or other programmable data processing devices, to generate machine so that (via computer or
What the processors of other programmable data processing devices executed) instruction creates for realizing in flowchart and or block diagram block or block
The device of specified function action.
Meanwhile as skilled in the art will be aware of, the various aspects of the embodiment of the present invention may be implemented as be
System, method or computer program product.Therefore, the various aspects of the embodiment of the present invention can take following form: complete hardware
Implementation, complete software implementation (including firmware, resident software, microcode etc.) usually can all claim herein
For the implementation for combining software aspects with hardware aspect of circuit, " module " or " system ".In addition, side of the invention
Face can take following form: the computer program product realized in one or more computer-readable medium, computer can
Reading medium has the computer readable program code realized on it.
It can use any combination of one or more computer-readable mediums.Computer-readable medium can be computer
Readable signal medium or computer readable storage medium.Computer readable storage medium can be such as (but not limited to) electronics,
Magnetic, optical, electromagnetism, infrared or semiconductor system, device or any suitable combination above-mentioned.Meter
The more specific example (exhaustive to enumerate) of calculation machine readable storage medium storing program for executing will include the following terms: with one or more electric wire
Electrical connection, hard disk, random access memory (RAM), read-only memory (ROM), erasable is compiled portable computer diskette
Journey read-only memory (EPROM or flash memory), optical fiber, portable optic disk read-only storage (CD-ROM), light storage device,
Magnetic memory apparatus or any suitable combination above-mentioned.In the context of the embodiment of the present invention, computer readable storage medium
It can be that can include or store the program used by instruction execution system, device or combine instruction execution system, set
Any tangible medium for the program that standby or device uses.
Computer-readable signal media may include the data-signal propagated, and the data-signal of the propagation has wherein
The computer readable program code realized such as a part in a base band or as carrier wave.The signal of such propagation can use
Any form in diversified forms, including but not limited to: electromagnetism, optical or its any combination appropriate.It is computer-readable
Signal media can be following any computer-readable medium: not be computer readable storage medium, and can be to by instructing
Program that is that execution system, device use or combining instruction execution system, device to use is communicated, is propagated
Or transmission.
Computer program code for executing the operation for being directed to various aspects of the present invention can be with one or more programming languages
Any combination of speech is write, the programming language include: programming language such as Java, Smalltalk of object-oriented, C++,
PHP, Python etc.；And conventional process programming language such as " C " programming language or similar programming language.Program code can be made
It fully on the user computer, is partly executed on the user computer for independent software package；Partly in subscriber computer
Above and partly execute on the remote computer；Or it fully executes on a remote computer or server.In latter feelings
It, can be by remote computer by including that any type of network connection of local area network (LAN) or wide area network (WAN) are extremely used under condition
Family computer, or (such as internet by using ISP) can be attached with outer computer.
The above description is only a preferred embodiment of the present invention, is not intended to restrict the invention, for those skilled in the art
For, the invention can have various changes and changes.All any modifications made within the spirit and principles of the present invention are equal
Replacement, improvement etc., should all be included in the protection scope of the present invention.
The embodiment of the invention discloses A1, a kind of data processing method, wherein the described method includes:
Receive data read command, wherein mapping techniques are stored in and are made of multiple storage units data based on memory
In buffer area；
Determine that the corresponding index data of the data, the index data include level-one rope according to the data read command
Argument evidence and secondary index data；
The serial number information of the affiliated storage unit of the data is determined according to the level-one index data；
The position letter of the data is determined according to the serial number information of the affiliated storage unit and the secondary index data
Breath, to read the data.
A2, the data processing method according to claim A1, wherein the method also includes:
Serial number is set gradually for the multiple storage unit；
Data are sequentially written in storage unit according to storage unit serial number；
Data number is set gradually for the data of each storage unit are written.
A3, the data processing method according to claim A2, wherein the method also includes:
It is fully written in response to all storage units, by the data in the storage unit of presently written data cover serial number 1.
A4, the data processing method according to claim A1, wherein believed according to the serial number of the affiliated storage unit
Breath and the secondary index data determine the location information of the data, include: to read the data
It is determined according to the serial number information of the affiliated storage unit and the secondary index data by binary chop algorithm
The location information of the data；
The data are read in determining position.
A5, the data processing method according to claim A1, wherein store the index number in the following way
According to:
The level-one index data is stored in heap memory in the form of skip list；
The secondary index data are stored in out-pile memory with array form.
A6, the data processing method according to claim A1, wherein the level-one index data includes:
The data in the serial number information of the affiliated storage unit, in the positional shift information of the affiliated storage unit
And the location information in the secondary index data.
A7, the data processing method according to claim A1, wherein the secondary index data include:
Data number information of the data in the affiliated storage unit.
A8, the data processing method according to claim A1, wherein the method also includes:
The data and the index data are managed based on the memory mapping technique.
A9, the data processing method according to claim A1, wherein the buffer area is Circular buffer area.
The embodiment of the invention discloses B1, a kind of data processing equipment, wherein described device includes:
Instruction receiving unit reads instruction, wherein mapping techniques are stored in by more data based on memory for receiving data
In the buffer area that a storage unit is constituted；
Index data determination unit, for determining the corresponding index data of the data according to the data read command,
The index data includes level-one index data and secondary index data；
Serial number determination unit, for determining that the serial number of the affiliated storage unit of the data is believed according to the level-one index data
Breath；
Data-reading unit, for being determined according to the serial number information of the affiliated storage unit and the secondary index data
The location information of the data, to read the data.
B2, the data processing equipment according to claim B1, wherein described device further include:
Serial number setting unit, for setting gradually serial number for the multiple storage unit；
Writing unit, for data to be sequentially written in storage unit according to storage unit serial number；
Number setting unit, for setting gradually data number for the data of each storage unit of write-in.
B3, the data processing equipment according to claim B2, wherein said write data are also used to:
It is fully written in response to all storage units, by the data in the storage unit of presently written data cover serial number 1.
B4, the data processing equipment according to claim B1, wherein the data-reading unit includes:
Data Position determining module, for according to the affiliated storage unit serial number information and the secondary index data
The location information of the data is determined by binary chop algorithm；
Data read module, for reading the data in determining position.
B5, the data processing equipment according to claim B1, wherein described device further include: MMU memory management unit,
For storing the index data in a predefined manner；
The MMU memory management unit specifically includes:
First memory management module, for the level-one index data to be stored in heap memory in the form of skip list；
Second memory management module, for being stored in the secondary index data in out-pile memory with array form.
B6, the data processing equipment according to claim B1, wherein the one of the index data determination unit determination
Grade index data include:
The data in the serial number information of the affiliated storage unit, in the positional shift information of the affiliated storage unit
And the location information in the secondary index data.
B7, the data processing equipment according to claim B1, wherein the two of the index data determination unit determination
Grade index data include:
Data number information of the data in the affiliated storage unit.
B8, the data processing equipment according to claim B1, wherein described device further include:
Data Management Unit, for managing the data and the index data based on the memory mapping technique.
B9, the data processing equipment according to claim B1, wherein the buffer area is Circular buffer area.
The embodiment of the invention discloses C1, a kind of computer readable storage medium, store computer program instructions thereon,
In, the computer program instructions realize the method as described in any one of above-mentioned A1-A9 when being executed by processor.
The embodiment of the invention discloses D1, a kind of electronic equipment, including memory and processor, wherein the memory
For storing one or more computer program instructions, wherein one or more computer program instructions are by the processing
Device is executed to realize the method as described in any one of above-mentioned A1-A9.
Claims (10)
1. a kind of data processing method, which is characterized in that the described method includes:
Receive data read command, wherein mapping techniques are stored in the caching being made of multiple storage units to data based on memory
In area；
Determine that the corresponding index data of the data, the index data include level-one index number according to the data read command
According to secondary index data；
The serial number information of the affiliated storage unit of the data is determined according to the level-one index data；
The location information of the data is determined according to the serial number information of the affiliated storage unit and the secondary index data, with
Read the data.
2. data processing method according to claim 1, which is characterized in that the method also includes:
Serial number is set gradually for the multiple storage unit；
Data are sequentially written in storage unit according to storage unit serial number；
Data number is set gradually for the data of each storage unit are written.
3. data processing method according to claim 2, which is characterized in that the method also includes:
It is fully written in response to all storage units, by the data in the storage unit of presently written data cover serial number 1.
4. data processing method according to claim 1, which is characterized in that believed according to the serial number of the affiliated storage unit
Breath and the secondary index data determine the location information of the data, include: to read the data
Passed through described in the determination of binary chop algorithm according to the serial number information of the affiliated storage unit and the secondary index data
The location information of data；
The data are read in determining position.
5. data processing method according to claim 1, which is characterized in that store the index number in the following way
According to:
The level-one index data is stored in heap memory in the form of skip list；
The secondary index data are stored in out-pile memory with array form.
6. data processing method according to claim 1, which is characterized in that the level-one index data includes:
Serial number information of the data in the affiliated storage unit, the positional shift information in the affiliated storage unit and
Location information in the secondary index data.
7. data processing method according to claim 1, which is characterized in that the secondary index data include:
Data number information of the data in the affiliated storage unit.
8. a kind of data processing equipment, which is characterized in that described device includes:
Instruction receiving unit reads instruction for receiving data, wherein mapping techniques are stored in and are deposited by multiple data based on memory
In the buffer area that storage unit is constituted；
Index data determination unit, it is described for determining the corresponding index data of the data according to the data read command
Index data includes level-one index data and secondary index data；
Serial number determination unit, for determining the serial number information of the affiliated storage unit of the data according to the level-one index data；
Data-reading unit, for according to the serial number information of the affiliated storage unit and secondary index data determination
The location information of data, to read the data.
9. a kind of computer readable storage medium, stores computer program instructions thereon, which is characterized in that the computer program
Such as method of any of claims 1-7 is realized in instruction when being executed by processor.
10. a kind of electronic equipment, including memory and processor, which is characterized in that the memory is for storing one or more
Computer program instructions, wherein one or more computer program instructions are executed by the processor to realize such as power
Benefit requires method described in any one of 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910175898.9A CN109918380A (en) | 2019-03-08 | 2019-03-08 | Data processing method, device, storage medium and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910175898.9A CN109918380A (en) | 2019-03-08 | 2019-03-08 | Data processing method, device, storage medium and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109918380A true CN109918380A (en) | 2019-06-21 |
Family
ID=66963912
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910175898.9A Pending CN109918380A (en) | 2019-03-08 | 2019-03-08 | Data processing method, device, storage medium and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109918380A (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111427883A (en) * | 2020-02-18 | 2020-07-17 | 深圳壹账通智能科技有限公司 | Data processing method and device based on AeroPike, computer equipment and storage medium |
| CN111651438A (en) * | 2020-04-28 | 2020-09-11 | 银江股份有限公司 | MapDB-based structured data deduplication method, device, equipment and medium |
| CN113253926A (en) * | 2021-05-06 | 2021-08-13 | 天津大学深圳研究院 | Memory internal index construction method for improving query and memory performance of novel memory |
| CN113626446A (en) * | 2021-10-09 | 2021-11-09 | 阿里云计算有限公司 | Data storage and search method, device, electronic equipment and medium |
| CN113703413A (en) * | 2021-11-01 | 2021-11-26 | 西安热工研究院有限公司 | Data interaction method, system, equipment and storage medium based on secondary index |
| CN115563103A (en) * | 2022-09-15 | 2023-01-03 | 河南星环众志信息科技有限公司 | Multi-dimensional aggregation method, system, electronic device and storage medium |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102043795A (en) * | 2009-10-13 | 2011-05-04 | 上海新华控制技术(集团)有限公司 | Establishing method for process control historical data file structure and data read-write method |
| CN103294710A (en) * | 2012-02-28 | 2013-09-11 | 北京新媒传信科技有限公司 | Data access method and device |
| CN104899297A (en) * | 2015-06-08 | 2015-09-09 | 南京航空航天大学 | Hybrid index structure with storage perception |
| US20150347479A1 (en) * | 2014-05-28 | 2015-12-03 | International Business Machines Corporation | Storing and querying multidimensional data using first and second indicies |
| CN105159987A (en) * | 2015-08-31 | 2015-12-16 | 深圳市茁壮网络股份有限公司 | Data storage and query method and apparatus |
| US20170068465A1 (en) * | 2015-09-09 | 2017-03-09 | Sap Se | Hybrid Heap Memory Management |
| CN108255958A (en) * | 2017-12-21 | 2018-07-06 | 百度在线网络技术（北京）有限公司 | Data query method, apparatus and storage medium |
| CN108647280A (en) * | 2018-05-03 | 2018-10-12 | 北京云中融信网络科技有限公司 | A kind of method and apparatus of memory communicating information |
| CN109101554A (en) * | 2018-07-12 | 2018-12-28 | 厦门中控智慧信息技术有限公司 | For the data buffering system of JAVA platform, method and terminal |
-
2019
- 2019-03-08 CN CN201910175898.9A patent/CN109918380A/en active Pending
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102043795A (en) * | 2009-10-13 | 2011-05-04 | 上海新华控制技术(集团)有限公司 | Establishing method for process control historical data file structure and data read-write method |
| CN103294710A (en) * | 2012-02-28 | 2013-09-11 | 北京新媒传信科技有限公司 | Data access method and device |
| US20150347479A1 (en) * | 2014-05-28 | 2015-12-03 | International Business Machines Corporation | Storing and querying multidimensional data using first and second indicies |
| CN104899297A (en) * | 2015-06-08 | 2015-09-09 | 南京航空航天大学 | Hybrid index structure with storage perception |
| CN105159987A (en) * | 2015-08-31 | 2015-12-16 | 深圳市茁壮网络股份有限公司 | Data storage and query method and apparatus |
| US20170068465A1 (en) * | 2015-09-09 | 2017-03-09 | Sap Se | Hybrid Heap Memory Management |
| CN108255958A (en) * | 2017-12-21 | 2018-07-06 | 百度在线网络技术（北京）有限公司 | Data query method, apparatus and storage medium |
| CN108647280A (en) * | 2018-05-03 | 2018-10-12 | 北京云中融信网络科技有限公司 | A kind of method and apparatus of memory communicating information |
| CN109101554A (en) * | 2018-07-12 | 2018-12-28 | 厦门中控智慧信息技术有限公司 | For the data buffering system of JAVA platform, method and terminal |
Non-Patent Citations (2)
| Title |
|---|
| 刘晓建 等: ""环形缓冲区"", 《计算机操作系统》 * |
| 黄小数: ""2018年第52周-Kafka知识点"", 《HTTPS://SEGMENTFAULT.COM/A/1190000017576243》 * |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111427883A (en) * | 2020-02-18 | 2020-07-17 | 深圳壹账通智能科技有限公司 | Data processing method and device based on AeroPike, computer equipment and storage medium |
| CN111651438A (en) * | 2020-04-28 | 2020-09-11 | 银江股份有限公司 | MapDB-based structured data deduplication method, device, equipment and medium |
| CN113253926A (en) * | 2021-05-06 | 2021-08-13 | 天津大学深圳研究院 | Memory internal index construction method for improving query and memory performance of novel memory |
| CN113626446A (en) * | 2021-10-09 | 2021-11-09 | 阿里云计算有限公司 | Data storage and search method, device, electronic equipment and medium |
| CN113703413A (en) * | 2021-11-01 | 2021-11-26 | 西安热工研究院有限公司 | Data interaction method, system, equipment and storage medium based on secondary index |
| CN113703413B (en) * | 2021-11-01 | 2022-01-25 | 西安热工研究院有限公司 | Data interaction method, system, equipment and storage medium based on secondary index |
| CN115563103A (en) * | 2022-09-15 | 2023-01-03 | 河南星环众志信息科技有限公司 | Multi-dimensional aggregation method, system, electronic device and storage medium |
| CN115563103B (en) * | 2022-09-15 | 2023-12-08 | 河南星环众志信息科技有限公司 | Multi-dimensional aggregation method, system, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109918380A (en) | Data processing method, device, storage medium and electronic equipment | |
| US20150039837A1 (en) | System and method for tiered caching and storage allocation | |
| US9177028B2 (en) | Deduplicating storage with enhanced frequent-block detection | |
| CN110062925A (en) | Snapshot metadata arrangement for converging | |
| US8280858B2 (en) | Storage pool scrubbing with concurrent snapshots | |
| US20110167239A1 (en) | Methods and apparatuses for usage based allocation block size tuning | |
| WO2016086819A1 (en) | Method and apparatus for writing data into shingled magnetic record smr hard disk | |
| RU2671049C2 (en) | Flushing in file system | |
| CN103098043A (en) | On demand virtual machine image streaming | |
| CN106326229B (en) | File storage method and device of embedded system | |
| CN105897859B (en) | Storage system | |
| CN109725842B (en) | System and method for accelerating random write placement for bucket allocation within a hybrid storage system | |
| CN110018998A (en) | A kind of file management method, system and electronic equipment and storage medium | |
| US11169968B2 (en) | Region-integrated data deduplication implementing a multi-lifetime duplicate finder | |
| US10152278B2 (en) | Logical to physical sector size adapter | |
| CN114253908A (en) | Data management method and device of key value storage system | |
| CN109074308A (en) | The block conversion table (BTT) of adaptability | |
| KR20110046118A (en) | Adaptive logging apparatus and method | |
| WO2006130768A2 (en) | Transactional file system with client partitioning | |
| US10346077B2 (en) | Region-integrated data deduplication | |
| CN105653539A (en) | Index distributed storage implement method and device | |
| CN106775501B (en) | Elimination of Data Redundancy system based on nonvolatile memory equipment | |
| US20120209891A1 (en) | Database management method, database management system and database management program | |
| Xu et al. | Building a fast and efficient LSM-tree store by integrating local storage with cloud storage | |
| US11803469B2 (en) | Storing data in a log-structured format in a two-tier storage system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190621 |
|
| RJ01 | Rejection of invention patent application after publication |