CN110377535A - Distributed KV storage system - Google Patents
Distributed KV storage system Download PDFInfo
- Publication number
- CN110377535A CN110377535A CN201810332295.0A CN201810332295A CN110377535A CN 110377535 A CN110377535 A CN 110377535A CN 201810332295 A CN201810332295 A CN 201810332295A CN 110377535 A CN110377535 A CN 110377535A
- Authority
- CN
- China
- Prior art keywords
- equipment
- cold data
- storage
- key
- value
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
- G06F12/02—Addressing or allocation; Relocation
- G06F12/08—Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
- G06F12/10—Address translation
- G06F12/1072—Decentralised address translation, e.g. in distributed shared memory systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
- G06F12/02—Addressing or allocation; Relocation
- G06F12/08—Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
- G06F12/10—Address translation
- G06F12/1009—Address translation using page tables, e.g. page table structures
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
This application discloses distributed KV storage systems.The distributed KV storage system of the application, comprising: client computer and multiple KV storage devices, client computer are coupled with KV storage device by network；Each KV storage device includes: KV storage equipment and calculating equipment；It calculates equipment to be coupled with KV storage equipment, client computer is coupled with multiple KV storage devices by network.
Description
Technical field
This application involves KV storages, and in particular, to distributed KV storage system.
Background technique
Figure 1A illustrates the block diagram of solid storage device.Solid storage device 102 is coupled with host, for mentioning for host
For storage capacity.Host can be coupled in several ways between solid storage device 102, and coupled modes include but is not limited to
For example, by SATA (Serial Advanced Technology Attachment, Serial Advanced Technology Attachment), SCSI
(Small Computer System Interface, small computer system interface), SAS (Serial Attached
SCSI, Serial Attached SCSI (SAS)), IDE (Integrated Drive Electronics, integrated drive electronics), USB
(Universal Serial Bus, universal serial bus), PCIE (Peripheral Component Interconnect
Express, PCIe, high speed peripheral component interconnection), NVMe (NVM Express, high speed non-volatile memory), Ethernet, optical fiber it is logical
Road, cordless communication network etc. connect host and solid storage device 102.Host, which can be, to be set through the above way with storage
The standby information processing equipment communicated, for example, personal computer, tablet computer, server, portable computer, network exchange
Machine, router, cellular phone, personal digital assistant etc..Storing equipment 102 includes interface 103, control unit 104, one or more
A NVM chip 105 and DRAM (Dynamic Random Access Memory, dynamic RAM) 110.
Nand flash memory, phase transition storage, FeRAM (Ferroelectric RAM, ferroelectric memory), MRAM (Magnetic
Random Access Memory, magnetoresistive memory), RRAM (Resistive Random Access Memory, resistance-change memory
Device) etc. be common NVM.
Interface 103 can be adapted to for example, by the side such as SATA, IDE, USB, PCIE, NVMe, SAS, Ethernet, optical-fibre channel
Formula and host exchanging data.
Control unit 104 is used to control the data transmission between interface 103, NVM chip 105 and DRAM 110, also
For storage management, host logical address to flash memory physical address map, erasure balance, bad block management etc..Control unit 104 can
It is realized by the various ways of software, hardware, firmware or combinations thereof, for example, control unit 104 can be FPGA (Field-
Programmable gate array, field programmable gate array), ASIC (Application Specific
Integrated Circuit, application specific integrated circuit) or a combination thereof form.Control unit 104 also may include place
Device or controller are managed, software is executed in processor or controller and carrys out the hardware of manipulation and control component 104 to handle IO
(Input/Output) it orders.Control unit 104 is also coupled to DRAM 110, and may have access to the data of DRAM 110.?
DRAM can store the data of the I/O command of FTL table and/or caching.
Control unit 104 includes flash interface controller (or being Media Interface Connector controller, flash memory channel controller), is dodged
It deposits interface controller and is coupled to NVM chip 105, and sent out in a manner of the interface protocol to follow NVM chip 105 to NVM chip 105
It orders out, to operate NVM chip 105, and receives the command execution results exported from NVM chip 105.Known NVM chip connects
Mouth agreement includes " Toggle ", " ONFI " etc..
Data are usually stored and read on storage medium by page.And data are erased in blocks.Block (also referred to as physical block) packet
Containing multiple pages.Block includes multiple pages.Page (referred to as Physical Page) on storage medium has fixed size, such as 17664 bytes.
Physical Page also can have other sizes.
In solid storage device, safeguarded using FTL (Flash Translation Layer, flash translation layer (FTL)) from
Map information of the logical address to physical address.Logical address constitutes the solid-state that the upper layer software (applications)s such as operating system are perceived and deposits
Store up the memory space of equipment.Physical address is the address for accessing the physical memory cell of solid storage device.In related skill
Also implement address of cache using intermediate address form in art.Such as logical address is mapped as intermediate address, and then will be intermediate
Address is further mapped as physical address.
The table structure for storing the map information from logical address to physical address is referred to as FTL table.FTL table is that solid-state is deposited
Store up the important metadata in equipment.The data item of usual FTL table has recorded the ground in solid storage device as unit of data page
Location mapping relations.
The storage equipment for supporting Key-Value (key-data, also referred to as " KV ") storage model, is provided based on key (Key)
Read operation (Get (Key)) and write operation (Put (Key, Value)).To execute write operation, host provides key to storage equipment
(Key) with data (Value), to write data into storage equipment, and using key as the index for the data being written.To execute reading
Operation, host provide key to storage equipment, store equipment according to key and find data, and serve data to host.Thus in KV
In storage system, key is the index for accessing data, and data (Value) are accessed data.Generally, key and data
Length be not fixed length.And optionally, in order to reduce complexity, the length of key and/or data can have specified range.
Figure 1B illustrates the signal of the address conversion system of the solid storage device of the support KV storage model of the prior art
Figure.The address conversion system (also referred to as FTL table) of the solid storage device of KV storage model is supported to provide from key (K) to logically
The mapping of location or physical address (for example, PPA, Physical Page Address, physical page address).Optionally, in addition to physics
Page address, the physical address recorded in FTL table can be the initial address of the data frame (Data frame) inside Physical Page.Number
It is the data cell with fixed size according to frame, Physical Page includes one or more data frames.In response to obtaining key (K), key is used
(K) it is used as search index FTL table, obtains corresponding logical address or physical address.Optionally, the key of random length is subjected to Hash
Operation obtains the Hash keys (for example, 4 bytes/8 bytes) of fixed length, the index as FTL table.
Summary of the invention
According to a first aspect of the present application, the first distribution KV storage system according to the application first aspect is provided,
Including client computer and multiple KV storage devices, client computer is coupled with KV storage device by network；Each KV storage device includes:
KV stores equipment and calculates equipment；It calculates equipment to be coupled with KV storage equipment, client computer and multiple KV storage devices pass through net
Network coupling.
The first distribution KV storage system according to a first aspect of the present application, provides according to the application first aspect
Second distribution KV storage system further includes cold data storage equipment.
The second distribution KV storage system according to a first aspect of the present application, provides according to the application first aspect
Third distribution KV storage system, further includes data acquisition equipment.
Third distribution KV storage system according to a first aspect of the present application, provides according to the application first aspect
4th distribution KV storage system, cold data storage equipment are used to record the initial data of data acquisition equipment capture.
One of first to fourth distribution KV storage system according to a first aspect of the present application, provides according to the application
5th distribution KV storage system of first aspect calculates integration of equipments in KV storage equipment.
One of first to the 5th distribution KV storage system according to a first aspect of the present application, provides according to the application
6th distribution KV storage system of first aspect includes that cold data stores equipment in one or more KV storage devices.
The 6th distribution KV storage system according to a first aspect of the present application, provides according to the application first aspect
7th distribution KV storage system, the KV storage equipment in KV storage device store equipment with corresponding one or more cold datas
It is coupled by network.
The 7th distribution KV storage system according to a first aspect of the present application, provides according to the application first aspect
8th distribution KV storage system, KV storage equipment in record<key, value>to cold data corresponding with its store equipment phase
Association, the original data storage location recorded in " value " are located at the corresponding cold data storage equipment.
One of first to the 8th distribution KV storage system according to a first aspect of the present application, provides according to the application
9th distribution KV storage system of first aspect, the searching request of client computer are sent to one or more calculating by network
Equipment.
One of first to the 9th distribution KV storage system according to a first aspect of the present application, provides according to the application
Tenth distribution KV storage system of first aspect calculates equipment according to searching request and generates corresponding key, and certainly with key access
KV belonging to oneself stores equipment.
One of first to the tenth distribution KV storage system according to a first aspect of the present application, provides according to the application
11st distribution KV storage system of first aspect mentions if having recorded the corresponding value of same key in KV storage equipment to client computer
For value.
The 11st distribution KV storage system according to a first aspect of the present application, provides according to the application first aspect
The 12nd distribution KV storage system, calculate equipment or client computer also according to the original data storage location recorded in value, visit
Ask cold data storage equipment to obtain initial data.
One of first to the 12nd distribution KV storage system according to a first aspect of the present application, provides according to this Shen
Please first aspect the 13rd distribution KV storage system, further include the KV database for being coupled to network.
The 13rd distribution KV storage system according to a first aspect of the present application, provides according to the application first aspect
The 14th distribution KV storage system, KV database root generates corresponding key according to searching request, and record belonging to the key <
Key, value > store or respond to by which or which KV storage equipment.
The the 13rd or the 14th distribution KV storage system according to a first aspect of the present application, provides according to the application
15th distribution KV storage system of first aspect, the searching request of client computer are sent to KV database by network.
The 15th distribution KV storage system according to a first aspect of the present application, provides according to the application first aspect
The 16th distribution KV storage system, KV database root generates corresponding key according to searching request and obtains which KV stores equipment
It is able to respond the inquiry to key.
The 16th distribution KV storage system according to a first aspect of the present application, provides according to the application first aspect
The 17th distribution KV storage system, KV storage equipment sequencing table in have recorded with key be index entry.
One of the 15th to the 17th distribution KV storage system according to a first aspect of the present application is provided according to this
Apply for the 18th distribution KV storage system of first aspect, the searching request of client computer is transmitted to respond and look by KV database
The KV of inquiry stores equipment, or according to the instruction of KV database, and searching request is then forwarded to the KV that can respond inquiry by client computer
Store equipment.
One of second to the 18th distribution KV storage system according to a first aspect of the present application, provides according to this Shen
Please first aspect the 19th distribution KV storage system, in response to update distribution KV storage system in cold data storage set
It is standby, the cold data storage equipment being added is authenticated.
The 19th distribution KV storage system according to a first aspect of the present application, provides according to the application first aspect
The 20th distribution KV storage system, for by certification cold data store equipment, by cold data store equipment list with
The cold data storage equipment for being identified by certification is newly added cold data storage equipment or stores equipment to old cold data
Replacement.
The the 19th or the 20th distribution KV storage system according to a first aspect of the present application, provides according to the application
21st distribution KV storage system of first aspect, if storing equipment by the cold data authenticated is newly added cold data
Equipment is stored, the initial data from data acquisition equipment is written to.
The 21st distribution KV storage system according to a first aspect of the present application, provides according to the application first party
The 22nd distribution KV storage system in face is written into initial data in response to the cold data storage equipment of the new addition, also
According to the possible search generation<key to initial data, value>corresponding with the cold data storage equipment of the new addition to being recorded in
KV is stored in equipment.
The 21st distribution KV storage system according to a first aspect of the present application, provides according to the application first party
The 23rd distribution KV storage system in face is allocated for access storage in response to the cold data storage equipment being newly added
The global address of initial data, also by KV store the initial data address of reference recorded in equipment belong to it is assigned globally
Location<key, value>to deletion.
The 20th distribution KV storage system according to a first aspect of the present application, provides according to the application first aspect
The 24th distribution KV storage system, in response to having existed and being deposited by the cold data of certification in cold data storage equipment list
Equipment is stored up, stores Raw Data Generation<key for recording in equipment, value>right with the cold data by certification, and use<key, be worth>
Equipment is stored to the first KV is updated.
The 24th distribution KV storage system according to a first aspect of the present application, provides according to the application first party
The 25th distribution KV storage system in face, if before updating the first KV storage equipment, having in the first KV storage equipment to this
The reference of the initial data recorded in equipment is stored by the cold data of certification, then updates in the first KV storage equipment and refer to this
The record that the initial data recorded in equipment is stored by the cold data of certification, so that storing in the record through the cold of certification
Data storage device relevant information.
The 24th distribution KV storage system according to a first aspect of the present application, provides according to the application first party
The 26th distribution KV storage system in face, if there is no set to the cold data storage by certification in the first KV storage equipment
The reference of first initial data of standby middle record, also according to the possible search generation<key to the first initial data, value>right will
Generate<key, value>to be recorded in the 2nd KV store equipment in.
One of second to the 18th distribution KV storage system according to a first aspect of the present application, provides according to this Shen
Please first aspect the 27th distribution KV storage system, in response to distributed KV storage system addition cold data storage set
Standby, distributed KV storage system authenticates the cold data storage equipment being added；Pass through in response to cold data storage equipment
Certification will pass through the cold data storage device records of certification in waiting list；For being newly added to distributed KV storage system
In cold data store equipment, KV, which is stored in equipment, leads to invalid < key due to the cold data storage equipment that is newly added, value
> to deletion；For the initial data that the cold data storage equipment of the new addition is written into, searched according to the possible of initial data
Suo Shengcheng<key, value>it is right, and by generation<key, value>to be recorded in KV store equipment in.
The 27th distribution KV storage system according to a first aspect of the present application, provides according to the application first party
The 28th distribution KV storage system in face is recorded in if the cold data storage equipment being added into is unauthenticated
In set to be deleted.
The 27th distribution KV storage system according to a first aspect of the present application, provides according to the application first party
The 29th distribution KV storage system in face, storing equipment in response to the cold data by authenticating is to store to be in distributed KV
The cold data storage equipment once occurred in system；It traverses KV and stores equipment, obtain each<key, value>right from KV storage equipment；
If<key, value>pair value in the initial data that records be stored in the storage equipment of the cold data in waiting list, according to
Cold data storage equipment in waiting list, which updates, is somebody's turn to do<key, value>right.
One of the 27th to the 29th distribution KV storage system according to a first aspect of the present application, provides root
According to the 30th distribution KV storage system of the application first aspect, during distributed KV storage system use, if based on the
One<key, value>pair value in record by cold data storage equipment storage initial data read failure, the original number will be stored
According to cold data storage device flag be invalid, and the cold data in KV storage equipment is stored into equipment invalid and leads to nothing
Effect<key, value>to deletion.
According to a second aspect of the present application, it provides and updates distribution KV storage system according to the first of the application second aspect
The method of cold data storage equipment in system, comprising: the cold data storage equipment being added is authenticated；To the cold data
It stores equipment and the initial data from data acquisition equipment is written, and stored in response to searching request from the cold data of the new addition
Equipment reads initial data.
First according to a second aspect of the present application updates the side of the cold data storage equipment in distribution KV storage system
Method provides the side that the cold data storage equipment in distribution KV storage system is updated according to the second of the application second aspect
Method, the cold data storage equipment being added, has been written to equipment authentication information.
Second according to a second aspect of the present application updates the side of the cold data storage equipment in distribution KV storage system
Method provides the side that the cold data storage equipment in distribution KV storage system is updated according to the third of the application second aspect
Method, equipment authentication information are calculated with the exclusive information of Encryption Algorithm and key pair equipment.
Third according to a second aspect of the present application updates the side of the cold data storage equipment in distribution KV storage system
Method provides the side that the cold data storage equipment in distribution KV storage system is updated according to the 4th of the application second aspect the
Method, Encryption Algorithm are rivest, shamir, adelman, and the exclusive information of equipment is the identifier or sequence symbol of equipment.
The cold data storage in first to fourth update distribution KV storage system according to a second aspect of the present application is set
One of standby method provides the cold data storage updated in distribution KV storage system according to the 5th of the application second aspect the
The method of equipment, distributed KV storage system is by rivest, shamir, adelman decrypted authentication information, to identify the certification after decryption
Whether information is consistent with the exclusive information of cold data storage equipment, and certification cold data stores equipment accordingly, and refuses not pass through
The cold data storage equipment of certification is added to distributed KV storage system.
The cold data storage in the first to the 5th update distribution KV storage system according to a second aspect of the present application is set
One of standby method provides the cold data storage updated in distribution KV storage system according to the 6th of the application second aspect the
The method of equipment stores equipment by the cold data of distributed KV storage system for storing equipment by the cold data of certification
Table is newly added cold data storage equipment with the cold data storage equipment for being identified by certification or stores to old cold data
The replacement of equipment.
The 6th according to a second aspect of the present application updates the side of the cold data storage equipment in distribution KV storage system
Method provides the side that the cold data storage equipment in distribution KV storage system is updated according to the 7th of the application second aspect the
Method, storing equipment in response to the cold data by authenticating is that newly added cold data stores equipment, updates cold data and stores equipment
Table.
The cold data storage in the first to the 7th update distribution KV storage system according to a second aspect of the present application is set
One of standby method provides the cold data storage updated in distribution KV storage system according to the 8th of the application second aspect the
The method of equipment stores equipment using the cold data being newly added in response to KV storage device, is written to and sets from data acquisition
Standby initial data, and initial data is read from the cold data of new addition storage equipment in response to searching request.
The cold data storage in the first to the 8th update distribution KV storage system according to a second aspect of the present application is set
One of standby method provides the cold data storage updated in distribution KV storage system according to the 9th of the application second aspect the
The method of equipment, in response to the new addition cold data storage equipment be written into initial data, also according to initial data can
Search generation<the key of energy, the value>corresponding KV of cold data storage equipment being recorded in the new addition is stored in equipment.
The 9th according to a second aspect of the present application updates the side of the cold data storage equipment in distribution KV storage system
Method provides the side that the cold data storage equipment in distribution KV storage system is updated according to the tenth of the application second aspect the
Method is allocated for the global address of the initial data of access storage in response to the cold data storage equipment being newly added, also by KV
The initial data address of reference recorded in storage equipment belong to assigned global address<key, value>to deletion.
The cold data storage in the first to the tenth update distribution KV storage system according to a second aspect of the present application is set
One of standby method is provided and is deposited according to the cold data in the 11st of the application second aspect the update distribution KV storage system
The method for storing up equipment, if the cold data storage device records that are activated initial data, is searched according to the possible of initial data
Suo Shengcheng<key, value>it is right, by generation<key, value>to be recorded in KV store equipment in.
First according to a second aspect of the present application updates the side of the cold data storage equipment in distribution KV storage system
Method provides the side that the cold data storage equipment in distribution KV storage system is updated according to the 12nd of the application second aspect the
Method stores equipment by the first cold data of certification in response to having existed in cold data storage equipment list, with the first cold data
Raw Data Generation<key for recording in storage equipment, value>right, and by generated<key, value>set to the first KV storage is written
It is standby.
The 12nd according to a second aspect of the present application updates the cold data storage equipment in distribution KV storage system
Method provides the cold data storage equipment updated in distribution KV storage system according to the 13rd of the application second aspect the
Method, the first KV of traversal store equipment, and each initial data recorded in inquiry the first cold data storage equipment is stored in the first KV
Whether it is cited in equipment.
The cold data in the 12nd or the 13rd update distribution KV storage system according to a second aspect of the present application is deposited
The method for storing up equipment is provided and is deposited according to the cold data in the 14th of the application second aspect the update distribution KV storage system
The method for storing up equipment, if before updating the first KV storage equipment, there is the cold data to this by certification in the first KV storage equipment
The reference of the initial data recorded in storage equipment then updates in the first KV storage equipment and refers to the cold data by certification
The record of the initial data recorded in storage equipment stores equipment correlation by the cold data of certification so that storing in the record
Information.
The cold data in the 12nd or the 13rd update distribution KV storage system according to a second aspect of the present application is deposited
The method for storing up equipment is provided and is deposited according to the cold data in the 15th of the application second aspect the update distribution KV storage system
The method for storing up equipment, if former there is no storing record in equipment first to the cold data by certification in the first KV storage equipment
The reference of beginning data, also according to the possible search generation<key to the first initial data, value>right, by generation<key, value>right
It is recorded in the 2nd KV storage equipment.
The 15th according to a second aspect of the present application updates the cold data storage equipment in distribution KV storage system
Method provides the cold data storage equipment updated in distribution KV storage system according to the 16th of the application second aspect the
Method also authenticates the initial data not being cited in the first KV storage equipment to it.
The 16th according to a second aspect of the present application updates the cold data storage equipment in distribution KV storage system
Method provides the cold data storage equipment updated in distribution KV storage system according to the 17th of the application second aspect the
Method, cold data store data segment authentication information also corresponding for every original data record in equipment.
The 17th according to a second aspect of the present application updates the cold data storage equipment in distribution KV storage system
Method provides the cold data storage equipment updated in distribution KV storage system according to the 18th of the application second aspect the
Method, only to by certification initial data obtain the 2nd KV store equipment in record refer to the initial data < key,
Value>right, and updated according to new cold data storage equipment and be somebody's turn to do<key, value>right.
According to the third aspect of the application, provide according to the first of the application third aspect to distributed KV storage system
The method for adding cold data storage equipment, comprising: distributed KV storage system recognizes the cold data storage equipment being added
Card；In response to cold data storage equipment by certification, the cold data storage device records of certification will be passed through in waiting list；It is right
Equipment is stored in the cold data being newly added in distributed KV storage system, KV is stored into the cold data in equipment due to being newly added
Storage equipment and lead to invalid<key, value>to deletion；The original number being written into for the cold data storage equipment of the new addition
According to according to the possible search generation<key to initial data, value>right.
The method for storing equipment to distributed KV storage system addition cold data according to the first of the third aspect of the application,
The second method for storing equipment to distributed KV storage system addition cold data according to the application third aspect is provided, by root
According to initial data possible search generate<key, value>to be recorded in KV store equipment in.
Equipment is stored to distributed KV storage system addition cold data according to the first or second of the third aspect of the application
Method, provide according to the third of the application third aspect to the side of distributed KV storage system addition cold data storage equipment
Method is recorded in set to be deleted if the cold data storage equipment being added into is unauthenticated.
The method that cold data stores equipment is added to distributed KV storage system according to the third of the third aspect of the application,
Provide the method that cold data storage equipment is added according to the four-way distribution KV storage system of the application third aspect, cold number
There is global address according to storage equipment, also by KV store the initial data address of reference recorded in equipment belong to be assigned to
The global address of cold data storage equipment in deletion set<key (K), value (V)>to deletion.
Equipment is stored to distributed KV storage system addition cold data according to the first to fourth of the third aspect of the application
One of method, provide and cold data storage equipment added according to the five-way distribution KV storage system of the application third aspect
Method, in response to by authenticate cold data store equipment be that the cold data once occurred in distributed KV storage system is deposited
Store up equipment；It traverses KV and stores equipment, obtain each<key, value>right from KV storage equipment；If<key, value>pair value in record
It is stored in the storage equipment of the cold data in waiting list by the initial data of cold data storage equipment storage, according to etc.
It is updated to the cold data storage equipment in queue and is somebody's turn to do<key, value>right.
The method that cold data stores equipment is added according to the five-way distribution KV storage system of the third aspect of the application,
The method that cold data storage equipment is added according to the six-way distribution KV storage system of the application third aspect is provided, if <
Key, value>pair value in record cold data storage equipment be deleted or be recorded in set to be deleted, delete should<key, be worth
> right.
The method that cold data stores equipment is added according to the five-way distribution KV storage system of the third aspect of the application,
The method that cold data storage equipment is added according to the seven-way distribution KV storage system of the application third aspect is provided, if <
Key, value>pair value in record by cold data storage equipment storage initial data do not existed, delete should<key, value>it is right.
The method that cold data stores equipment is added according to the five-way distribution KV storage system of the third aspect of the application,
The 8th method for storing equipment to distributed KV storage system addition cold data according to the application third aspect is provided, if <
Key, value > pair value in record by cold data storage equipment storage initial data be stored in it is cold in waiting list
In data storage device, to this<key, the value>initial data that stores in corresponding cold data storage equipment is authenticated.
The method for storing equipment to distributed KV storage system addition cold data according to the 8th of the third aspect of the application the,
Provide the 9th method for storing equipment to distributed KV storage system addition cold data according to the application third aspect, response
In initial data by certification, just to this<key, value>to being updated.
The method for storing equipment to distributed KV storage system addition cold data according to the 8th of the third aspect of the application the,
Provide the tenth method for storing equipment to distributed KV storage system addition cold data according to the application third aspect, response
It is unauthenticated in initial data, the initial data for deleting authentification failure is corresponding<key, value>right.
Equipment is stored to distributed KV storage system addition cold data according to the first to the tenth of the third aspect of the application
One of method, provide and set according to the 11st of the application third aspect the to the addition cold data storage of distributed KV storage system
Standby method, during distributed KV storage system use, if based on certain<key, value>pair value in record stored by cold data
The initial data of equipment storage reads failure, deletes and is somebody's turn to do<key, value>right.
Equipment is stored to distributed KV storage system addition cold data according to the first to the tenth of the third aspect of the application
One of method, provide and set according to the 12nd of the application third aspect the to the addition cold data storage of distributed KV storage system
Standby method, during distributed KV storage system use, if be based on first<key, value>pair value in record deposited by cold data
The initial data for storing up equipment storage reads failure, is invalid by the cold data for storing initial data storage device flag, and
KV is stored into cold data storage equipment invalid in equipment and leads to invalid<key, value>to deletion.
It is stored according to the 11st or the 12nd of the third aspect of the application the to distributed KV storage system addition cold data
The method of equipment is provided and is set according to the addition cold data storage of the tenth three-dimensional distribution KV storage system of the application third aspect
Standby method, if the frequency of initial data reading failure is more than threshold value, traversal KV stores each<key of equipment, be worth>right,
Obtain the cold data storage equipment that can not be accessed.
The side of cold data storage equipment is added according to the tenth three-dimensional distribution KV storage system of the third aspect of the application
Method provides the side that cold data storage equipment is added according to the tenth four-way distribution KV storage system of the application third aspect
Method is needed replacing cold in response to identifying that one or more cold data storage equipment can not be used and make warning with prompt
Data storage device.
Detailed description of the invention
In order to illustrate the technical solutions in the embodiments of the present application or in the prior art more clearly, to embodiment or will show below
There is attached drawing needed in technical description to be briefly described, it should be apparent that, the accompanying drawings in the following description is only this
The some embodiments recorded in application can also be obtained according to these attached drawings other for those of ordinary skill in the art
Attached drawing.
Figure 1A is the block diagram of solid storage device in the prior art；
Figure 1B is the schematic diagram of the address conversion system of the solid storage device of the prior art；
Fig. 2 is the block diagram according to the generation system of the key and value of the KV of the embodiment of the present application storage；
Fig. 3 is the schematic diagram according to the machine learning component of the embodiment of the present application；
Fig. 4 is to store the schematic diagram that equipment responds retrieval request by KV according to the embodiment of the present application；
Fig. 5 is the schematic diagram according to the distributed KV storage system of the embodiment of the present application；
Fig. 6 is the schematic diagram according to the distributed KV storage system of the another embodiment of the application；
Fig. 7 is the schematic diagram for updating cold data storage equipment in distributed KV storage system according to the embodiment of the present application；
Fig. 8 is the flow chart according to the embodiment of the present application to distributed KV storage system addition cold data storage equipment.
Specific embodiment
Below with reference to the attached drawing in the embodiment of the present application, technical solutions in the embodiments of the present application carries out clear, complete
Ground description, it is clear that described embodiment is some embodiments of the present application, instead of all the embodiments.Based on the application
In embodiment, those skilled in the art's every other embodiment obtained without making creative work, all
Belong to the range of the application protection.
Embodiment one
Fig. 2 is the block diagram according to the generation system of the key and value of the KV of the embodiment of the present application storage.
Initial data is the unstructured data such as video, picture, text file.
Structuring processing is carried out to initial data.For example, label is added to initial data, to indicate coming for initial data
The structured messages such as source, format, storage location, required access authority；The content of initial data is analyzed, such as from view
Extractions, thumbnail, abstract in frequency or picture, extract portrait, the objects such as automobile, extraction human face's feature, height, gender,
The features such as age extract the features such as automobile brand, license plate.Structuring is all used as with the associated label of initial data, feature etc.
Handle the structured message generated.Structured message is easy for retrieval, passes through the database or search engine skill of the prior art
Art or other search techniques existing or occur in the future retrieve structured message.
The structured message that structuring processing module 210 generates includes a variety of searchable elements (for example, video or picture
Contained in object, the feature of object, the keyword etc. in document).User can retrieve these elements.For example, user
Retrieval includes the picture or video of personage, and further specifies that facial characteristics, height, the gender of personage etc. as searched targets
Feature.According to an embodiment of the present application, retrieval type generation module 220 is wanted according to the retrieving of providing of structuring processing module 210
Element and combinations thereof generates a variety of possible retrieval types.It include the one or more retrieval elements being retrieved in retrieval type.
Optionally, or further, retrieval type generation module 220 also obtains the retrieval behavior of user, from the retrieval of user
Retrieval type described in user is obtained in behavior.
Retrieval type generation module 220 implements retrieval using the retrieval type generated.Such as by being mentioned to search engine, database
Implement to retrieve for retrieval type.Search engine or database 230 are according to retrieval type generation search result.Optionally, some search
Engine/database 230 supports the retrieval modes such as fuzzy search, semantic retrieval, image retrieval.Retrieval type generation module 220 also makes
Implement to retrieve with these retrieval modes, and the retrieval command provided in retrieving to search engine/database 230 will be provided
Referred to as retrieval type.
The retrieval type for implementing retrieval is also provided to machine learning component 240 by retrieval type generation module 220.Machine learning portion
Part 240 is the machine learning component of such as prior art or the machine learning component that future occurs.
Machine learning component 240 is using the retrieval type that retrieval type generation module 220 provides as input.Optionally, it will retrieve
Each retrieval element of formula is respectively supplied to each input node of machine learning component 240.Machine learning component 240 is according to quilt
The retrieval type of input generates the marking to retrieval type as output.Optionally, marking is as the evaluation to retrieval type superiority and inferiority.Example
Such as, relatively high marking, it is meant that retrieval type has a preferable retrieval effectiveness, and relatively low marking, it is meant that retrieval type have compared with
Bad retrieval effectiveness is not to be proposed use.Preferable retrieval effectiveness, for example, can effectively be obtained rapidly from search result
Searched targets, the irrelevant contents in search result are less；Conversely, then meaning worse retrieval effectiveness.
Optionally, machine learning component 240 is trained to generate similar beat to the retrieval type with similar to search result
Point.For example, the evaluation according to the search result for implementing to retrieve as the marking of the retrieval type exported to machine learning component 240,
So that the retrieval type of similar search result, marking as 240 output phase of machine learning component.
In the learning process of machine learning component 240, using retrieval type as input, marking is generated according to input, to beat
The evaluation for dividing evaluation module 250 to provide is as the feedback to generated marking.Machine learning component 240 updates it according to evaluation
The weight of internal node, gradually to generate the marking of the more acurrate evaluation retrieval type superiority and inferiority of energy.
It is input to the retrieval type of machine learning component 240, is also supplied to search engine/database 230.Search engine
Or database 230 generates search result according to retrieval type.Evaluation module 250 of giving a mark is generated according to search engine or database 230
Search result marking that machine learning component 240 is generated evaluate, and evaluation is supplied to machine learning component 240.
Optionally, evaluation to retrieval type is obtained by manually marking, or is implemented according to user using search engine or database 230
The evaluation of subsequent behavior after retrieval as the marking of the retrieval type exported to machine learning component 240, and evaluation is supplied to
Machine learning component 240 is (for example, user clicks one or more search results, it is meant that retrieval type has preferable retrieval
Effect, and user ignores all or most of search result that search engine or database 230 provide, it is meant that retrieval type tool
There is poor retrieval effectiveness).
In another embodiment, marking evaluation module 250 also implements the inspection of retrieval to search engine or database 230
The classification of hitch fruit, different classification correspond to different values, the difference given a mark with the value and machine learning component 240 that correspond to classification
The evaluation of marking as the retrieval type exported to machine learning component 240, and evaluation is supplied to machine learning component 240.
For experienced the machine learning component 240 of learning process, for the retrieval type (being denoted as s) of input, machine learning
The calculated result of the internal node of component 240 is provided to key and generates (K generation) module 260.K generation module 260 is according to machine
The calculated result for learning each node that component 240 provides generates the key (K) that equipment is stored for KV.Key (K) corresponds to retrieval type
(s)。
Search engine/database 230 handles the result that retrieval type (s) is obtained and is provided to value generation (V generation) module
270.V generation module 270 generates the value (V) for KV storage equipment according to the result that search engine/database 230 provides.Value
(V) retrieval type (s) is also corresponded to.
The key (K) for corresponding to identical retrieval type (s) and value (V) are supplied to KV storage equipment, remembered in KV storage equipment
Record key (K) and value (V).
Fig. 3 is the schematic diagram according to the machine learning component of the embodiment of the present application.
Machine learning component 240 includes the artificial neural network being for example made of multilayer node.The artificial neuron that Fig. 2 is shown
Network includes 4 layers of (such as L0Layer, L1Layer, L2Layer and L3Layer), L0Layer is input layer, L3Layer is output layer, L1Layer and L2Layer is internal
Layer.Each layer includes multiple nodes.As an example, L0Layer includes node C1、C2……C5, L1Layer includes node 0, node 1, node 2
With node 3, L2Layer includes node 4, node 5, node 6 and node 7, output layer L3Including node 8 and node 9.
Input layer L0Each node receive input retrieval type.For example, the retrieval that each node receives composition retrieval type is wanted
One of element.
Output layer L3Each node value instruction machine learning component 240 output (marking to the retrieval type of input).
Each node on behalf of interior layer (is denoted as N to the value that node with its coupling is calculatedn, wherein n indicates section
Serial number of the point in Fig. 3).As an example:
N0=C1*F0(C1)+C2*F0(C2)+C3*F0(C3)+C4*F0(C4)+C5*F0(C5)
N1=C1*F1(C1)+C2*F1(C2)+C3*F1(C3)+C4*F1(C4)+C5*F1(C5), wherein
C1~C5Indicate node C1~C5Respective received input (being quantified as being worth), F0() indicates with the associated letter of node 0
Number, F1() indicates with the associated function of node 1.For other nodes in artificial neural network, saved in a similar way
The corresponding value of point.
To which in response to specified input, (for example, retrieval type s), node 0 to node 9 will generate respective value and (remember respectively
For N0~N9)。
Optionally, each node of the neural network of the prior art obtains the value of node according to a variety of known functions.
Remember (N0,N1,N2,N3) it is to nodal value N0、N1、N2With N3Connection.For example, if nodal value N0~N3Respectively count
Word 0,1,2 and 3, then (N0,N1,N2,N3) it is Serial No. " 0123 ".According to an embodiment of the present application, to artificial neural network
The nodal value of each layer is separately connected, and obtains the corresponding sequence of each layer of same artificial neural network.For example, L1The corresponding sequence of layer be
(N0,N1,N2,N3), L2The corresponding sequence of layer is (N4,N5,N6,N7), and L3The corresponding sequence of layer is (N8,N9)。
Hash calculation is carried out to sequence corresponding with each layer respectively, obtains the corresponding cryptographic Hash with each layer.For example, same L1Layer
Corresponding cryptographic Hash is Hash ((N0,N1,N2,N3)), same to L2The corresponding cryptographic Hash of layer is Hash ((N4,N5,N6,N7)), same to L3Layer
Corresponding cryptographic Hash is Hash ((N8,N9)).Optionally, the corresponding cryptographic Hash of each layer length having the same.
One or more layers corresponding cryptographic Hash with artificial neural network is connected, as the key for being supplied to KV storage equipment
(K).For example, the L of connection artificial neural network2-L4The corresponding cryptographic Hash of layer, obtains (Hash ((N0,N1,N2,N3)),Hash
((N4,N5,N6,N7)),Hash((N8,N9))) as the key (K) for being supplied to KV storage equipment.Optionally, to same artificial neural network
The connection result of one or more layers corresponding cryptographic Hash of network is (for example, (Hash ((N0,N1,N2,N3)),Hash((N4,N5,N6,
N7)),Hash((N8,N9)))) Hash calculation is carried out again, using obtained result as the key (K) for being supplied to KV storage equipment, make
The length for obtaining key (K) is shortened or has designated length.
Referring also to Fig. 2, K generation module 160 is supplied to according to the generation of the value of one or more layers node of artificial neural network
The key (K) of KV storage equipment.
Referring back to Fig. 2, the corresponding pairs of key (K) of same retrieval type (s) and value (V) are had recorded in KV storage equipment.
And KV storage equipment exports corresponding value (V) conduct of same key (K) and asks to inquiry in response to receiving the inquiry request to key (K)
The response asked.
Embodiment two
Fig. 4 is to store the schematic diagram that equipment responds retrieval request by KV according to the embodiment of the present application.
User provides the retrieval type for searching for database or search engine, and retrieval type includes that one or more retrievals are wanted
Element.Optionally, retrieval type is analyzed, retrieval element is extracted from retrieval type.
Retrieval element is supplied to machine learning component 240.Referring also to such as Fig. 3, each retrieval element is provided respectively
To each input node of the artificial neural network of machine learning component 240.The artificial neural network of machine learning component 240 it is each
A node generates respective nodal value in response to the retrieval element of input.Nodal value is provided to K generation module 260.According to knot
The K generation module 260 for closing Fig. 2 and Fig. 3 description generates the mode of key (K), and the K generation module 260 of Fig. 4 is according to the node being provided
Value generates key (K).
Key (K) generated is provided to KV storage equipment, for reading the corresponding value of same key (K) from KV storage equipment
(V).The corresponding structured message of same key (K) is had recorded in value (V).It further include the original for generating structured message in structured message
The storage location of beginning data.Use the corresponding structured message of same key (K) obtained from KV storage equipment as the response to search.
Optionally, the result to user's display structure information as search.It as an example, include being searched according to user in structured message
The thumbnail of image or video that the feature of rope is hit.These thumbnails are shown to user, in order to needed for user's identification
Search for target.Further selection in response to user to search target obtains corresponding with search target from structured message
Original data storage location, and initial data is obtained to be supplied to user.
Optionally, also according to the original data storage location recorded in structured message, before user further selects
And initial data is loaded in advance.To accelerate further to select user the response speed of search target.
Embodiment three
Fig. 5 is the schematic diagram according to the distributed KV storage system of the embodiment of the present application.The reality shown according to the application Fig. 5
The distributed KV storage system for applying example includes cold data storage equipment (for example, the mechanical hard disk HDD of the prior art, solid-state drive
Device SSD) with according to the KV of the embodiment of the present application store equipment, further include using distributed KV storage system client computer (for example,
Computer, mobile terminal or server).
The cold data storage same data acquisition equipment of equipment (for example, camera or video monitoring equipment) is coupled, for remembering
Record the initial data of data acquisition equipment capture.
Calculating equipment with machine learning component is coupled with according to the KV of the embodiment of the present application storage equipment.It is optional
Ground calculates integration of equipments and stores in equipment to according to the KV of the embodiment of the present application.
The various pieces of distributed KV storage system are coupled by network.
According to the embodiment of Fig. 5, KV stores equipment and corresponds with cold data storage equipment.For example, KV storage equipment is same
Corresponding cold data storage equipment is coupled by direct-connected data path.As shown in figure 5, KV storage equipment 510 is deposited with cold data
It is corresponding to store up equipment 520, it is corresponding with cold data storage equipment 540 that KV stores equipment 530.Recorded in KV storage equipment < key (K),
Value (V) > be associated to cold data corresponding with its storage equipment, the original data storage location recorded in " value (V) " is located at
The corresponding cold data stores equipment.
As an example, client computer is passed through net to the searching request for storing equipment according to the distributed KV of the embodiment of the present application
Network (such as broadcast) is sent to each calculating equipment.Equipment is calculated to generate corresponding key (K) according to searching request and visited with key (K)
Ask the KV storage equipment belonging to oneself.If the corresponding value (V) of same key (K) is had recorded in KV storage equipment, to client computer offer value
(V).And equipment or client computer are optionally calculated also according to the original data storage location recorded in value (V), access cold data
Equipment is stored to obtain initial data.
Example IV
Fig. 6 is the schematic diagram according to the distributed KV storage system of the another embodiment of the application.
Compared with the embodiment that Fig. 5 is shown, the distributed KV storage system of Fig. 6 further includes being coupled to the KV data of network
Library.KV database root generates corresponding key (K) according to searching request, and has recorded belonging to the key (K)<key, and value>to by which
Or which KV storage equipment stores or response.
As an example, client computer is passed through net to the searching request for storing equipment according to the distributed KV of the embodiment of the present application
Network is sent to KV database.KV database root generates corresponding key (K) according to searching request and obtains which KV storage equipment can
Respond the inquiry (for example, being had recorded in its sequencing table with key (K) is the entry indexed) to key (K).KV database is by client computer
Searching request be transmitted to the KV storage equipment that can respond inquiry, or according to the instruction of KV database, client computer asks search
It asks and is then forwarded to the KV storage equipment that can respond inquiry.If the corresponding value (V) of same key (K) is had recorded in KV storage equipment, to visitor
Family machine offer value (V).And optionally calculate equipment or client computer also according to the original data storage location recorded in value (V),
Cold data storage equipment is accessed to obtain initial data.
According to the embodiment of Fig. 6, it is many-one relationship that KV, which stores equipment with cold data storage equipment,.For example, KV storage is set
It is standby to be coupled with corresponding one or more cold data storage equipment by network.As shown in fig. 6, KV storage equipment 610 corresponds to
Cold data stores equipment 620 and cold data and stores equipment 630, KV store equipment 640 correspond to cold data storage equipment 65 with
Cold data stores equipment 660.Recorded in KV storage equipment<key (K), value (V)>store equipment to cold data corresponding with its
Associated, the original data storage location recorded in " value (V) " is located at the corresponding cold data storage equipment.
Embodiment five
Fig. 7 is the schematic diagram for updating cold data storage equipment in distributed KV storage system according to the embodiment of the present application.
Under some cases, need to update cold data storage equipment.For example, being added into distributed KV storage system new
Data acquisition equipment, or needed replacing in the cold data storage device fails used.
It is added to and equipment is stored according to the cold data of the distributed KV storage system of the embodiment of the present application, have been written to set
Standby authentication information (referring to Fig. 7, equipment authentication information 0, equipment authentication information 1).Equipment authentication information is calculated with specified encryption
Method is calculated with exclusive information (such as identifier, sequence number etc.) of the key (such as rivest, shamir, adelman) to equipment
's.
Distributed KV storage system to the cold data storage equipment (referring to Fig. 7, new cold data stores equipment) being added into
Row certification (Fig. 7, by 1. indicated).For example, distribution KV storage system by rivest, shamir, adelman decryption be added it is cold
Authentication information in data storage device, to identify that the authentication information after decryption stores the exclusive of equipment with the cold data being added
Whether information is consistent, and authenticates the cold data storage equipment being added accordingly, and refuse the unauthenticated cold number being added
Distributed KV storage system is added to according to storage equipment.
(ginseng can be implemented by the calculating equipment of KV database or KV storage equipment to the certification of the cold data storage equipment of addition
See Fig. 5 and Fig. 6).
For storing equipment by the cold data of certification, by the cold data of distributed KV storage system store equipment list with
The cold data storage equipment for being identified by certification is newly added cold data storage equipment or stores equipment to old cold data
The replacement of (Fig. 7, by 2. indicated).
If cold data stores the record for storing equipment in equipment list without the cold data that there are this by certification, it is meant that should
Storing equipment by the cold data of certification is that newly added cold data stores equipment, if storing equipment by the cold data authenticated is
Newly added cold data stores equipment, updates cold data storage equipment list (Fig. 7, by 3. indicated), and store in distributed KV
Cold data storage equipment (Fig. 7, by 4. indicated) is enabled in system, so that distribution KV storage equipment will use the new addition
Cold data store equipment, be written to the initial data from data acquisition equipment, and in response to searching request from this it is new plus
The cold data storage equipment entered reads initial data.And original number is written into response to the cold data storage equipment of the new addition
According to also according to the possible search generation<key (K) to initial data, value (V)>deposited to the cold data being recorded in the new addition
It stores up in the corresponding KV storage equipment of equipment.
Optionally, the cold data storage equipment being activated, is assigned to the global address for distributed KV storage system,
Global address is used to access the initial data of cold data storage equipment storage.In response to assigned global address, also KV is deposited
The initial data address of reference recorded in storage equipment belong to assigned global address<key (K), value (V)>to deletion, because
For these<key (K), value (V)>to the value (V) recorded is in vain.Still optionally, if the cold data storage being activated is set
Note initial data is worth (V)>right according to possible search generation<key (K) to initial data, by generation<key
(K), value (V) > to be recorded in KV storage equipment in.
If having existed the cold data storage equipment by certification in cold data storage equipment list, it is meant that should be by recognizing
The cold data storage equipment of card is old cold data storage equipment, raw with the initial data recorded in old cold data storage equipment
At<key (K), it is worth (V)>right, and use<key (K), value (V)>to updating KV storage equipment (Fig. 7, indicated by 5.).As an example,
It traverses KV and stores equipment, inquire each initial data recorded in old cold data storage equipment and (be denoted as D0) in KV storage equipment
Whether be cited (address of initial data).If there is the D to initial data in KV storage equipment0Reference (for example, having recorded original
Beginning data D0Address), then update KV storage equipment in refer to initial data D0Record, update and deposited in the value (V) of the record
The cold data of storage stores device-dependent message.If KV is stored in equipment, there is no the D to initial data0Reference (for example, record
Initial data D0Address), also according to initial data D0Possible search generation<key (K), value (V)>right, by generation
<key (K), value (V)>to be recorded in KV storage equipment (being denoted as KS1) in.
Optionally, KV database (referring to Fig. 6) also is updated, KV database is enabled to know that KV storage equipment KS1 is responded pair
The inquiry of key (K).
Optionally, the initial data D for not being cited in KV storage equipment0, (Fig. 7, by 6. is also authenticated to it
It is indicated).Cold data stores data segment authentication information also corresponding for every original data record in equipment.To cold number
Initial data is authenticated according to the certification of storage equipment similar mode.Only obtained to by the original data record of certification
Recorded in KV storage equipment refer to the initial data<key (K), value (V)>right, and equipment is stored according to new cold data
Update should<key (K), value (V)>right, to avoid in KV storage equipment<key (K), value (V)>to maliciously being updated.
Embodiment six
Fig. 8 is the flow chart according to the embodiment of the present application to distributed KV storage system addition cold data storage equipment.
For being authenticated to it first by new cold data storage equipment (810) that distribution KV storage system is added, with
It determines if that distributed KV storage system (820) can be added into.
Optionally, the cold data storage device records of certification will be passed through in waiting list.What is recorded in waiting list is cold
Data storage device will not be used, to prevent new cold data from storing equipment with other numbers recorded in distributed memory system
Conflict according to existing.
If the cold data storage equipment being added into is unauthenticated, it is recorded in queue to be deleted at it.
Optionally, cold data storage equipment has global address, and KV is also stored to the original number of the reference recorded in equipment
Belong to according to address the global address for the cold data storage equipment being assigned in queue to be deleted<key (K), value (V)>to deleting
It removes, because these<key (K), value (V)>to the value (V) recorded is in vain.
For storing equipment (for example, obtaining from waiting list) by the cold data of certification, identify that it is new cold number
According to storage equipment, or cold data storage equipment (830) once occurred in distributed KV storage system.
Equipment is stored for the cold data being newly added in distributed KV storage system, KV is stored in equipment due to newly adding
The cold data storage equipment that enters and lead to invalid<key (K), value (V)>to deletion (840).For example, cold data storage equipment tool
Have a global address, the addition of new cold data storage equipment so that stored in KV storage equipment corresponding to the cold number being newly added
According to storage equipment global address<key (K), value (V)>to becoming invalid.
For the initial data that the cold data storage equipment of the new addition is written into, searched according to the possible of initial data
Suo Shengcheng<key (K), value (V)>to (845), by newly-generated<key (K), value (V)>store in equipment (850) to being recorded in KV.
If storing equipment by the cold data authenticated is that the cold data once occurred in distributed KV storage system stores
Equipment, traversal KV store equipment, obtain each<key (K) from KV storage equipment, and value (V)>to (860).If < key (K), it is worth (V)
>pair value (V) in record cold data storage equipment be deleted or be recorded in queue to be deleted (865), delete be somebody's turn to do<key
(K), it is worth (V) > to (870).Optionally, if<key (K), value (V)>pair value (V) in record by cold data storage equipment storage
Initial data do not existed, delete should<key (K), value (V)>right.
If<key (K), value (V)>pair value (V) in record by cold data storage equipment storage initial data stored
In the cold data storage equipment being located in waiting list, is updated according to the cold data storage equipment in waiting list and be somebody's turn to do < key
(K), it is worth (V) > to (880).
Optionally, if<key (K), value (V)>pair value (V) in record by cold data storage equipment storage initial data
Be stored in waiting list cold data storage equipment in, to this<key (K), value (V)>to corresponding cold data store
The initial data stored in equipment is authenticated.Only the initial data by certification in the case where, just to this < key (K), value
(V)>to being updated, to avoid in KV storage equipment<key (K), value (V)>to maliciously being updated.Optionally, certification is deleted
The initial data of failure is corresponding<key (K), value (V)>right.
Still optionally, during distributed KV storage system use, if based on certain<key (K), value (V)>pair value (V)
The initial data for storing equipment storage by cold data of middle record reads failure, deletes and is somebody's turn to do<key (K), value (V)>right.Or it will deposit
It is invalid for storing up the cold data storage device flag of the initial data, and KV is stored in equipment because of cold data storage equipment
It is invalid and lead to invalid<key (K), value (V)>to deletion.Still optionally, if the frequency that initial data reads failure surpasses
Threshold value, each<key (K) of traversal KV storage equipment are crossed, value (V)>right obtains the cold data storage equipment that can not be accessed
(885).The cold data storage equipment that can not be used one or more makes warning, needs replacing cold data storage with prompt
Equipment (890).
Although the preferred embodiment of the application has been described, it is created once a person skilled in the art knows basic
Property concept, then additional changes and modifications may be made to these embodiments.So it includes excellent that the following claims are intended to be interpreted as
It selects embodiment and falls into all change and modification of the application range.Obviously, those skilled in the art can be to the application
Various modification and variations are carried out without departing from spirit and scope.If in this way, these modifications and variations of the application
Belong within the scope of the claim of this application and its equivalent technologies, then the application is also intended to encompass these modification and variations and exists
It is interior.
Claims (10)
1. a kind of distribution KV storage system, which is characterized in that including client computer and multiple KV storage devices, client computer is deposited with KV
Storage device is coupled by network；Each KV storage device includes: KV storage equipment and calculating equipment；Equipment is calculated to set with KV storage
Standby to be coupled, client computer is coupled with multiple KV storage devices by network.
2. distribution KV storage system as described in claim 1, which is characterized in that in response to updating distribution KV storage system
In cold data store equipment, to be added cold data storage equipment authenticate.
3. distribution KV storage system as described in claim 1, which is characterized in that in response to adding to distributed KV storage system
Cold data is added to store equipment, distributed KV storage system authenticates the cold data storage equipment being added；In response to cold number
According to storage equipment by certification, the cold data storage device records of certification will be passed through in waiting list；For being newly added to point
Cold data in cloth KV storage system stores equipment, and KV is stored in equipment and is led since the cold data being newly added stores equipment
Cause invalid<key, value>to deletion；For the initial data that the cold data storage equipment of the new addition is written into, according to original
The possible search generation<key of data, value>it is right, and by generation<key, value>to be recorded in KV store equipment in.
4. distribution KV storage system as claimed in claim 3, which is characterized in that in distributed KV storage system validity period
Between, if be based on first<key, value>pair value in record by cold data storage equipment storage initial data read fail, will deposit
It is invalid for storing up the cold data storage device flag of the initial data, and KV is stored to the nothing of cold data storage equipment in equipment
It imitates and leads to invalid<key, value>to deletion.
5. a kind of method of the cold data storage equipment updated in distribution KV storage system characterized by comprising
The cold data storage equipment being added is authenticated；
The initial data from data acquisition equipment is written to cold data storage equipment, and new from this in response to searching request
The cold data storage equipment of addition reads initial data.
6. method as claimed in claim 5, which is characterized in that stored in response to KV storage device using the cold data being newly added
Equipment is written to the initial data from data acquisition equipment, and deposits in response to searching request from the cold data of the new addition
It stores up equipment and reads initial data.
7. such as method described in claim 5 or 6, which is characterized in that write in response to the cold data storage equipment of the new addition
Enter initial data, also according to the possible search generation<key to initial data, value>to the cold data being recorded in the new addition
It stores in the corresponding KV storage equipment of equipment.
8. a kind of method to distributed KV storage system addition cold data storage equipment characterized by comprising
Distributed KV storage system authenticates the cold data storage equipment being added；
In response to cold data storage equipment by certification, the cold data storage device records of certification will be passed through in waiting list；
Equipment is stored for the cold data that is newly added in distributed KV storage system, KV is stored in equipment due to being newly added
Cold data storage equipment and lead to invalid<key, value>to deletion；
It is raw according to the possible search to initial data for the initial data that the cold data storage equipment of the new addition is written into
At<key, value>right.
9. method according to claim 8, which is characterized in that
Storing equipment in response to the cold data by authenticating is that the cold data once occurred in distributed KV storage system stores
Equipment；
It traverses KV and stores equipment, obtain each<key, value>right from KV storage equipment；
If<key, value>pair value in record by cold data storage equipment storage initial data be stored in waiting list
In cold data storage equipment in, according in waiting list cold data storage equipment update should<key, value>it is right.
10. method as claimed in claim 8 or 9, which is characterized in that during distributed KV storage system use, if being based on
Certain<key, value>pair value in record by cold data storage equipment storage initial data read failure, delete should<key, value>
It is right.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810332295.0A CN110377535B (en) | 2018-04-13 | 2018-04-13 | Distributed KV storage system |
| CN202011059548.5A CN112181872A (en) | 2018-04-13 | 2018-04-13 | Distributed KV memory system for edge computing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810332295.0A CN110377535B (en) | 2018-04-13 | 2018-04-13 | Distributed KV storage system |
Related Child Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011059548.5A Division CN112181872A (en) | 2018-04-13 | 2018-04-13 | Distributed KV memory system for edge computing |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110377535A true CN110377535A (en) | 2019-10-25 |
| CN110377535B CN110377535B (en) | 2020-11-06 |
Family
ID=68243315
Family Applications (2)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011059548.5A Pending CN112181872A (en) | 2018-04-13 | 2018-04-13 | Distributed KV memory system for edge computing |
| CN201810332295.0A Active CN110377535B (en) | 2018-04-13 | 2018-04-13 | Distributed KV storage system |
Family Applications Before (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011059548.5A Pending CN112181872A (en) | 2018-04-13 | 2018-04-13 | Distributed KV memory system for edge computing |
Country Status (1)
| Country | Link |
|---|---|
| CN (2) | CN112181872A (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102073742A (en) * | 2011-01-31 | 2011-05-25 | 清华大学 | Mass object storage system and running method thereof |
| CN102668457A (en) * | 2009-09-30 | 2012-09-12 | 柯蔼文 | Systems and methods for social graph data analytics to determine connectivity within a community |
| CN102955845A (en) * | 2012-10-23 | 2013-03-06 | 北京亿赞普网络技术有限公司 | Data access method and device as well as distributed database system |
| CN105814560A (en) * | 2013-12-09 | 2016-07-27 | 赛灵思公司 | Memory arrangement for implementation of high-throughput key-value stores |
| US9767107B1 (en) * | 2013-06-29 | 2017-09-19 | EMC IP Holding Company LLC | Parallel file system with metadata distributed across partitioned key-value store |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104112020A (en) * | 2014-07-25 | 2014-10-22 | 沈阳美行科技有限公司 | Frame type retrieval method for navigation equipment |
| CN105389190B (en) * | 2015-12-08 | 2019-05-24 | 华为技术有限公司 | A kind of method, apparatus and system of os starting |
| CN106294573A (en) * | 2016-07-28 | 2017-01-04 | Tcl集团股份有限公司 | A kind of mass data Query method in real time and system |
-
2018
- 2018-04-13 CN CN202011059548.5A patent/CN112181872A/en active Pending
- 2018-04-13 CN CN201810332295.0A patent/CN110377535B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102668457A (en) * | 2009-09-30 | 2012-09-12 | 柯蔼文 | Systems and methods for social graph data analytics to determine connectivity within a community |
| CN102073742A (en) * | 2011-01-31 | 2011-05-25 | 清华大学 | Mass object storage system and running method thereof |
| CN102955845A (en) * | 2012-10-23 | 2013-03-06 | 北京亿赞普网络技术有限公司 | Data access method and device as well as distributed database system |
| US9767107B1 (en) * | 2013-06-29 | 2017-09-19 | EMC IP Holding Company LLC | Parallel file system with metadata distributed across partitioned key-value store |
| CN105814560A (en) * | 2013-12-09 | 2016-07-27 | 赛灵思公司 | Memory arrangement for implementation of high-throughput key-value stores |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112181872A (en) | 2021-01-05 |
| CN110377535B (en) | 2020-11-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11270734B2 (en) | Method for accessing shingled magnetic recording SMR disk, and server | |
| US10891074B2 (en) | Key-value storage device supporting snapshot function and operating method thereof | |
| US9710283B2 (en) | System and method for pre-storing small data files into a page-cache and performing reading and writing to the page cache during booting | |
| WO2020211236A1 (en) | Read-write conflict resolution method and apparatus employing b+ tree and storage medium | |
| CN104054071A (en) | Method for accessing storage device and storage device | |
| CN105975878B (en) | Method for secure storing and system based on Nand Flash flash memory | |
| CN110888837B (en) | Object storage small file merging method and device | |
| WO2017213908A1 (en) | Method and apparatus for improving performance of sequential logging in a storage device | |
| Geier | The differences between SSD and HDD technology regarding forensic investigations | |
| CN108614668A (en) | Data access method based on KV models and solid storage device | |
| CN109976664B (en) | Log data organization for solid state storage devices | |
| CN101576854A (en) | File access method, device and system | |
| CN109947669A (en) | The address conversion method and system of KV storage equipment | |
| Marupudi | Solid State Drive: New Challenge for Forensic Investigation | |
| CN111158606B (en) | Storage method, storage device, computer equipment and storage medium | |
| CN109840048A (en) | Store command processing method and its storage equipment | |
| CN109426436A (en) | Rubbish recovering method and device based on variable length bulk | |
| CN112463055A (en) | Method, system, equipment and medium for optimizing and using L2P table of solid state disk | |
| CN110377535A (en) | Distributed KV storage system | |
| CN110324380A (en) | Utilize the cloud computing and mist computing system of KV storage equipment | |
| CN110968520B (en) | Multi-stream storage device based on unified cache architecture | |
| Kambalapalli | Different forensic tools on a single SSD and HDD, their differences and drawbacks | |
| CN113721838A (en) | Writing and reading data method for storage device, storage controller and DMA engine | |
| Fatima et al. | Forensic analysis of image deletion applications | |
| Kondam | Comparing SSD forensics with HDD forensics |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |