CN107291628B - Method and apparatus for accessing data storage device - Google Patents
Method and apparatus for accessing data storage device Download PDFInfo
- Publication number
- CN107291628B CN107291628B CN201710540110.0A CN201710540110A CN107291628B CN 107291628 B CN107291628 B CN 107291628B CN 201710540110 A CN201710540110 A CN 201710540110A CN 107291628 B CN107291628 B CN 107291628B
- Authority
- CN
- China
- Prior art keywords
- matrix
- storage device
- data storage
- keyword
- hash
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
- G06F12/02—Addressing or allocation; Relocation
- G06F12/0223—User address space allocation, e.g. contiguous or non contiguous base addressing
- G06F12/023—Free address space management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
- G06F12/02—Addressing or allocation; Relocation
- G06F12/0207—Addressing or allocation; Relocation with multidimensional access, e.g. row/column, matrix
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
- G06F12/02—Addressing or allocation; Relocation
- G06F12/08—Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
- G06F12/0802—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches
- G06F12/0844—Multiple simultaneous or quasi-simultaneous cache accessing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
- G06F12/02—Addressing or allocation; Relocation
- G06F12/08—Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
- G06F12/0802—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches
- G06F12/0893—Caches characterised by their organisation or structure
- G06F12/0895—Caches characterised by their organisation or structure of parts of caches, e.g. directory or tag array
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/52—Program synchronisation; Mutual exclusion, e.g. by means of semaphores
- G06F9/526—Mutual exclusion algorithms
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2212/00—Indexing scheme relating to accessing, addressing or allocation within memory systems or architectures
- G06F2212/10—Providing a specific technical effect
- G06F2212/1016—Performance improvement
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2212/00—Indexing scheme relating to accessing, addressing or allocation within memory systems or architectures
- G06F2212/10—Providing a specific technical effect
- G06F2212/1016—Performance improvement
- G06F2212/1024—Latency reduction
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present disclosure provides a method of accessing a data storage device and a corresponding apparatus. The data storage device comprises a plurality of storage areas, the storage areas correspond to nodes in an N x M matrix one by one, N is the row number of the matrix, M is the column number of the matrix, both N and M are positive integers greater than or equal to 1, and the rows and columns of the matrix are crossed to form the nodes. The method comprises the following steps: determining a row in the matrix by a key and a first at least one hash function corresponding to the row of the matrix; determining a column in the matrix by the keyword and a second at least one hash function corresponding to the column of the matrix; determining a node in the matrix by the determined row and column; and accessing a storage area corresponding to the determined node.
Description
Technical Field
The present disclosure relates to the field of computer technologies, and in particular, to a method and an apparatus for accessing a data storage device.
Background
Fast and efficient data access is critical to the computer system background response speed. At present, data access to a data storage device is mainly realized through a red-black tree, a hash table or a third-party Key-Value storage tool. The operation time complexity of the hash table is O (1), the time consumption is a constant in theory, and the operation time consumption is obviously reduced compared with that of a red-black tree; meanwhile, compared with a third-party Key-Value storage tool, the hash table does not need to be accessed through a network socket, so that the problems of network time consumption and network fault risk do not exist. However, in the course of implementing the disclosed concept, the inventors found that there are at least the following problems in the prior art: in a big data application, as the data volume increases, the probability of hash value collision when accessing through the hash table increases, and the speed of data access is reduced.
Disclosure of Invention
In view of the above, the present disclosure provides a method and an apparatus for accessing a data storage device, which can effectively reduce the probability of hash value collision when data access is performed through a hash table, and simultaneously improve the data access speed.
One aspect of the present disclosure provides a method of accessing a data storage device, the data storage device including a plurality of storage regions, the plurality of storage regions corresponding one-to-one to nodes in an N x M matrix, where N is a number of rows of the matrix, M is a number of columns of the matrix, N and M are both positive integers greater than or equal to 1, and the rows and columns of the matrix intersect to form the nodes. The method comprises the following steps: determining a row in the matrix by a key and a first at least one hash function corresponding to the row of the matrix, determining a column in the matrix by the key and a second at least one hash function corresponding to the column of the matrix, determining a node in the matrix by the determined row and column, and accessing a storage area corresponding to the determined node.
According to an embodiment of the present disclosure, the first at least one hash function is the same as or different from the second at least one hash function.
According to an embodiment of the present disclosure, the method further comprises creating the N x M matrix; and mapping the addresses of the storage regions to nodes in an N x M matrix so that the storage regions correspond to the nodes in the N x M matrix one by one.
According to an embodiment of the present disclosure, the method further comprises storing data in each of the at least one storage area by a red-black tree structure.
According to an embodiment of the present disclosure, the method further comprises setting a read-write lock to the at least one storage area.
According to an embodiment of the present disclosure, the data storage device includes a memory or a cache.
Another aspect of the present disclosure provides an apparatus for accessing a data storage device, the data storage device including a plurality of storage regions, the plurality of storage regions corresponding to nodes in an N x M matrix, wherein N is a number of rows of the matrix, M is a number of columns of the matrix, N and M are positive integers greater than or equal to 1, and the rows and columns of the matrix intersect to form the nodes. The apparatus includes a row determination module, a column determination module, a node determination module, and an access module. A row determination module to determine a row in the matrix by a key and a first at least one hash function corresponding to the row of the matrix. A column determination module to determine a column in the matrix by the key and a second at least one hash function corresponding to the column of the matrix. A node determination module for determining a node in the matrix from the determined row and column. And the access module is used for accessing the storage area corresponding to the determined node.
According to an embodiment of the present disclosure, the first at least one hash function is the same as or different from the second at least one hash function.
According to an embodiment of the present disclosure, the apparatus further includes a matrix creation module and a mapping module. And the matrix creating module is used for creating the N x M matrix. And the mapping module is used for mapping the addresses of the storage regions to nodes in the N x M matrix so as to enable the storage regions to be in one-to-one correspondence with the nodes in the N x M matrix.
According to an embodiment of the present disclosure, the apparatus further includes a red-black tree storage module for storing data in each of the at least one storage area through a red-black tree structure.
According to an embodiment of the present disclosure, the apparatus further includes a read/write lock setting module, configured to set a read/write lock for at least one storage area.
According to an embodiment of the present disclosure, the data storage device in the apparatus includes a memory or a cache.
Another aspect of the present disclosure provides an apparatus for accessing a data storage device, comprising one or more memories storing executable instructions; and one or more processors executing the executable instructions to implement the method of accessing a data storage device of an embodiment of the present disclosure.
Another aspect of the present disclosure provides a computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, may implement the method of accessing a data storage device of an embodiment of the present disclosure.
According to the embodiment of the disclosure, the probability of hash value collision when data access is performed through the hash table can be at least partially reduced, and therefore, the technical effect of accurately and efficiently accessing data can be achieved.
According to the embodiment of the disclosure, because the read-write lock can be set for the storage area corresponding to at least one node in the data storage device, the probability of contention occurring at the at least one node when different readers and/or writers access the data storage device is greatly reduced.
According to the embodiment of the disclosure, because data can be stored in each storage area of at least one storage area through the red-black tree structure, when the data is quickly located to any storage area of the at least one storage area through the keyword, the time-consuming balance of operating the data in any storage area can be ensured.
Drawings
The above and other objects, features and advantages of the present disclosure will become more apparent from the following description of embodiments of the present disclosure with reference to the accompanying drawings, in which:
FIG. 1 schematically illustrates an exemplary system architecture to which the methods and apparatus for accessing a data storage device may be applied, according to an embodiment of the present disclosure;
FIG. 2 schematically illustrates a flow diagram of a method of accessing a data storage device according to an embodiment of the present disclosure;
FIG. 3 schematically illustrates a flow diagram of a method of accessing a data storage device according to another embodiment of the present disclosure;
FIG. 4 schematically illustrates a flow diagram of a method of accessing a data storage device according to yet another embodiment of the present disclosure;
FIG. 5 schematically illustrates a flow diagram of a method of accessing a data storage device according to yet another embodiment of the present disclosure;
FIG. 6 schematically illustrates an application scenario diagram of a method of accessing a data storage device according to an embodiment of the present disclosure;
FIG. 7 schematically illustrates a block diagram of an apparatus for accessing a data storage device, in accordance with an embodiment of the present disclosure; and
FIG. 8 schematically illustrates a block diagram of a computer system accessing a data storage device, according to an embodiment of the present disclosure.
Detailed Description
Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood that the description is illustrative only and is not intended to limit the scope of the present disclosure. Moreover, in the following description, descriptions of well-known structures and techniques are omitted so as to not unnecessarily obscure the concepts of the present disclosure.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The words "a", "an" and "the" and the like as used herein are also intended to include the meanings of "a plurality" and "the" unless the context clearly dictates otherwise. Furthermore, the terms "comprises," "comprising," and the like, as used herein, specify the presence of stated features, steps, operations, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, or components.
All terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art unless otherwise defined. It is noted that the terms used herein should be interpreted as having a meaning that is consistent with the context of this specification and should not be interpreted in an idealized or overly formal sense.
Embodiments of the present disclosure provide a method of accessing a data storage device and a corresponding apparatus. The data storage device applied by the method and the device of the embodiment of the disclosure comprises a plurality of storage areas, wherein the plurality of storage areas correspond to nodes in an N x M matrix one by one, wherein N is the row number of the matrix, M is the column number of the matrix, both N and M are positive integers greater than or equal to 1, and the rows and columns of the matrix are crossed to form the nodes. The method for accessing the data storage device according to the embodiment of the disclosure comprises the following steps: determining a row in the matrix by a key and a first at least one hash function corresponding to the row of the matrix, determining a column in the matrix by the key and a second at least one hash function corresponding to the column of the matrix, determining a node in the matrix by the determined row and column, and accessing a storage area corresponding to the determined node.
According to the method for accessing the data storage device and the corresponding device, in the process of determining the corresponding storage area in the data storage device through the keyword, at least one hash operation is performed on the keyword on each dimension in the two-dimensional space, the calculation results on the two dimensions are combined to finally determine the position of the corresponding storage area, and the hash operations of each dimension are mutually independent in the process, so that the probability of hash value collision when the keyword mapping is performed on only one dimension through the hash operation in the hash table is greatly reduced, the efficiency of determining the storage area through the keyword is improved, and the speed of data access to the data storage device is further improved.
FIG. 1 schematically illustrates an exemplary system architecture 100 to which the methods and apparatus for accessing a data storage device may be applied, according to an embodiment of the present disclosure.
As shown in fig. 1, the system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 serves as a medium for providing communication links between the terminal devices 101, 102, 103 and the server 105. Network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few.
The user may use the terminal devices 101, 102, 103 to interact with the server 105 via the network 104 to receive or send messages or the like. The terminal devices 101, 102, 103 may have installed thereon various communication client applications, such as shopping-like applications, web browser applications, search-like applications, instant messaging tools, mailbox clients, social platform software, etc. (by way of example only).
The terminal devices 101, 102, 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.
The server 105 may be a server providing various services, such as a background management server (for example only) providing support for websites browsed by users using the terminal devices 101, 102, 103. The background management server may analyze and perform other processing on the received data such as the user request, and feed back a processing result (e.g., a webpage, information, or data obtained or generated according to the user request) to the terminal device.
It should be noted that the method for accessing a data storage device provided by the embodiment of the present disclosure may be generally executed by the server 105. Accordingly, the means for accessing a data storage device provided by the embodiments of the present disclosure may be generally disposed in the server 105. The method of accessing a data storage device provided by the embodiments of the present disclosure may also be performed by a server or a cluster of servers different from the server 105 and capable of communicating with the terminal devices 101, 102, 103 and/or the server 105. Accordingly, the access data storage device apparatus provided by the embodiment of the present disclosure may also be disposed in a server or a server cluster different from the server 105 and capable of communicating with the terminal devices 101, 102, 103 and/or the server 105.
In addition, the method for accessing the data storage device provided by the embodiment of the disclosure can also be implemented in the terminal device 101, 102, and/or 103, so as to improve the access speed of the terminal device 101, 102, and/or 103 to the own data storage device. Accordingly, the apparatus for accessing a data storage device provided by the embodiment of the present disclosure may also be disposed in the terminal device 101, 102, and/or 103.
It should be understood that the number of terminal devices, networks, and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
FIG. 2 schematically illustrates a flow diagram of a method of accessing a data storage device, in accordance with an embodiment of the present disclosure.
As shown in fig. 2, a method of accessing a data storage device according to an embodiment of the present disclosure includes operations S201 to S204, where the data storage device includes a plurality of storage regions, and the storage regions correspond to nodes in an N × M matrix, where N is a number of rows of the matrix, M is a number of columns of the matrix, N and M are positive integers greater than or equal to 1, and the rows and columns of the matrix intersect to form the nodes.
According to an embodiment of the present disclosure, the data storage device includes a memory or a cache.
In operation S201, a row in the matrix is determined by a key and a first at least one hash function corresponding to the row of the matrix.
According to the embodiment of the present disclosure, the first at least one hash function corresponding to the row of the matrix may be one hash function or may be multiple hash functions.
And when the first at least one hash function is one hash function, performing one hash operation on the keyword through the first hash function, and determining one row in the matrix according to an operation result.
And when the first at least one hash function is a plurality of hash functions, carrying out hash operation on the keyword in sequence through the first plurality of hash functions, and determining a row in the matrix according to an operation result.
In addition, the determining of the row in the matrix according to the operation result may be directly determining the row in the matrix according to a hash value obtained by performing a hash operation on the keyword, or may be determining the row in the matrix according to a result of further processing by further processing the hash value.
For example, when the value of the hash value obtained by hashing the key is greater than the number of rows N of the matrix, the hash value may be made modulo N, and the modulo value may be used as the number of rows for mapping the key to the matrix.
In operation S202, a column in the matrix is determined by the key and a second at least one hash function corresponding to the column of the matrix.
According to the embodiment of the present disclosure, the second at least one hash function corresponding to the column of the matrix may be one hash function or may be multiple hash functions.
And when the second at least one hash function is one hash function, performing one hash operation on the keyword through the second hash function, and determining a column in the matrix according to an operation result.
And when the second at least one hash function is a plurality of hash functions, carrying out hash operation on the key words in sequence through the second plurality of hash functions, and determining a column in the matrix according to the operation result.
In addition, the determining of one column in the matrix according to the operation result may be directly determining one column in the matrix according to a hash value obtained by performing a hash operation on the keyword, or may be determining one column in the matrix according to a result of further processing by further processing the hash value.
For example, when the hash value obtained by hashing the keyword is greater than the column number M of the matrix, the hash value may be modulo the M, and the modulo value may be used as the column number for mapping the keyword to the matrix.
In operation S203, one node in the matrix is determined by the determined row and column.
In operation S204, a storage area corresponding to the determined node is accessed. For example, data is written in or read from a storage area corresponding to the determined node, or data is deleted from or inserted into the storage area, or the like.
It is to be understood that operation S201 is performed before operation S202 in fig. 2 is merely an example.
According to the embodiment of the present disclosure, the operations S201 and S202 are independent from each other, and may be performed simultaneously or sequentially. If the processes are performed sequentially, the sequential order of the processes is not limited at all.
According to the method for accessing the data storage device, the rows and the columns in the matrix are determined by the first at least one hash function corresponding to the key word and the rows and the second at least one hash function corresponding to the columns respectively, and then one node in the matrix is determined by the determined rows and columns, so that the storage area corresponding to the determined node can be accessed.
By the method, in the process of mapping the keyword to the corresponding storage area of the data storage device, at least one hash operation is performed on the keyword in each dimension in the two-dimensional space, the position of the corresponding storage area is finally determined by combining the calculation results in the two dimensions, and the hash operations in each dimension are independent from each other in the process, so that the probability of hash value collision when the keyword mapping is performed only in one dimension through the hash operation in the hash table is greatly reduced.
Furthermore, the method for accessing the data storage device according to the embodiment of the disclosure can efficiently and accurately map the keyword to a specific storage area, and reduce the processing time and consumption of invalid data caused by hash value collision, thereby improving the access speed of the data storage device.
According to an embodiment of the present disclosure, the first at least one hash function and the second at least one hash function may be the same or different.
When the first at least one hash function and the second at least one hash function are different, the probability of hash value collisions resulting from mapping a key into a particular storage area of the data storage device will be significantly reduced.
According to an embodiment of the present disclosure, the first at least one hash function may include any one or more of: an MD5 function, a CRC16 function, a square-of-sum function, and/or a fold-add function, among others.
According to an embodiment of the present disclosure, the second at least one hash function may include any one or more of: an MD5 function, a CRC16 function, a square-of-sum function, and/or a fold-add function, among others.
FIG. 3 schematically shows a flow diagram of a method of accessing a data storage device according to another embodiment of the present disclosure.
As shown in fig. 3, a method for accessing a data storage device according to another embodiment of the present disclosure further includes operations S301 and S302 before operations S201 to S204.
In operation S301, the N × M matrix is created.
In operation S302, addresses of the plurality of storage regions are mapped to nodes in the N × M matrix, so that the plurality of storage regions correspond to the nodes in the N × M matrix one to one.
Then, operations S201 to S204 are performed, and a hash operation is performed on the keyword in the N × M matrix through a first at least one hash function corresponding to the row and a second at least one hash function corresponding to the column, so as to map the keyword to the corresponding row and column, respectively, thereby determining a node corresponding to the keyword, and accessing the storage area corresponding to the node.
According to the embodiment of the disclosure, the N x M matrix is created in the data storage device, and the plurality of storage areas of the data storage device are in one-to-one correspondence with the nodes in the N x M matrix, so that on one hand, a user can conveniently select specific data of N and M according to the needs of the user, and the matrix with the corresponding size can be created in a personalized manner.
FIG. 4 schematically illustrates a flow diagram of a method of accessing a data storage device according to yet another embodiment of the present disclosure.
As shown in fig. 4, a method for accessing a data storage device according to still another embodiment of the present disclosure includes operation S405 in addition to operations S201 to S204.
In operation S405, data is stored in each of the at least one storage area through a red-black tree structure.
At least one memory area, for example, may have only one memory area. At this time, data is stored in the one storage area by the red-black tree structure.
Alternatively, the at least one memory area may be, for example, two or more memory areas. At this time, data is stored in each of the two or more storage areas by a red-black tree structure.
The red and black tree data structure is a self-balancing binary search tree data structure, can efficiently keep the balance of a binary search tree when data insertion and deletion operations are carried out, and has higher data search performance.
According to the embodiment of the disclosure, the data is stored in each storage area of the at least one storage area in the red-black tree structure, so that when the keyword is mapped to any one storage area of the at least one storage area, the time consumption balance of operating the data in any one storage area can be ensured, and the stability of data access is improved.
FIG. 5 schematically illustrates a flow diagram of a method of accessing a data storage device according to yet another embodiment of the present disclosure.
As shown in fig. 5, a method for accessing a data storage device according to still another embodiment of the present disclosure further includes operation S501 before operations S201 to S204.
In operation S501, a read-write lock is set for at least one storage region.
Specifically, the at least one storage area may include only one storage area, for example. At this time, the read/write lock is set to the one storage area.
Alternatively, the at least one memory area may include, for example, two or more memory areas. In this case, a read/write lock may be provided for each of the two or more storage areas, or a read/write lock may be provided for a plurality of the two or more storage areas in common.
The read-write lock is a special spin lock, which divides the visitor to the shared resource into a reader and a writer, the reader only performs read access to the shared resource, and the writer needs to perform write operation to the shared resource. Read-write phase locks improve concurrency over spin locks because in a multiprocessor system, they allow multiple readers to access shared resources simultaneously, the maximum possible number of readers being the actual number of logical CPUs. Writers are exclusive, and a read-write lock can only have one writer or multiple readers (related to the number of CPUs) at the same time, but cannot have both readers and writers at the same time.
According to an embodiment of the disclosure, setting a read-write lock on at least one storage region may be implemented by constructing and saving a lock object, commonly referred to as a handle in windows systems and a descriptor in linux systems, when initializing the at least one storage region.
After the read-write lock is initialized and set for at least one storage area, when a writer writes data into the at least one storage area, the lock object in the at least one storage area is set as the write lock at the same time, so that other writers or readers cannot access the at least one storage area currently.
Accordingly, when a reader needs to read data from the at least one storage area, the lock object in the at least one storage area is set as a read lock at the same time. The read lock does not prevent other readers from reading the data in the at least one storage area, but prevents other writers from accessing the at least one storage area.
When no reader or writer accesses the at least one storage area, the read lock or write lock set for the lock object is automatically released.
It is to be understood that operation S501 in fig. 5 is just illustrative before operation S201.
According to an embodiment of the present disclosure, operation S501 may be located at any time before operation S204.
According to the embodiment of the disclosure, the read-write lock is set for the storage area corresponding to the at least one storage area in the data storage device, so that the probability of contention occurring at the at least one storage area when different readers and/or writers access the storage device is greatly reduced.
Compared with the method for setting a read-write lock for the whole storage device, the method for setting the read-write lock for the data storage device has the advantages that the data storage device can be accessed by a plurality of readers or writers in different storage areas of the data storage device at the same time by setting the read-write lock for at least one storage area, and the efficiency of accessing the data storage device is improved.
The specific implementation process of the method for accessing the data storage device according to the embodiment of the present disclosure can refer to the schematic description of fig. 6.
Fig. 6 schematically illustrates an application scenario of a method of accessing a data storage device according to an embodiment of the present disclosure.
As shown in fig. 6, the data storage device 610 may be a memory, or a cache.
The data storage device 610 includes a plurality of storage regions, which correspond to nodes in a 6 x 4 matrix, where the matrix has a number of rows N of 6 and a number of columns M of 4, and the rows and columns of the matrix intersect to form each node.
Specifically, the 6 × 4 matrix may be created according to operations S401 and S402 in the initialization of the data storage device 610, and addresses of a plurality of storage regions in the data storage device 610 are mapped to nodes in the 6 × 4 matrix.
Then, in operation S201, a row in the matrix is determined by a first at least one hash function of the key 601 corresponding to the row of the matrix.
For example, the key 601 is operated by the first at least one hash function to obtain a hash value, and then the hash value is modulo 6, and the result of the modulo operation is taken as the number of rows of the determined row.
For example, when the result of the modulo operation is 6, it means that a hash value obtained by operating the key 601 through the first at least one hash function is an integer multiple of the number of rows 6 of the matrix, and the number of rows of the matrix determined by the key is the 6 th row.
In operation S202, a column in the matrix is determined by a second at least one hash function of the key 601 corresponding to the column of the matrix.
For example, the key 601 is operated by the second at least one hash function to obtain a hash value, then the hash value is modulo 4, and the result of the modulo operation is taken as the column number of the determined column.
For example, when the result of the modulo operation is 3, the number of columns of the matrix determined by the key is column 3.
In operation S203, one node in the matrix is determined by the determined row and column.
For example, when the determined row is row 6 and column 3, the determined node in the matrix is (6, 3) in fig. 3.
In operation S204, a storage area corresponding to the determined node is accessed.
For example, after the node (6, 3) is determined in fig. 3, the storage area corresponding to the node (6, 3) can be accessed.
Further, data may be stored in each of the at least one storage area in the data storage device 610 by a red-black tree structure.
For example, data may be stored through a red-black tree for each of all storage areas in the data storage device 610.
Specifically, when each keyword (including the keyword 601) is mapped to a corresponding storage area, data may be stored in the storage area in a red-black tree structure. For example, after mapping the key 601 to the storage area corresponding to the node (6.3), the data is stored in the storage area corresponding to the node (6.3) in a red-black tree structure.
In addition, a read-write lock may also be set to at least one storage area in the data storage device 610.
For example, when initializing the data storage device 610, a lock object is set for each storage area, and then the lock object in each storage area is correspondingly set as a read lock or a write lock according to whether a reader or a writer accesses each storage area.
In this way, when a plurality of logical CPUs access the data storage device 610 simultaneously during multi-thread operation, when one writer writes data to the storage area corresponding to the node (6.3) through the key 601, the lock object of the storage area is set as write lock, so as to prevent other writers or readers from accessing the storage area.
Or, when a reader reads data from the storage area corresponding to the node (6.3) through the key 601, the lock object of the storage area is set as a read lock, and at this time, other readers are still allowed to read data from the storage area, but other writers are not allowed to write data into the storage area.
In this process, other storage areas of the data storage device 610 can be accessed by corresponding readers or writers without affecting access to the storage area corresponding to the node (6.3).
In this manner, multiple logical CPUs may be allowed to access data storage device 610 simultaneously, and the probability of lock contention occurring in the storage area corresponding to each node is greatly reduced.
And when the writer or all readers accessing the storage area corresponding to the node (6.3) quit, automatically releasing the read lock or the write lock set for the lock object in the storage area.
When the data storage device 610 is a memory or a cache, the data stored in the data storage device 610 in a 6 × 4 matrix structure may be written into the nonvolatile storage medium 620 such as a hard disk correspondingly at regular intervals. In this way, when the operating system is restarted, the data written in the nonvolatile storage medium 620 is directly loaded, thereby reducing data loss in the abnormal situation as much as possible.
FIG. 7 schematically illustrates a block diagram of an apparatus for accessing a data storage device, in accordance with an embodiment of the present disclosure.
As shown in fig. 7, an apparatus 700 for accessing a data storage device according to an embodiment of the present disclosure includes a row determination module 710, a column determination module 720, a node determination module 730, and an access module 740.
According to an embodiment of the present disclosure, the data storage device accessed by apparatus 700 includes a plurality of storage regions, which correspond to nodes in an N × M matrix one to one, where N is the number of rows of the matrix, M is the number of columns of the matrix, N and M are both positive integers greater than or equal to 1, and the rows and columns of the matrix intersect to form the nodes.
According to an embodiment of the present disclosure, the data storage device includes a memory or a cache.
The row determination module 710 is configured to determine a row in the matrix by a key and a first at least one hash function corresponding to the row of the matrix.
The column determination module 720 is configured to determine a column in the matrix by the key and a second at least one hash function corresponding to the column of the matrix.
The node determining module 730 is configured to determine a node in the matrix by the determined row and column.
The accessing module 740 is configured to access a storage area corresponding to the determined node.
According to the embodiment of the present disclosure, the apparatus 700 for accessing a data storage device determines rows and columns in the matrix through a first at least one hash function corresponding to a key and a second at least one hash function corresponding to a column, and then determines a node in the matrix according to the determined rows and columns, so as to access a storage area corresponding to the determined node.
The apparatus 700 for accessing a data storage device performs at least one hash operation on a keyword in each dimension in a two-dimensional space during the process of mapping the keyword to a corresponding storage region of the data storage device, and combines the calculation results in the two dimensions to finally determine the position of the corresponding storage region, and the hash operations in each dimension are independent from each other during the process, thereby greatly reducing the probability of hash value collision when the keyword is mapped only in one dimension through the hash operation in the hash table.
Furthermore, the apparatus 700 for accessing a data storage device can efficiently and accurately map the keyword to a specific storage area, and reduce the processing time and consumption of invalid data caused by hash value collision, thereby increasing the access speed to the data storage device.
According to an embodiment of the present disclosure, the first at least one hash function and the second at least one hash function may be the same or different.
When the first at least one hash function and the second at least one hash function are different, the probability of hash value collisions resulting from mapping a key into a particular storage area of the data storage device will be significantly reduced.
According to an embodiment of the present disclosure, the first at least one hash function may be an MD5 function, a CRC16 function, a square-of-squares-sum function, and/or a fold-add function.
According to an embodiment of the present disclosure, the second at least one hash function may be an MD5 function, a CRC16 function, a square-of-squares-median function, and/or a fold-and-add function.
The apparatus for accessing a data storage device 700 may further include a matrix creation module 750 and a mapping module 760, according to embodiments of the present disclosure.
In particular, the matrix creation module 750 is configured to create the N x M matrix.
The mapping module 760 is configured to map the addresses of the storage regions to nodes in the N × M matrix, so that the storage regions correspond to the nodes in the N × M matrix one by one.
According to the embodiment of the present disclosure, the apparatus 700 for accessing a data storage device creates an N × M matrix in the data storage device through the matrix creation module 750, and corresponds a plurality of storage areas of the data storage device to nodes in the N × M matrix one by one through the mapping module 760, which facilitates a user to select specific data of N and M according to the need of the user, so as to create a matrix with a corresponding size in a personalized manner.
The apparatus for accessing a data storage device 700 may further include a red-black tree storage module 770 according to an embodiment of the present disclosure.
The red-black tree storage module 770 is configured to store data in each storage area of the at least one storage area through a red-black tree structure.
According to the embodiment of the present disclosure, the apparatus 700 for accessing a data storage device stores data in a red-black tree structure in each storage area of at least one storage area through the red-black tree storage module 770, so that when a keyword is mapped to any one storage area of the at least one storage area, time consuming balance of operations on the data in the any one storage area can be ensured, and stability of data access is improved.
The apparatus for accessing a data storage device 700 may further include a read-write lock setup module 780 according to an embodiment of the present disclosure.
The read/write lock setting module 780 is configured to set a read/write lock on at least one storage region.
According to an embodiment of the present disclosure, the apparatus 700 for accessing a data storage device sets a read/write lock on a storage area corresponding to at least one storage area in the data storage device through the read/write lock setting module 780. In this way, the probability of contention occurring at the at least one storage region when different readers and/or writers access the storage device is greatly reduced, thereby allowing multiple readers or writers to simultaneously access different storage regions of the data storage device, and improving the efficiency of accessing the data storage device.
FIG. 8 schematically illustrates a block diagram of a computer system accessing a data storage device, according to an embodiment of the present disclosure.
FIG. 8 schematically illustrates a block diagram of a computer system suitable for implementing access to a data storage device according to an embodiment of the present disclosure. The computer system illustrated in FIG. 8 is only one example and should not impose any limitations on the scope of use or functionality of embodiments of the disclosure.
As shown in fig. 8, a computer system 800 according to an embodiment of the present disclosure includes a processor 801 that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM)802 or a program loaded from a storage section 808 into a Random Access Memory (RAM) 803. The processor 801 may include, for example, a general purpose microprocessor (e.g., a CPU), an instruction set processor and/or associated chipset, and/or a special purpose microprocessor (e.g., an Application Specific Integrated Circuit (ASIC)), among others. The processor 810 may also include on-board memory for caching purposes. Processor 810 may include a single processing unit or multiple processing units for performing different actions of the method flows described with reference to fig. 2-5 in accordance with embodiments of the disclosure.
In the RAM 803, various programs and data necessary for the operation of the system 800 are stored. The processor 801, the ROM 802, and the RAM 803 are connected to each other by a bus 804. The processor 801 performs various operations of the method of accessing a data storage device described above with reference to fig. 2 to 5 by executing programs in the ROM 802 and/or the RAM 803. Note that the programs may also be stored in one or more memories other than the ROM 802 and RAM 803. The processor 801 may also perform various operations of the methods of accessing a data storage device described above with reference to fig. 2-5 by executing programs stored in the one or more memories.
According to an embodiment of the present disclosure, the method described above with reference to the flow chart may be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication section 809 and/or installed from the removable medium 811. The computer program, when executed by the processor 801, performs the above-described functions defined in the system of the embodiments of the present disclosure. The systems, devices, apparatuses, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the present disclosure.
It should be noted that the computer readable media shown in the present disclosure may be computer readable signal media or computer readable storage media or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In contrast, in the present disclosure, a computer-readable signal medium may include a propagated data signal with computer-readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing. According to embodiments of the present disclosure, a computer-readable medium may include one or more memories other than the ROM 802 and/or the RAM 803 and/or the ROM 802 and the RAM 803 described above.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
As another aspect, the present disclosure also provides a computer-readable medium, which may be contained in the apparatus described in the above embodiments; or may be separate and not incorporated into the device. The computer readable medium carries one or more programs which, when executed by a device, cause the device to perform a method for accessing a data storage device according to an embodiment of the present disclosure, where the data storage device includes a plurality of storage regions, the plurality of storage regions corresponding to nodes in an N x M matrix, where N is a number of rows of the matrix, M is a number of columns of the matrix, and both N and M are positive integers greater than or equal to 1, and rows and columns of the matrix intersect to form the nodes. The method comprises the following steps: determining a row in the matrix by a key and a first at least one hash function corresponding to the row of the matrix, determining a column in the matrix by the key and a second at least one hash function corresponding to the column of the matrix, determining a node in the matrix by the determined row and column, and accessing a storage area corresponding to the determined node.
According to an embodiment of the present disclosure, the first at least one hash function is the same as or different from the second at least one hash function.
According to an embodiment of the present disclosure, the method further comprises creating the N x M matrix; and mapping the addresses of the storage regions to nodes in the N-M matrix so that the storage regions correspond to the nodes in the N-M matrix one by one.
According to an embodiment of the present disclosure, the method further comprises storing data in each of the at least one storage area by a red-black tree structure.
According to an embodiment of the present disclosure, the method further comprises setting a read-write lock to the at least one storage area.
According to an embodiment of the present disclosure, the data storage device includes a memory or a cache.
The embodiments of the present disclosure have been described above. However, these examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although the embodiments are described separately above, this does not mean that the measures in the embodiments cannot be used in advantageous combination. The scope of the disclosure is defined by the appended claims and equivalents thereof. Various alternatives and modifications can be devised by those skilled in the art without departing from the scope of the present disclosure, and such alternatives and modifications are intended to be within the scope of the present disclosure.
Claims (14)
1. A method of accessing a data storage device, the data storage device comprising a plurality of storage regions, the plurality of storage regions corresponding one-to-one to nodes in an N x M matrix, wherein N is a number of rows of the matrix, M is a number of columns of the matrix, N and M are both positive integers greater than or equal to 1, the rows and columns of the matrix intersecting to form the nodes, the method comprising:
determining a row in the matrix through a keyword and a first at least one hash function corresponding to the row of the matrix, wherein the step of performing hash operation on the keyword sequentially through the first at least one hash function comprises determining the row number for mapping the keyword to the matrix according to a hash value obtained by operation;
determining a column in the matrix through the keyword and a second at least one hash function corresponding to the column of the matrix, wherein the step of performing hash operation on the keyword sequentially through the second at least one hash function comprises determining the number of columns for mapping the keyword to the matrix according to a hash value obtained through operation;
determining a node in the matrix by the determined number of rows and columns mapping the key to the matrix; and
access a storage area corresponding to the determined node.
2. The method of claim 1, wherein:
the first at least one hash function is the same as or different from the second at least one hash function.
3. The method of claim 1, further comprising:
creating the N x M matrix; and
and mapping the addresses of the storage regions to nodes in the N-M matrix so that the storage regions correspond to the nodes in the N-M matrix one by one.
4. The method of claim 1, further comprising:
in each of the at least one storage area, data is stored by a red-black tree structure.
5. The method of claim 1, further comprising:
and setting a read-write lock for at least one storage area.
6. The method of claim 1, wherein the data storage device comprises a memory or a cache.
7. An apparatus for accessing a data storage device, the data storage device including a plurality of storage regions, the plurality of storage regions corresponding one-to-one to nodes in an N x M matrix, where N is a number of rows of the matrix, M is a number of columns of the matrix, N and M are positive integers greater than or equal to 1, and the rows and columns of the matrix intersect to form the nodes, the apparatus comprising:
a row determining module, configured to determine a row in the matrix through a keyword and at least one hash function corresponding to the row of the matrix, where the determining includes performing hash operations on the keyword sequentially through the at least one hash function, and determining, according to a hash value obtained through the hash operations, a row number in which the keyword is mapped to the matrix;
a column determination module, configured to determine a column in the matrix through the keyword and a second at least one hash function corresponding to the column of the matrix, where the determining includes performing hash operations on the keyword sequentially through the second at least one hash function, and determining, according to a hash value obtained through the hash operations, a number of columns in the matrix to which the keyword is mapped;
a node determining module for determining a node in the matrix by the determined number of rows and columns mapping the keyword into the matrix; and
and the access module is used for accessing the storage area corresponding to the determined node.
8. The apparatus of claim 7, wherein:
the first at least one hash function is the same as or different from the second at least one hash function.
9. The apparatus of claim 7, further comprising:
a matrix creation module for creating the N x M matrix; and
and the mapping module is used for mapping the addresses of the storage regions to nodes in the N x M matrix so as to enable the storage regions to be in one-to-one correspondence with the nodes in the N x M matrix.
10. The apparatus of claim 7, further comprising
And the red and black tree storage module is used for storing data in each storage area of the at least one storage area through a red and black tree structure.
11. The apparatus of claim 7, further comprising:
and the read-write lock setting module is used for setting a read-write lock for at least one storage area.
12. The apparatus of claim 7, wherein the data storage device comprises a memory or a cache.
13. An apparatus for accessing a data storage device, comprising
One or more memories storing executable instructions; and
one or more processors executing the executable instructions to implement the method of any one of claims 1-6.
14. A computer readable storage medium having stored thereon executable instructions which, when executed by a processor, cause the processor to perform the method of any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710540110.0A CN107291628B (en) | 2017-07-04 | 2017-07-04 | Method and apparatus for accessing data storage device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710540110.0A CN107291628B (en) | 2017-07-04 | 2017-07-04 | Method and apparatus for accessing data storage device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107291628A CN107291628A (en) | 2017-10-24 |
| CN107291628B true CN107291628B (en) | 2020-09-01 |
Family
ID=60100184
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710540110.0A Active CN107291628B (en) | 2017-07-04 | 2017-07-04 | Method and apparatus for accessing data storage device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107291628B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109522299B (en) * | 2018-10-23 | 2020-12-18 | 咪咕数字传媒有限公司 | Data processing method, device, system and storage medium |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101206912A (en) * | 2006-12-22 | 2008-06-25 | 富士通株式会社 | Memory device, memory controller and memory system |
| CN101478608A (en) * | 2009-01-09 | 2009-07-08 | 南京联创科技股份有限公司 | Fast operating method for mass data based on two-dimensional hash |
| CN104158744A (en) * | 2014-07-09 | 2014-11-19 | 中国电子科技集团公司第三十二研究所 | Method for building table and searching for network processor |
| CN104572983A (en) * | 2014-12-31 | 2015-04-29 | 北京锐安科技有限公司 | Construction method based on hash table of memory, text searching method and corresponding device |
| CN104809179A (en) * | 2015-04-16 | 2015-07-29 | 华为技术有限公司 | Device and method for accessing Hash table |
| CN105335411A (en) * | 2014-07-31 | 2016-02-17 | 国际商业机器公司 | Method and system for data processing |
| CN105389360A (en) * | 2015-11-05 | 2016-03-09 | 浪潮(北京)电子信息产业有限公司 | AVL tree-based data writing method and apparatus |
| US9311359B2 (en) * | 2013-01-30 | 2016-04-12 | International Business Machines Corporation | Join operation partitioning |
-
2017
- 2017-07-04 CN CN201710540110.0A patent/CN107291628B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101206912A (en) * | 2006-12-22 | 2008-06-25 | 富士通株式会社 | Memory device, memory controller and memory system |
| CN101478608A (en) * | 2009-01-09 | 2009-07-08 | 南京联创科技股份有限公司 | Fast operating method for mass data based on two-dimensional hash |
| US9311359B2 (en) * | 2013-01-30 | 2016-04-12 | International Business Machines Corporation | Join operation partitioning |
| CN104158744A (en) * | 2014-07-09 | 2014-11-19 | 中国电子科技集团公司第三十二研究所 | Method for building table and searching for network processor |
| CN105335411A (en) * | 2014-07-31 | 2016-02-17 | 国际商业机器公司 | Method and system for data processing |
| CN104572983A (en) * | 2014-12-31 | 2015-04-29 | 北京锐安科技有限公司 | Construction method based on hash table of memory, text searching method and corresponding device |
| CN104809179A (en) * | 2015-04-16 | 2015-07-29 | 华为技术有限公司 | Device and method for accessing Hash table |
| CN105389360A (en) * | 2015-11-05 | 2016-03-09 | 浪潮(北京)电子信息产业有限公司 | AVL tree-based data writing method and apparatus |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107291628A (en) | 2017-10-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111078147B (en) | Processing method, device and equipment for cache data and storage medium | |
| CN107870728B (en) | Method and apparatus for moving data | |
| US10523585B2 (en) | System on a chip comprising multiple compute sub-systems | |
| CN109388626B (en) | Method and apparatus for assigning numbers to services | |
| CN113419824A (en) | Data processing method, device, system and computer storage medium | |
| US10936499B2 (en) | Method, device and computer programme product for storage management | |
| US10788997B2 (en) | Method and device for storage management with metadata stored in disk extents in an extent pool | |
| CN110781159B (en) | Ceph directory file information reading method and device, server and storage medium | |
| CN114667506A (en) | Management of multi-physical function non-volatile memory devices | |
| CN109478171B (en) | Improving throughput in openfabics environment | |
| CN112287010B (en) | Map service providing method, device, terminal and storage medium based on android system | |
| US11055813B2 (en) | Method, electronic device and computer program product for expanding memory of GPU | |
| CN107291628B (en) | Method and apparatus for accessing data storage device | |
| US11438403B2 (en) | Page presentation method and system, computer system, and computer readable medium | |
| US11989281B2 (en) | White list generation device, control method, and program | |
| US10922001B2 (en) | Vector-based storage management | |
| US9805177B1 (en) | Processing large data sets from heterogeneous data sources using federated computing resources | |
| CN111949648A (en) | Memory cache data system and data indexing method | |
| US20150106884A1 (en) | Memcached multi-tenancy offload | |
| CN115562871A (en) | Memory allocation management method and device | |
| CN113076175B (en) | Memory sharing method and device for virtual machine | |
| US20140181502A1 (en) | Dynamically manipulating rules for adding new devices | |
| CN113986833A (en) | File merging method, system, computer system and storage medium | |
| CN113377376A (en) | Data packet generation method, data packet generation device, electronic device, and storage medium | |
| CN108959405B (en) | Strong consistency reading method of data and terminal equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |