CN111104067A - Column-oriented caching method, device, equipment and computer-readable storage medium - Google Patents
Column-oriented caching method, device, equipment and computer-readable storage medium Download PDFInfo
- Publication number
- CN111104067A CN111104067A CN201911326754.5A CN201911326754A CN111104067A CN 111104067 A CN111104067 A CN 111104067A CN 201911326754 A CN201911326754 A CN 201911326754A CN 111104067 A CN111104067 A CN 111104067A
- Authority
- CN
- China
- Prior art keywords
- data
- column
- memory
- cached
- metadata
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 111
- 238000013507 mapping Methods 0.000 claims abstract description 19
- 230000008569 process Effects 0.000 claims description 42
- 238000012545 processing Methods 0.000 claims description 12
- 238000004364 calculation method Methods 0.000 abstract description 8
- 238000004891 communication Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 6
- 239000002699 waste material Substances 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 230000009467 reduction Effects 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000004807 localization Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
- G06F3/065—Replication mechanisms
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0662—Virtualisation aspects
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Memory System Of A Hierarchy Structure (AREA)
Abstract
The invention discloses a column-oriented caching method, a column-oriented caching device, column-oriented caching equipment and a computer-readable storage medium, wherein the method comprises the following steps: acquiring column type format information of data to be cached; allocating memory blocks for the data to be cached in memory resources according to the column-type format information; and adding the memory block into a continuous virtual memory, and generating metadata of the data to be cached, wherein the metadata includes a corresponding relationship between the column format information and the virtual memory, and a mapping relationship between the virtual memory and the memory block. The invention realizes the cross-process calculation without the operations of serialization and deserialization, saves the calculation resources and improves the calculation efficiency.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a column-oriented caching method, apparatus, device, and computer-readable storage medium.
Background
With the development of technologies such as big data and artificial intelligence, the requirement on the data processing efficiency of the computer is higher and higher. The existing cache system needs serialization and deserialization operation in cross-process calculation, wherein serialization is to call a process of converting an object into a byte sequence as object serialization, and deserialization is to call a process of recovering the byte sequence into the object as object deserialization. Both of these processes require the expenditure of computing resources of the computer, resulting in a decrease in efficiency of the computer in cross-process computing.
Disclosure of Invention
The invention mainly aims to provide a column-oriented caching method, a column-oriented caching device, column-oriented caching equipment and a computer-readable storage medium, and aims to solve the technical problem that cross-process computing of an existing caching system needs serialization and deserialization operations and wastes computing resources.
In order to achieve the above object, the present invention provides a column-oriented caching method, including:
acquiring column type format information of data to be cached;
allocating memory blocks for the data to be cached in memory resources according to the column-type format information;
and adding the memory block into a continuous virtual memory, and generating metadata of the data to be cached, wherein the metadata includes a corresponding relationship between the column format information and the virtual memory, and a mapping relationship between the virtual memory and the memory block.
Optionally, after the step of adding the memory block to a continuous virtual memory and generating the metadata of the data to be cached, the method further includes:
after the data to be cached is subjected to columnar caching to obtain columnar cache data, a target virtual row or a target column in the columnar cache data is appointed for a parallel process or a parallel thread;
and calling the parallel process or the parallel thread to determine a virtual memory block corresponding to the target virtual row or the target column according to the metadata, and performing parallel processing on data in the virtual memory block.
Optionally, after the step of adding the memory block to a continuous virtual memory and generating the metadata of the data to be cached, the method further includes:
after the data to be cached is subjected to line-type caching to obtain line-type caching data, determining a target memory block corresponding to the data to be written in the line-type caching data according to the metadata;
copying the content in the target memory block to a copied memory block;
and updating the metadata, wherein the data to be written in the updated metadata is associated with the copied memory block correspondingly.
Optionally, after the step of adding the memory block to a continuous virtual memory and generating the metadata of the data to be cached, the method further includes:
after the data to be cached is subjected to line-type caching to obtain line-type caching data, determining a target memory block corresponding to the data to be modified in the line-type caching data according to the metadata;
and transmitting the data to be modified and the modified value into a concurrent written assembly instruction, and executing the concurrent written assembly instruction to update the data to be modified in the target memory block into the modified value.
Optionally, after the step of adding the memory block to a continuous virtual memory and generating the metadata of the data to be cached, the method further includes:
after the data to be cached is subjected to line-type caching to obtain line-type caching data, determining a target virtual block from the line-type caching data;
detecting whether there is an already locked virtual block in the metadata that overlaps the target virtual block;
if not, adding the virtual row information or the column information corresponding to the target virtual block into the metadata to lock the target virtual block;
and unlocking the target virtual block after the writing of the target virtual block is finished.
Optionally, after the step of adding the memory block to a continuous virtual memory and generating the metadata of the data to be cached, the method further includes:
after the data to be cached is subjected to line-type caching to obtain line-type caching data, determining a target memory block corresponding to the data to be processed in the line-type caching data according to the metadata;
and calling an instruction in the single instruction multiple data stream instruction set to process the target memory block.
Optionally, the step of allocating a memory block for the data to be cached in a memory resource according to the column information includes:
when the column in the column format information is a column with fixed element size, calculating the actual size of the column according to the number of the elements in the column and the data type, and allocating the memory block with the actual size to the column in the memory resource;
when a column in the column format information is a column with a non-fixed element size, allocating a memory block with a default size to the column in memory resources, and when the default size is smaller than the actual size of the column, allocating a memory block with the default size to the column again.
In order to achieve the above object, the present invention also provides a column-oriented cache device, including:
the acquisition module is used for acquiring the column format information of the data to be cached;
the allocation module is used for allocating memory blocks for the data to be cached in the memory resources according to the column format information;
a generating module, configured to add the memory block to a continuous virtual memory, and generate metadata of the data to be cached, where the metadata includes a correspondence between the column format information and the virtual memory, and a mapping relationship between the virtual memory and the memory block.
In addition, to achieve the above object, the present invention further provides a column-oriented cache device, where the column-oriented cache device includes a memory, a processor, and a column-oriented cache program stored in the memory and operable on the processor, and the column-oriented cache program implements the steps of the column-oriented cache method described above when executed by the processor.
In addition, to achieve the above object, the present invention further provides a computer-readable storage medium having a column-oriented caching program stored thereon, where the column-oriented caching program is executed by a processor to implement the steps of the column-oriented caching method as described above.
According to the method and the device, memory blocks are allocated to the data to be cached according to the column-type format information, the memory blocks are added into the continuous virtual memory, metadata of the data to be cached is generated, and the column-type cache data are obtained. Because the corresponding relation between the column format information of the column-type cache data and the virtual memory and the mapping relation between the virtual memory and the memory block are recorded through the metadata, the data are cached in a section of completely continuous virtual memory for an upper column-type cache interface, and the metadata can be mapped to the actually stored memory block when specific contents are accessed, so that a column-type cache method capable of sensing the memory is realized, namely, the memory address of the data can be clearly known through the metadata. When the cross-process calculation is carried out, according to the traditional caching method, data are required to be serialized and then transmitted among the processes, and then the data are subjected to deserialization and reduction so as to ensure the integrity of the data. Because serialization and deserialization operations are not needed, computing resources are saved during cross-process computing, and computing efficiency is improved.
Drawings
FIG. 1 is a schematic diagram of a hardware operating environment according to an embodiment of the present invention;
FIG. 2 is a flow chart illustrating a column-oriented caching method according to a first embodiment of the present invention;
FIG. 3 is a diagram illustrating a metadata record mapping relationship according to an embodiment of the present invention;
FIG. 4 is a block diagram illustrating a preferred embodiment of a column-oriented cache device according to the present invention.
The implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.
Detailed Description
It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
An embodiment of the present invention provides a column-oriented cache device, and referring to fig. 1, fig. 1 is a schematic structural diagram of a hardware operating environment according to an embodiment of the present invention.
Fig. 1 is a schematic structural diagram of a hardware operating environment of a column-oriented cache device. The column-oriented cache device in the embodiment of the invention can be a PC, and can also be a terminal device with a display function, such as a smart phone, a smart television, a tablet computer, a portable computer, and the like.
As shown in fig. 1, the column-oriented cache device may include: a processor 1001, such as a CPU, a network interface 1004, a user interface 1003, a memory 1005, a communication bus 1002. Wherein a communication bus 1002 is used to enable connective communication between these components. The user interface 1003 may include a Display screen (Display), an input unit such as a Keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface, a wireless interface. The network interface 1004 may optionally include a standard wired interface, a wireless interface (e.g., WI-FI interface). The memory 1005 may be a high-speed RAM memory or a non-volatile memory (e.g., a magnetic disk memory). The memory 1005 may alternatively be a storage device separate from the processor 1001.
Optionally, the column-oriented cache device may further include a camera, RF (Radio Frequency) circuitry, sensors, audio circuitry, WiFi modules, and the like. Those skilled in the art will appreciate that the configuration of the array-facing cache devices shown in fig. 1 does not constitute a limitation of the array-facing cache devices, and may include more or fewer components than shown, or some components may be combined, or a different arrangement of components.
As shown in fig. 1, a memory 1005, which is a kind of computer storage medium, may include therein an operating system, a network communication module, a user interface module, and a column-oriented cache program.
In the column-oriented cache device shown in fig. 1, the network interface 1004 is mainly used for connecting to a background server and performing data communication with the background server; the user interface 1003 is mainly used for connecting a client (user side) and performing data communication with the client; and the processor 1001 may be configured to call a column-oriented cache program stored in the memory 1005 and perform the following operations:
acquiring column type format information of data to be cached;
allocating memory blocks for the data to be cached in memory resources according to the column-type format information;
and adding the memory block into a continuous virtual memory, and generating metadata of the data to be cached, wherein the metadata includes a corresponding relationship between the column format information and the virtual memory, and a mapping relationship between the virtual memory and the memory block.
Further, after the step of adding the memory block to the continuous virtual memory and generating the metadata of the data to be cached, the processor 1001 may be further configured to call a column-oriented caching program stored in the memory 1005, and perform the following operations:
after the data to be cached is subjected to columnar caching to obtain columnar cache data, a target virtual row or a target column in the columnar cache data is appointed for a parallel process or a parallel thread;
and calling the parallel process or the parallel thread to determine a virtual memory block corresponding to the target virtual row or the target column according to the metadata, and performing parallel processing on data in the virtual memory block.
Further, after the step of adding the memory block to the continuous virtual memory and generating the metadata of the data to be cached, the processor 1001 may be further configured to call a column-oriented caching program stored in the memory 1005, and perform the following operations:
after the data to be cached is subjected to line-type caching to obtain line-type caching data, determining a target memory block corresponding to the data to be written in the line-type caching data according to the metadata;
copying the content in the target memory block to a copied memory block;
and updating the metadata, wherein the data to be written in the updated metadata is associated with the copied memory block correspondingly.
Further, after the step of adding the memory block to the continuous virtual memory and generating the metadata of the data to be cached, the processor 1001 may be further configured to call a column-oriented caching program stored in the memory 1005, and perform the following operations:
after the data to be cached is subjected to line-type caching to obtain line-type caching data, determining a target memory block corresponding to the data to be modified in the line-type caching data according to the metadata;
and transmitting the data to be modified and the modified value into a concurrent written assembly instruction, and executing the concurrent written assembly instruction to update the data to be modified in the target memory block into the modified value.
Further, after the step of adding the memory block to the continuous virtual memory and generating the metadata of the data to be cached, the processor 1001 may be further configured to call a column-oriented caching program stored in the memory 1005, and perform the following operations:
after the data to be cached is subjected to line-type caching to obtain line-type caching data, determining a target virtual block from the line-type caching data;
detecting whether there is an already locked virtual block in the metadata that overlaps the target virtual block;
if not, adding the virtual row information or the column information corresponding to the target virtual block into the metadata to lock the target virtual block;
and unlocking the target virtual block after the writing of the target virtual block is finished.
Further, after the step of adding the memory block to the continuous virtual memory and generating the metadata of the data to be cached, the processor 1001 may be further configured to call a column-oriented caching program stored in the memory 1005, and perform the following operations:
after the data to be cached is subjected to line-type caching to obtain line-type caching data, determining a target memory block corresponding to the data to be processed in the line-type caching data according to the metadata;
and calling an instruction in the single instruction multiple data stream instruction set to process the target memory block.
Further, the step of allocating a memory block for the data to be cached in the memory resource according to the column information includes:
when the column in the column format information is a column with fixed element size, calculating the actual size of the column according to the number of the elements in the column and the data type, and allocating the memory block with the actual size to the column in the memory resource;
when a column in the column format information is a column with a non-fixed element size, allocating a memory block with a default size to the column in memory resources, and when the default size is smaller than the actual size of the column, allocating a memory block with the default size to the column again.
Based on the above hardware structure, various embodiments of the column-oriented caching method of the present invention are proposed.
Referring to fig. 2, a first embodiment of a column-oriented caching method of the present invention provides a column-oriented caching method, it being noted that although a logical order is shown in the flow chart, in some cases, the steps shown or described may be performed in an order different than that shown here. The execution subject of each embodiment of the column-oriented caching method of the present invention may be a terminal device such as a PC, a smart phone, a smart television, a tablet computer, and a portable computer, and for convenience of description, the following embodiments are explained with reference to the caching system execution subject. The column-oriented caching method comprises the following steps:
step S10, acquiring column format information of the data to be cached;
the cache system can be a system program installed in the terminal device, runs on the basis of an operating system, and can perform data interaction with other application programs. The columnar data format is a data format in which data is organized in columns, and the data type of each column is the same. The data to be cached may be data stored in an external storage, and the storage format in the external storage may be a columnar data format or other formats. The cache system can acquire column type format information of the data to be cached, and when the data to be cached is the data in the column type data format, the cache system can directly acquire the column type format information corresponding to the data to be cached; when the data to be cached is data in other formats, the caching system can convert the data to be cached into data in a column type data format to generate column type format information of the data to be cached; or other programs can perform format conversion and transmit the obtained columnar format information to the cache system. The columnar format information may be information for describing how many columns the data to be cached is organized in a columnar format, for example, information including how many columns, an index of each column, nesting information of columns and sub-columns, the number of cells of each column, a data type of each column, how many storage batches are included or which columns each storage batch includes, and the like. The data to be cached is cached in a columnar data format, and the continuous memory is used for storing the cell data with the same data type, so that higher reading and writing efficiency and higher calculation efficiency than that of the line data format cache can be provided.
Step S20, allocating memory blocks for the data to be cached in the memory resources according to the column format information;
and the cache system allocates memory blocks for the data to be cached in the memory resources according to the columnar format information. The cache system may allocate and manage the memory in the process space, may allocate and manage the memory outside the managed heap for a programming language (e.g., java python) supporting a GC (Garbage Collection), reduces the waste of computing resources caused by the GC, and may also allocate and manage the memory in the shared memory and the video memory. For a programming language supporting GC, the memory resource is in an array structure; NPU (network-processing units) memory, video memory, shared memory, and the like are memory resources allocated by a malloc-like function, and the starting address and the length of the memory resources are recorded, for example, a cache system allocates a memory block for data to be cached in the memory resources with the starting address of 0 and the length of 100.
Specifically, there are many ways for the cache system to allocate memory blocks to the data to be cached according to the columnar format information, for example, the data to be cached may be divided into data blocks according to storage batches or columns according to the columnar format information, and one memory block is allocated to each data block. The memory blocks are not necessarily consecutive, and the sizes of the memory blocks are not necessarily the same. The memory blocks form a memory pool, and the memory blocks in the memory pool are organized in a tree, graph or linked list mode, so that the memory allocation efficiency is improved, and meanwhile, the memory can be flexibly increased or reduced.
Further, step S20 includes:
step S201, when a column in the column format information is a column with a fixed element size, calculating an actual size of the column according to the number of elements in the column and a data type, and allocating a memory block of the actual size to the column in a memory resource;
in this embodiment, a feasible memory block allocation method is as follows: when a column in the column format information is a column with a fixed element size, the cache system may calculate the actual size of the column according to the number of elements in the column and the data type, and allocate a memory block of the actual size to the column in the memory resources. That is, a column of data is allocated to a memory block, and one memory block is continuous and is used for caching a column of data with the same data type, so that the data access efficiency is improved. Wherein, the element is fixed in size means that the data type of the element belongs to a fixed-length type, such as integer. If the data type of an element in a column is integer, the integer occupies 4 bytes, and the column has 100 elements in total, then the actual size S of the column is 4 × 100 — 400 bytes, and then the cache system divides a memory block of 400 bytes in the memory resources for caching the column of data.
Step S202, when a column in the column format information is a column with a non-fixed element size, allocating a memory block with a default size to the column in the memory resource, and when the default size is smaller than the actual size of the column, allocating a memory block with the default size to the column again.
When a column in the columnar format information is a column with an unfixed element size, the cache system may allocate a memory block with a default size to the column first, and may set the default size in advance according to a requirement. If the default size is smaller than the actual size of the row when the row of data is cached, that is, the memory blocks with the default size store no data of the row, then allocating one or more memory blocks with the default size to the row, so that the allocated memory blocks can store the row of data. Where the size of an element is not fixed, it means that the data type of the element belongs to a type of a non-fixed length, for example, a string type.
Step S30, adding the memory block to a continuous virtual memory, and generating metadata of the data to be cached, where the metadata includes a corresponding relationship between the column format information and the virtual memory, and a mapping relationship between the virtual memory and the memory block.
The cache system adds the memory blocks to the continuous virtual memory and generates metadata of the data to be cached. Specifically, the virtual memory is a continuous memory segment, does not occupy the actual memory, and is composed of a virtual address segment. The virtual memory blocks in the virtual memory are continuous. The cache system allocates a virtual memory block with the same size as the memory block from the virtual memory, and the newly allocated virtual memory block is connected with the existing virtual memory block in the virtual memory, and records the mapping relationship between the memory block and the virtual memory block by using metadata, so as to add the memory block to the virtual memory. For example, the first address of the memory block 1 in the memory resource is 11, the length is 10, the cache system allocates a virtual memory block 1 with the length of 10 in the virtual memory, and since the virtual memory does not yet have a virtual memory block, the first address of the newly allocated virtual memory block 1 is 0; the first address of the memory block 2 in the memory resource is 50, the length is 12, the cache system allocates a virtual memory block 2 with the length of 12 in the virtual memory, the virtual memory has a virtual memory block 1 with the length of 10 and the first address of 0, so the first address of the newly allocated virtual memory block 2 is 10; the metadata records a mapping relationship between the virtual memory and the memory block, for example: 11-20 of the memory resources correspond to 0-9 of the virtual memory, and 50-61 of the memory resources correspond to 10-21 of the virtual memory. The metadata further includes a corresponding relationship between the column format information and the virtual memory, for example, which column or columns are stored in the virtual memory block in the virtual memory, and information such as the data type of each column. Fig. 3 is a schematic diagram of a metadata record mapping relationship, where each mem-block is a memory block, is continuous in the virtual memory, and is dispersed in the memory resources.
The cache system allocates memory blocks for the data to be cached, and caches the data to be cached in the corresponding memory blocks, thereby obtaining the column-type cache data. Because the cache system records the corresponding relationship between the column format information of the column-type cache data and the virtual memory and the mapping relationship between the virtual memory and the memory block through the metadata, for the upper column-type cache interface, the data is cached in a section of completely continuous virtual memory, and can be mapped to the actually stored memory block through the metadata when specific contents are accessed, so that a column-type cache method capable of sensing the memory is realized, namely, the memory address where the data is stored can be clearly known through the metadata. When the cross-process calculation is performed, according to the conventional caching method, data needs to be serialized and transmitted between processes, and then the data integrity is ensured through deserialization reduction. Because serialization and deserialization operations are not needed, computing resources are saved during cross-process computing, and computing efficiency is improved.
Further, based on the first embodiment, a second embodiment of the present invention provides a column-oriented caching method. In this embodiment, after the step S30, the method further includes:
step A10, after the data to be cached is cached in a column type to obtain column type cache data, a target virtual row or a target column in the column type cache data is appointed for a parallel process or a parallel thread;
a20, invoking the parallel process or the parallel thread to determine, according to the metadata, a virtual memory block corresponding to the target virtual row or the target rank, and performing parallel processing on data in the virtual memory block.
The cache system performs the line-type cache on the data to be cached according to the cache manner in the above embodiment, and after obtaining the line-type cache data, the cache system may perform various operations such as reading, writing or calculating on the line-type cache data.
The caching system may specify a range of column indices, or set of indices, to create a column-level view of the columnar cached data; virtual rows may also be supported on the columnar cache data, specifying a range or set of row indices, creating a row-level view of the columnar cache data. By combining the two modes, the views split into rows and columns can be flexibly combined, and when the columnar cache data is accessed by using the row or column indexes, the real indexes are mapped by creating the parameters of the views.
When there are parallel computing tasks, the cache system may specify a target virtual row or a target rank in the columnar cache data for a parallel process or a parallel thread. And calling a parallel process or a parallel thread to perform parallel computation on the data of the target virtual row or the target column. Specifically, the parallel process or the parallel thread determines the virtual memory block corresponding to the target virtual row or the target rank according to the corresponding relationship between the columnar format information in the metadata and the virtual memory, and then performs parallel processing on the data in the virtual memory block. Because the process or the thread can directly determine the memory block where the target virtual row or the target column is located through the mapping information in the metadata, the process or the thread can directly read the data in the memory block, and during writing, non-overlapping virtual rows or non-overlapping virtual columns can be specified for the parallel process or the parallel thread through the cache system, so that the parallel process or the thread can simultaneously write the column-wise cache data in the memory. Therefore, the problem that the conventional cache system cannot support multi-process or multi-thread parallel computing is solved, and the computing capability of the chip is fully exerted.
Further, based on the first and second embodiments, a third embodiment of the present invention provides a column-oriented caching method. In this embodiment, after the step S30, the method further includes:
step B10, after performing line-type caching on the data to be cached to obtain line-type cached data, determining a target memory block corresponding to the data to be written in the line-type cached data according to the metadata;
step B20, copying the content in the target memory block to a copied memory block;
step B30, updating the metadata, where the data to be written in the updated metadata is associated with the copied memory block correspondingly.
In this embodiment, after the caching system performs columnar caching on data to be cached based on the caching method in the foregoing embodiments and obtains the columnar cached data, based on the characteristics of the columnar cached data, the caching system can support multiple concurrent read-write or calculation strategies and can improve processing efficiency, where the concurrent strategies include, but are not limited to, copy-on-write, concurrent write, fine-grained lock, and instruction optimization.
Specifically, copy-on-write refers to that when a new process is created, all resources are not copied to the newly created process, but when data needs to be written, pages to which data needs to be written are copied, so that resource waste caused by copy operation is avoided.
When the cache system reads data, the data in the memory block is not required to be copied, the data is directly mapped into the memory block according to the metadata, and the data is read from the memory block. During writing, the cache system may determine, according to the mapping relationship in the metadata, a target memory block corresponding to data to be written in the columnar cache data. For example, if the data to be written is column 1 in the column-wise cache data, the cache system searches for the metadata and determines the memory block corresponding to column 1. And then the cache system copies the content in the target memory block into a copy memory block, wherein the copy memory block is a memory block redistributed in the memory resources and has the same size as the target memory block. After copying the content of the target memory block, the cache system updates the metadata, and updates the corresponding relationship between the data to be written in the original metadata and the target memory block into the corresponding association between the data to be written in and the copied memory block. The copy memory block corresponding to the data to be written can be directly indexed according to the metadata, and then the data in the copy memory block is directly written.
Further, after step S30, the method further includes:
step C10, after performing line-type caching on the data to be cached to obtain line-type cached data, determining a target memory block corresponding to the data to be modified in the line-type cached data according to the metadata;
step C20, the data to be modified and the modified value are transmitted into a concurrently written assembly instruction, and the concurrently written assembly instruction is executed to update the data to be modified in the target memory block to the modified value.
When writing concurrently, the cache system may determine, according to the mapping relationship in the metadata, a target memory block corresponding to data to be modified in the columnar cache data. And the cache system reads the data to be modified from the target memory block, transmits the data to be modified and the modified value into a concurrent written assembly instruction, and executes the concurrent written assembly instruction to update the data to be modified in the target memory block into the modified value. It should be noted that the assembly instruction for concurrent writing may be a cmpxchg instruction, and the cmpxchg instruction is an assembly instruction for solving the problem of concurrent writing. The cmpxchg instruction supports vectorized updating of multiple values, and the cmpxchg will write one or a batch of values (if supported by the chip) into memory in turn, and then be compiled into chip-level instructions, which is far more capable than using threads or process locks.
Further, after step S30, the method further includes:
step D10, after the data to be cached is subjected to line-type caching to obtain line-type caching data, determining a target virtual block from the line-type caching data;
step D20, detecting whether there is an already locked virtual block overlapping the target virtual block in the metadata;
step D30, if not, adding the virtual row information or column information corresponding to the target virtual block to the metadata to lock the target virtual block;
and D40, unlocking the target virtual block after the writing to the target virtual block is completed.
In this embodiment, the cache system may lock a virtual block, which is a part of the columnar cache data, for a process that concurrently operates the columnar cache data. Specifically, the cache system determines a virtual row or column to be operated by the process, and the determined virtual row or column constitutes a target virtual block. The metadata records the information of the locked virtual blocks, and the cache system detects whether the metadata has the locked virtual blocks overlapped with the target virtual block. And if the target virtual block and the locked virtual block have overlapped virtual rows or columns, determining that the target virtual block is overlapped with the locked virtual block. If no locked virtual block that overlaps the target virtual block is detected, the cache system may add virtual row information or column information corresponding to the target virtual block to the metadata to lock the target virtual block. After the process completes writing the target virtual block, the target virtual block is unlocked, which may be deleting the information of the target virtual block in the metadata. If the cache system detects a locked virtual block overlapped with the target virtual block, waiting until no locked virtual block overlapped with the target virtual block exists in the metadata, and locking the target virtual block; or directly return the failed information to the process.
In this embodiment, because the metadata records the corresponding relationship between the columnar format information of the columnar cache data and the virtual memory, and also records the mapping relationship between the virtual memory and the memory block, the position of the data cache in the actual memory can be determined according to the metadata, so as to directly read and write the data in the memory, and a fine-grained lock can be added to the columnar cache data according to the fine-grained columnar format information recorded in the metadata, thereby improving the read and write efficiency of the concurrent process.
Further, after the step S30, the method further includes:
step E10, after performing line caching on the data to be cached to obtain line-type cached data, determining a target memory block corresponding to the data to be processed in the line-type cached data according to the metadata;
step E20, call an instruction in the simd instruction set to process the target memory block.
Further, in this embodiment, the cache system may determine, according to the mapping relationship of the metadata, a target memory block corresponding to data to be processed in the columnar cache data. And calling a Single Instruction Multiple Data (SIMD) instruction set according to the operation task to process the Data to be processed in the target memory block. SIMD is a technique that uses one controller to control a plurality of processors, and performs the same operation on each of a set of data (also referred to as "data vector") at the same time, thereby achieving spatial parallelism. Because the column-type cache data is data cached in columns, a SIMD instruction set can be adopted to process a column of data, and the processing efficiency is improved.
The embodiment generally covers the requirements of different chips such as a GPU/NPU and the like, and is beneficial to localization, instruction pipelining or register optimization because the small blocks of memory can be continuously and randomly scheduled and mapped.
In addition, an embodiment of the present invention further provides a column-oriented cache device, and referring to fig. 4, the column-oriented cache device includes:
the acquisition module 10 is used for acquiring column-type format information of data to be cached;
an allocating module 20, configured to allocate memory blocks for the data to be cached in memory resources according to the column format information;
a generating module 30, configured to add the memory block to a continuous virtual memory, and generate metadata of the data to be cached, where the metadata includes a corresponding relationship between the column format information and the virtual memory, and a mapping relationship between the virtual memory and the memory block.
Further, the column-oriented cache device further includes:
the specifying module is used for specifying a target virtual row or a target column in the line-type cache data for a parallel process or a parallel thread after the line-type cache is carried out on the data to be cached to obtain the line-type cache data;
and a parallel processing module, configured to invoke the parallel process or the parallel thread, determine, according to the metadata, a virtual memory block corresponding to the target virtual row or the target rank, and perform parallel processing on data in the virtual memory block.
Further, the column-oriented cache device further includes:
the first determining module is configured to determine, according to the metadata, a target memory block corresponding to data to be written in the line-type cache data after the line-type cache is performed on the data to be cached to obtain the line-type cache data;
the copying module is used for copying the content in the target memory block to a copied memory block;
an updating module, configured to update the metadata, where the data to be written in the updated metadata is associated with the copied memory block correspondingly.
Further, after the step of adding the memory block to a continuous virtual memory and generating the metadata of the data to be cached, the method further includes:
a second determining module, configured to determine, according to the metadata, a target memory block corresponding to data to be modified in the line-type cache data after line-type caching is performed on the data to be cached to obtain line-type cache data;
and the execution module is configured to transmit the data to be modified and the modified value into a concurrently written assembly instruction, and execute the concurrently written assembly instruction to update the data to be modified in the target memory block to the modified value.
Further, after the step of adding the memory block to a continuous virtual memory and generating the metadata of the data to be cached, the method further includes:
the third determining module is used for determining a target virtual block from the column-wise cache data after the column-wise cache is performed on the data to be cached to obtain the column-wise cache data;
a detection module, configured to detect whether there is a locked virtual block that overlaps with the target virtual block in the metadata;
a locking module, configured to add virtual row information or column information corresponding to the target virtual block to the metadata to lock the target virtual block if the target virtual block is not found;
and the releasing module is used for releasing the locking of the target virtual block after the writing of the target virtual block is completed.
Further, after the step of adding the memory block to a continuous virtual memory and generating the metadata of the data to be cached, the method further includes:
a fourth determining module, configured to determine, according to the metadata, a target memory block corresponding to data to be processed in the line-type cache data after line-type caching is performed on the data to be cached to obtain line-type cache data;
and the calling module is used for calling the instructions in the single instruction multiple data stream instruction set to process the target memory block.
Further, the step S20 includes:
a first allocation unit, configured to, when a column in the column-wise format information is a column with a fixed element size, calculate an actual size of the column according to the number of elements in the column and a data type, and allocate a memory block of the actual size to the column in a memory resource;
a second allocating unit, configured to allocate, when a column in the column-wise format information is a column with a non-fixed element size, a memory block with a default size to the column in a memory resource, and allocate, when the default size is smaller than an actual size of the column, a memory block with the default size to the column.
The expansion content of the specific implementation of the column-oriented cache device of the present invention is basically the same as that of each embodiment of the column-oriented cache method described above, and is not described herein again.
Furthermore, an embodiment of the present invention further provides a computer-readable storage medium, where a column-oriented caching program is stored on the computer-readable storage medium, and the steps of the column-oriented caching method are implemented when the column-oriented caching program is executed by a processor.
The expansion contents of the specific implementation modes of the column-oriented caching device and the computer-readable storage medium of the present invention are basically the same as those of the above-mentioned embodiments of the column-oriented caching method, and are not described herein again.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or system. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or system that comprises the element.
The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.
Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium (e.g., ROM/RAM, magnetic disk, optical disk) as described above and includes instructions for enabling a terminal device (e.g., a mobile phone, a computer, a server, an air conditioner, or a network device) to execute the method according to the embodiments of the present invention.
The above description is only a preferred embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications of equivalent structures and equivalent processes, which are made by using the contents of the present specification and the accompanying drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.
Claims (10)
1. A column-oriented caching method, comprising:
acquiring column type format information of data to be cached;
allocating memory blocks for the data to be cached in memory resources according to the column-type format information;
and adding the memory block into a continuous virtual memory, and generating metadata of the data to be cached, wherein the metadata includes a corresponding relationship between the column format information and the virtual memory, and a mapping relationship between the virtual memory and the memory block.
2. The column-oriented caching method according to claim 1, wherein after the step of adding the memory block to a continuous virtual memory and generating the metadata of the data to be cached, the method further comprises:
after the data to be cached is subjected to columnar caching to obtain columnar cache data, a target virtual row or a target column in the columnar cache data is appointed for a parallel process or a parallel thread;
and calling the parallel process or the parallel thread to determine a virtual memory block corresponding to the target virtual row or the target column according to the metadata, and performing parallel processing on data in the virtual memory block.
3. The column-oriented caching method according to claim 1, wherein after the step of adding the memory block to a continuous virtual memory and generating the metadata of the data to be cached, the method further comprises:
after the data to be cached is subjected to line-type caching to obtain line-type caching data, determining a target memory block corresponding to the data to be written in the line-type caching data according to the metadata;
copying the content in the target memory block to a copied memory block;
and updating the metadata, wherein the data to be written in the updated metadata is associated with the copied memory block correspondingly.
4. The column-oriented caching method according to claim 1, wherein after the step of adding the memory block to a continuous virtual memory and generating the metadata of the data to be cached, the method further comprises:
after the data to be cached is subjected to line-type caching to obtain line-type caching data, determining a target memory block corresponding to the data to be modified in the line-type caching data according to the metadata;
and transmitting the data to be modified and the modified value into a concurrent written assembly instruction, and executing the concurrent written assembly instruction to update the data to be modified in the target memory block into the modified value.
5. The column-oriented caching method according to claim 1, wherein after the step of adding the memory block to a continuous virtual memory and generating the metadata of the data to be cached, the method further comprises:
after the data to be cached is subjected to line-type caching to obtain line-type caching data, determining a target virtual block from the line-type caching data;
detecting whether there is an already locked virtual block in the metadata that overlaps the target virtual block;
if not, adding the virtual row information or the column information corresponding to the target virtual block into the metadata to lock the target virtual block;
and unlocking the target virtual block after the writing of the target virtual block is finished.
6. The column-oriented caching method according to claim 1, wherein after the step of adding the memory block to a continuous virtual memory and generating the metadata of the data to be cached, the method further comprises:
after the data to be cached is subjected to line-type caching to obtain line-type caching data, determining a target memory block corresponding to the data to be processed in the line-type caching data according to the metadata;
and calling an instruction in the single instruction multiple data stream instruction set to process the target memory block.
7. The column-oriented caching method according to any one of claims 1 to 6, wherein the step of allocating memory blocks for the data to be cached in the memory resources according to the column information includes:
when the column in the column format information is a column with fixed element size, calculating the actual size of the column according to the number of the elements in the column and the data type, and allocating the memory block with the actual size to the column in the memory resource;
when a column in the column format information is a column with a non-fixed element size, allocating a memory block with a default size to the column in memory resources, and when the default size is smaller than the actual size of the column, allocating a memory block with the default size to the column again.
8. A column-oriented cache device, the column-oriented cache device comprising:
the acquisition module is used for acquiring the column format information of the data to be cached;
the allocation module is used for allocating memory blocks for the data to be cached in the memory resources according to the column format information;
a generating module, configured to add the memory block to a continuous virtual memory, and generate metadata of the data to be cached, where the metadata includes a correspondence between the column format information and the virtual memory, and a mapping relationship between the virtual memory and the memory block.
9. A column-oriented cache device, characterized in that the column-oriented cache device comprises a memory, a processor and a column-oriented cache program stored on the memory and operable on the processor, which when executed by the processor implements the steps of the column-oriented cache method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that the computer-readable storage medium has stored thereon a column-oriented caching program, which when executed by a processor implements the steps of the column-oriented caching method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911326754.5A CN111104067B (en) | 2019-12-20 | 2019-12-20 | Cache method, device, equipment and computer readable storage medium for alignment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911326754.5A CN111104067B (en) | 2019-12-20 | 2019-12-20 | Cache method, device, equipment and computer readable storage medium for alignment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111104067A true CN111104067A (en) | 2020-05-05 |
| CN111104067B CN111104067B (en) | 2024-01-12 |
Family
ID=70422737
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911326754.5A Active CN111104067B (en) | 2019-12-20 | 2019-12-20 | Cache method, device, equipment and computer readable storage medium for alignment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111104067B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113296961A (en) * | 2021-07-22 | 2021-08-24 | 广州中望龙腾软件股份有限公司 | GPU-based dynamic memory allocation method and device and memory linked list |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102609305A (en) * | 2012-02-07 | 2012-07-25 | 中山爱科数字科技股份有限公司 | Method for sharing internal memory in server cluster |
| CN103092905A (en) * | 2011-11-07 | 2013-05-08 | Sap股份公司 | Columnar database using virtual file data objects |
| CN104516912A (en) * | 2013-09-29 | 2015-04-15 | 中国移动通信集团黑龙江有限公司 | Dynamic data storage method and device |
| CN104765574A (en) * | 2015-04-23 | 2015-07-08 | 成都博元时代软件有限公司 | Data cloud storage method |
| CN105144160A (en) * | 2013-03-15 | 2015-12-09 | 甲骨文国际公司 | A method to accelerate queries using dynamically generated alternate data formats in flash cache |
| CN107408132A (en) * | 2015-03-20 | 2017-11-28 | 国际商业机器公司 | The effective performance of insertion and point inquiry operation in row store |
| US20190096024A1 (en) * | 2017-09-26 | 2019-03-28 | Intel Corporation | Area-efficient implementations of graphics instructions |
| CN109542889A (en) * | 2018-10-11 | 2019-03-29 | 平安科技（深圳）有限公司 | Stream data column storage method, device, equipment and storage medium |
-
2019
- 2019-12-20 CN CN201911326754.5A patent/CN111104067B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103092905A (en) * | 2011-11-07 | 2013-05-08 | Sap股份公司 | Columnar database using virtual file data objects |
| CN102609305A (en) * | 2012-02-07 | 2012-07-25 | 中山爱科数字科技股份有限公司 | Method for sharing internal memory in server cluster |
| CN105144160A (en) * | 2013-03-15 | 2015-12-09 | 甲骨文国际公司 | A method to accelerate queries using dynamically generated alternate data formats in flash cache |
| CN104516912A (en) * | 2013-09-29 | 2015-04-15 | 中国移动通信集团黑龙江有限公司 | Dynamic data storage method and device |
| CN107408132A (en) * | 2015-03-20 | 2017-11-28 | 国际商业机器公司 | The effective performance of insertion and point inquiry operation in row store |
| CN104765574A (en) * | 2015-04-23 | 2015-07-08 | 成都博元时代软件有限公司 | Data cloud storage method |
| US20190096024A1 (en) * | 2017-09-26 | 2019-03-28 | Intel Corporation | Area-efficient implementations of graphics instructions |
| CN109542889A (en) * | 2018-10-11 | 2019-03-29 | 平安科技（深圳）有限公司 | Stream data column storage method, device, equipment and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 刘庆，韩蓓蓓, 电子科技大学出版社, pages: 185 - 201 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113296961A (en) * | 2021-07-22 | 2021-08-24 | 广州中望龙腾软件股份有限公司 | GPU-based dynamic memory allocation method and device and memory linked list |
| CN113296961B (en) * | 2021-07-22 | 2021-11-19 | 广州中望龙腾软件股份有限公司 | GPU-based dynamic memory allocation method and device and memory linked list |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111104067B (en) | 2024-01-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Wang et al. | Scimate: A novel mapreduce-like framework for multiple scientific data formats | |
| JP5147280B2 (en) | System and method for garbage collection in heterogeneous multiprocessor systems | |
| US10089119B2 (en) | API namespace virtualization | |
| US10007605B2 (en) | Hardware-based array compression | |
| US10817312B2 (en) | Programming model for performant computing in document-oriented storage services | |
| US20140195632A1 (en) | Immutable sharable zero-copy data and streaming | |
| EP2941704B1 (en) | Zero-copy caching | |
| US8316357B2 (en) | Type descriptor management for frozen objects | |
| US20160306739A1 (en) | Garbage collection of non-pinned objects within heap | |
| CN111104067B (en) | Cache method, device, equipment and computer readable storage medium for alignment | |
| Bhimani et al. | Fine-grained control of concurrency within KV-SSDs | |
| CN112214443B (en) | Secondary unloading device and method arranged in graphic processor | |
| CN104714792A (en) | Multi-process shared data processing method and device | |
| US20090007115A1 (en) | Method and apparatus for parallel XSL transformation with low contention and load balancing | |
| KR102130813B1 (en) | Re-configurable processor and method for operating re-configurable processor | |
| US8914779B2 (en) | Data placement for execution of an executable | |
| US9990303B2 (en) | Sharing data structures between processes by semi-invasive hybrid approach | |
| Ventocilla | Big data programming with Apache spark | |
| US10970189B2 (en) | Configuring data processing pipelines | |
| Scargall et al. | Volatile use of persistent memory | |
| KR102695529B1 (en) | Processing apparatus and operating method thereof and electronic apparatus including the processing apparatus | |
| CN112230931B (en) | Compiling method, device and medium suitable for secondary unloading of graphic processor | |
| Jones et al. | Gigapixel-size real-time interactive image processing with parallel computers | |
| US11954510B2 (en) | Native-image in-memory cache for containerized ahead-of-time applications | |
| CN116303135B (en) | Task data loading method and device and computer equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |