CN111104067B - Cache method, device, equipment and computer readable storage medium for alignment - Google Patents
Cache method, device, equipment and computer readable storage medium for alignment Download PDFInfo
- Publication number
- CN111104067B CN111104067B CN201911326754.5A CN201911326754A CN111104067B CN 111104067 B CN111104067 B CN 111104067B CN 201911326754 A CN201911326754 A CN 201911326754A CN 111104067 B CN111104067 B CN 111104067B
- Authority
- CN
- China
- Prior art keywords
- data
- column
- memory
- virtual
- cached
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 108
- 238000013507 mapping Methods 0.000 claims abstract description 19
- 230000008569 process Effects 0.000 claims description 44
- 238000012545 processing Methods 0.000 claims description 12
- 238000004364 calculation method Methods 0.000 abstract description 16
- 238000004891 communication Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 239000002699 waste material Substances 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000004807 localization Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 239000013598 vector Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
- G06F3/065—Replication mechanisms
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0662—Virtualisation aspects
Abstract
The invention discloses a cache method, a device, equipment and a computer readable storage medium for a column, wherein the method comprises the following steps: acquiring column format information of data to be cached; distributing memory blocks for the data to be cached in memory resources according to the column format information; and adding the memory blocks into a continuous virtual memory, and generating metadata of the data to be cached, wherein the metadata comprises a corresponding relation between the column format information and the virtual memory and a mapping relation between the virtual memory and the memory blocks. The invention realizes the operation of serialization and deserialization without performing the cross-process calculation, saves the calculation resources and improves the calculation efficiency.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a method, an apparatus, a device, and a computer readable storage medium for caching a group.
Background
With the development of technologies such as big data and artificial intelligence, the data processing efficiency of the computer is also required to be higher and higher. In the existing cache system, cross-process calculation requires serialization and deserialization operations, the process of converting an object into a byte sequence is called as serialization of the object, and the process of recovering the byte sequence into the object is called as deserialization of the object. Both of these processes require the computational resources of the computer to be expended, resulting in reduced efficiency of the computer in cross-process computing.
Disclosure of Invention
The invention mainly aims to provide a cache method, a cache device, cache equipment and a computer readable storage medium, which aim to solve the technical problem that the prior cache system needs serialization and anti-serialization operations in cross-process calculation and wastes calculation resources.
In order to achieve the above object, the present invention provides a column-oriented caching method, where the column-oriented caching method includes:
acquiring column format information of data to be cached;
distributing memory blocks for the data to be cached in memory resources according to the column format information;
and adding the memory blocks into a continuous virtual memory, and generating metadata of the data to be cached, wherein the metadata comprises a corresponding relation between the column format information and the virtual memory and a mapping relation between the virtual memory and the memory blocks.
Optionally, after the step of adding the memory block to the continuous virtual memory and generating the metadata of the data to be cached, the method further includes:
after the data to be cached is subjected to determinant caching to obtain columnar cache data, designating a target virtual row or a target column in the columnar cache data for a parallel process or a parallel thread;
And calling the parallel process or the parallel thread to determine a virtual memory block corresponding to the target virtual row or the target column according to the metadata, and carrying out parallel processing on data in the virtual memory block.
Optionally, after the step of adding the memory block to the continuous virtual memory and generating the metadata of the data to be cached, the method further includes:
after the data to be cached is subjected to determinant caching to obtain columnar cache data, determining a target memory block corresponding to data to be written in the columnar cache data according to the metadata;
copying the content in the target memory block into a copied memory block;
and updating the metadata, wherein the data to be written in the updated metadata is correspondingly associated with the copy memory block.
Optionally, after the step of adding the memory block to the continuous virtual memory and generating the metadata of the data to be cached, the method further includes:
after the data to be cached is subjected to determinant caching to obtain columnar cache data, determining a target memory block corresponding to the data to be modified in the columnar cache data according to the metadata;
And transmitting the data to be modified and the modified value into a concurrent written assembly instruction, and executing the concurrent written assembly instruction to update the data to be modified in the target memory block into the modified value.
Optionally, after the step of adding the memory block to the continuous virtual memory and generating the metadata of the data to be cached, the method further includes:
after the data to be cached is subjected to determinant caching to obtain columnar cache data, determining a target virtual block from the columnar cache data;
detecting whether the metadata has a locked virtual block overlapped with the target virtual block or not;
if not, adding virtual row information or column information corresponding to the target virtual block into the metadata to lock the target virtual block;
and unlocking the target virtual block after the writing of the target virtual block is completed.
Optionally, after the step of adding the memory block to the continuous virtual memory and generating the metadata of the data to be cached, the method further includes:
after the data to be cached is subjected to determinant caching to obtain columnar cache data, determining a target memory block corresponding to the data to be processed in the columnar cache data according to the metadata;
And calling an instruction in the single-instruction multi-data stream instruction set to process the target memory block.
Optionally, the step of allocating a memory block for the data to be cached in the memory resource according to the column information includes:
when the column in the column format information is a column with fixed element size, calculating the actual size of the column according to the number of elements in the column and the data type, and distributing the memory blocks with the actual size for the column in a memory resource;
when the column in the column format information is a column with non-fixed element size, memory blocks with default size are allocated to the column in the memory resource, and when the default size is smaller than the actual size of the column, the memory blocks with default size are allocated to the column.
In addition, in order to achieve the above object, the present invention further provides a column-oriented cache device, where the column-oriented cache device includes:
the acquisition module is used for acquiring column format information of the data to be cached;
the distribution module is used for distributing memory blocks for the data to be cached in the memory resources according to the column format information;
the generating module is used for adding the memory blocks into a continuous virtual memory and generating metadata of the data to be cached, wherein the metadata comprise a corresponding relation between the column format information and the virtual memory and a mapping relation between the virtual memory and the memory blocks.
In addition, to achieve the above object, the present invention further provides a column-oriented cache device, where the column-oriented cache device includes a memory, a processor, and a column-oriented cache program stored on the memory and executable on the processor, where the column-oriented cache program implements the steps of the column-oriented cache method as described above when executed by the processor.
In addition, to achieve the above object, the present invention further provides a computer readable storage medium having stored thereon a column-oriented caching program which, when executed by a processor, implements the steps of the column-oriented caching method as described above.
According to the method, the device and the system, the memory blocks are allocated for the data to be cached according to the column format information, and the memory blocks are added into the continuous virtual memory to generate metadata of the data to be cached, so that the column cache data are obtained. The corresponding relation between the columnar format information of the columnar cache data and the virtual memory and the mapping relation between the virtual memory and the memory blocks are recorded through the metadata, so that for an upper-layer columnar cache interface, the data is cached in a section of completely continuous virtual memory, and when specific contents are accessed, the data can be mapped to the actually stored memory blocks through the metadata, and therefore the column cache method with the perceivable memory is realized, namely, the memory address where the data is stored can be clearly known through the metadata. When the cross-process calculation is performed, according to the traditional caching method, the data are required to be serialized and then transferred between the processes, and then the data are restored through deserialization, so that the integrity of the data is ensured. Because the serialization and deserialization operations are not needed, the calculation resources are saved during the cross-process calculation, and the calculation efficiency is improved.
Drawings
FIG. 1 is a schematic diagram of a hardware operating environment according to an embodiment of the present invention;
FIG. 2 is a flow chart illustrating a first embodiment of a column-oriented caching method according to the present invention;
FIG. 3 is a schematic diagram of a metadata record mapping relationship according to an embodiment of the present invention;
FIG. 4 is a block diagram of a preferred embodiment of a column-oriented cache device according to the present invention.
The achievement of the objects, functional features and advantages of the present invention will be further described with reference to the accompanying drawings, in conjunction with the embodiments.
Detailed Description
It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.
The embodiment of the invention provides a cache device facing a column, and referring to fig. 1, fig. 1 is a schematic structural diagram of a hardware running environment related to an embodiment of the invention.
It should be noted that fig. 1 may be a schematic structural diagram of a hardware running environment of a cache device facing a column. The column-oriented cache device in the embodiment of the invention can be a PC, and also can be terminal devices with display functions, such as smart phones, smart televisions, tablet computers, portable computers and the like.
As shown in fig. 1, the column-oriented caching device may include: a processor 1001, such as a CPU, a network interface 1004, a user interface 1003, a memory 1005, a communication bus 1002. Wherein the communication bus 1002 is used to enable connected communication between these components. The user interface 1003 may include a Display, an input unit such as a Keyboard (Keyboard), and the optional user interface 1003 may further include a standard wired interface, a wireless interface. The network interface 1004 may optionally include a standard wired interface, a wireless interface (e.g., WI-FI interface). The memory 1005 may be a high-speed RAM memory or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also optionally be a storage device separate from the processor 1001 described above.
Optionally, the column-oriented caching device may further include a camera, an RF (Radio Frequency) circuit, a sensor, an audio circuit, a WiFi module, and so on. It will be appreciated by those skilled in the art that the array-facing cache device structure shown in fig. 1 does not constitute a limitation of the array-facing cache device and may include more or fewer components than shown, or may combine certain components, or may be arranged in a different arrangement of components.
As shown in FIG. 1, an operating system, a network communication module, a user interface module, and a column-oriented caching program may be included in memory 1005, which is a type of computer storage medium.
In the column-oriented cache device shown in fig. 1, the network interface 1004 is mainly used for connecting to a background server and performing data communication with the background server; the user interface 1003 is mainly used for connecting a client (user side) and performing data communication with the client; and the processor 1001 may be configured to call a cache program for a cache line stored in the memory 1005, and perform the following operations:
acquiring column format information of data to be cached;
distributing memory blocks for the data to be cached in memory resources according to the column format information;
And adding the memory blocks into a continuous virtual memory, and generating metadata of the data to be cached, wherein the metadata comprises a corresponding relation between the column format information and the virtual memory and a mapping relation between the virtual memory and the memory blocks.
Further, after the step of adding the memory block to the continuous virtual memory and generating the metadata of the data to be cached, the processor 1001 may be further configured to call a cache program facing the column stored in the memory 1005 to perform the following operations:
after the data to be cached is subjected to determinant caching to obtain columnar cache data, designating a target virtual row or a target column in the columnar cache data for a parallel process or a parallel thread;
and calling the parallel process or the parallel thread to determine a virtual memory block corresponding to the target virtual row or the target column according to the metadata, and carrying out parallel processing on data in the virtual memory block.
Further, after the step of adding the memory block to the continuous virtual memory and generating the metadata of the data to be cached, the processor 1001 may be further configured to call a cache program facing the column stored in the memory 1005 to perform the following operations:
After the data to be cached is subjected to determinant caching to obtain columnar cache data, determining a target memory block corresponding to data to be written in the columnar cache data according to the metadata;
copying the content in the target memory block into a copied memory block;
and updating the metadata, wherein the data to be written in the updated metadata is correspondingly associated with the copy memory block.
Further, after the step of adding the memory block to the continuous virtual memory and generating the metadata of the data to be cached, the processor 1001 may be further configured to call a cache program facing the column stored in the memory 1005 to perform the following operations:
after the data to be cached is subjected to determinant caching to obtain columnar cache data, determining a target memory block corresponding to the data to be modified in the columnar cache data according to the metadata;
and transmitting the data to be modified and the modified value into a concurrent written assembly instruction, and executing the concurrent written assembly instruction to update the data to be modified in the target memory block into the modified value.
Further, after the step of adding the memory block to the continuous virtual memory and generating the metadata of the data to be cached, the processor 1001 may be further configured to call a cache program facing the column stored in the memory 1005 to perform the following operations:
After the data to be cached is subjected to determinant caching to obtain columnar cache data, determining a target virtual block from the columnar cache data;
detecting whether the metadata has a locked virtual block overlapped with the target virtual block or not;
if not, adding virtual row information or column information corresponding to the target virtual block into the metadata to lock the target virtual block;
and unlocking the target virtual block after the writing of the target virtual block is completed.
Further, after the step of adding the memory block to the continuous virtual memory and generating the metadata of the data to be cached, the processor 1001 may be further configured to call a cache program facing the column stored in the memory 1005 to perform the following operations:
after the data to be cached is subjected to determinant caching to obtain columnar cache data, determining a target memory block corresponding to the data to be processed in the columnar cache data according to the metadata;
and calling an instruction in the single-instruction multi-data stream instruction set to process the target memory block.
Further, the step of allocating a memory block to the data to be cached in the memory resource according to the column information includes:
When the column in the column format information is a column with fixed element size, calculating the actual size of the column according to the number of elements in the column and the data type, and distributing the memory blocks with the actual size for the column in a memory resource;
when the column in the column format information is a column with non-fixed element size, memory blocks with default size are allocated to the column in the memory resource, and when the default size is smaller than the actual size of the column, the memory blocks with default size are allocated to the column.
Based on the above hardware structure, various embodiments of the present invention are presented for a column-oriented caching method.
Referring to fig. 2, a first embodiment of a column-oriented caching method of the present invention provides a column-oriented caching method, and it should be noted that although a logical order is illustrated in the flowchart, in some cases the steps illustrated or described may be performed in an order different than that illustrated herein. The execution subject of each embodiment of the cache method of the present invention may be a terminal device such as a PC, a smart phone, a smart television, a tablet computer, and a portable computer, and for convenience of description, the execution subject of the cache system is described in the following embodiments. The cache method for the array comprises the following steps:
Step S10, obtaining column format information of data to be cached;
the cache system can be a system program installed in the terminal equipment, runs on the basis of an operating system and can perform data interaction with other application programs. A columnar data format is a data format in which data is organized in columns, with the data type for each column being the same. The data to be cached may be data stored in an external storage, and the storage format in the external storage may be a column data format or other formats. The cache system can acquire the columnar format information of the data to be cached, and when the data to be cached is the data in the columnar data format, the cache system can directly acquire the columnar format information corresponding to the data to be cached; when the data to be cached is the data in other formats, the caching system can convert the data to be cached into the data in the column-type data format, and column-type format information of the data to be cached is generated; other programs may perform format conversion and then transfer the obtained columnar format information to the cache system. The columnar format information may be information describing that data to be cached is organized in a columnar format, for example, information including how many columns, indexes of each column, nesting information of columns and sub-columns, the number of cells of each column, the data type of each column, how many storage lots are included or which columns each storage lot includes, and the like. The data to be cached is cached in the column data format, and the continuous memory is used for storing the cell data with the same data type, so that higher reading-writing and calculation efficiency can be provided compared with the line data format cache.
Step S20, memory blocks are allocated to the data to be cached in the memory resources according to the column format information;
and the cache system allocates memory blocks for the data to be cached in the memory resources according to the column format information. The cache system can allocate and manage the memory in the process space, and for programming languages (such as java python) supporting GC (Garbage Collection), the memory can be allocated and managed externally in the managed heap, so that the computing resource waste caused by GC is reduced, and the memory can be allocated and managed in the shared memory and the video memory. For a programming language supporting GC, the memory resource is a array structure; NPU (network processor) memory, video memory, shared memory and the like are memory resources allocated through a malloc-like function, the starting address and the length of the memory resources are recorded, for example, a cache system allocates memory blocks for data to be cached in the memory resources with the starting address of 0 and the length of 100.
Specifically, the cache system allocates memory blocks for the data to be cached according to the column format information, for example, the data to be cached may be divided into data blocks according to the column format information, and each data block is allocated with a memory block. The memory blocks are not necessarily continuous, and the sizes of the memory blocks are not necessarily the same. Each memory block forms a memory pool, the memory blocks in the memory pool are organized in a tree, graph or linked list mode, the memory allocation efficiency is improved, and meanwhile, the memory can be flexibly increased or reduced.
Further, step S20 includes:
step S201, when a column in the column format information is a column with a fixed element size, calculating the actual size of the column according to the number of elements in the column and the data type, and distributing a memory block with the actual size for the column in a memory resource;
in this embodiment, a feasible memory block allocation method is: when a column in the column format information is a column with a fixed element size, the cache system may calculate the actual size of the column according to the number of elements in the column and the data type, and allocate a memory block of the actual size for the column in the memory resource. That is, a column of data is allocated with a memory block, and a memory block is continuous and is used for caching a column of data with the same data type, so that the data access efficiency is improved. Where the size of an element is fixed, it means that the data type of the element is of a fixed length type, such as integer. If the data type of the element in a column is integer, the integer occupies 4 byte lengths, and the total number of the elements in the column is 100, the actual size s=4x100=400 bytes of the column, and the cache system divides a memory block of 400 bytes in the memory resource for caching the column data.
Step S202, when the column in the column format information is a column with a non-fixed element size, allocating a memory block with a default size to the column in the memory resource, and when the default size is smaller than the actual size of the column, allocating the memory block with the default size to the column.
When the column in the column format information is a column with an unfixed element size, the cache system can allocate a memory block with a default size for the column first, and can set the default size according to the requirement in advance. If the default size is smaller than the actual size of the column when the column data is cached, that is, when the memory blocks of the default size store the data of the column, one or more memory blocks of the default size are allocated to the column, so that the allocated memory blocks can store the column data. Wherein the size of an element is not fixed means that the data type of the element belongs to a type of an unfixed length, for example, a string type.
Step S30, adding the memory blocks into a continuous virtual memory, and generating metadata of the data to be cached, wherein the metadata comprise a corresponding relation between the column format information and the virtual memory and a mapping relation between the virtual memory and the memory blocks.
The caching system adds the memory blocks into the continuous virtual memory and generates metadata of the data to be cached. Specifically, the virtual memory is a section of continuous memory, does not occupy actual memory, and is composed of a section of virtual addresses. The virtual memory blocks in the virtual memory are contiguous. The cache system allocates a virtual memory block with the same size as the memory block from the virtual memory, the newly allocated virtual memory block is connected with the existing virtual memory block in the virtual memory, and the mapping relation between the memory block and the virtual memory block is recorded by adopting metadata, so that the memory block is added into the virtual memory. For example, the first address of the memory block 1 in the memory resource is 11, the length is 10, the cache system allocates a segment of virtual memory block 1 with the length of 10 in the virtual memory, and since there is no virtual memory block in the virtual memory, the first address of the newly allocated virtual memory block 1 is 0; the head address of the memory block 2 in the memory resource is 50, the length is 12, the buffer memory system allocates a section of virtual memory block 2 with the length of 12 in the virtual memory, and the virtual memory has a virtual memory block 1 with the length of 10 and the head address of 0, so the head address of the newly allocated virtual memory block 2 is 10; the metadata records the mapping relationship between the virtual memory and the memory block, for example: 11-20 of the memory resources correspond to 0-9 of the virtual memory, and 50-61 of the memory resources correspond to 10-21 of the virtual memory. The metadata also includes a correspondence between the columnar format information and the virtual memory, for example, which column or columns the virtual memory block in the virtual memory corresponds to and stores, and the data type of each column. As shown in fig. 3, a mapping relationship of metadata records is shown, in which each mem-block is a memory block, and is continuous in virtual memory and dispersed in memory resources.
The caching system allocates memory blocks for the data to be cached, and caches the data to be cached in the corresponding memory blocks, so that columnar cache data are obtained. The cache system records the corresponding relation between the columnar format information of the columnar cache data and the virtual memory and the mapping relation between the virtual memory and the memory blocks through the metadata, so that for an upper-layer columnar cache interface, the data is cached in a section of completely continuous virtual memory, and when specific contents are accessed, the data can be mapped to the actually stored memory blocks through the metadata, and therefore a column cache method with a perceptible memory is realized, namely, the memory address where the data is stored can be clearly known through the metadata. When the cross-process calculation is performed, according to the traditional caching method, the data are required to be serialized and transferred between the processes, and then the data are subjected to deserialization reduction to ensure the integrity of the data. Because the serialization and deserialization operations are not needed, the calculation resources are saved during the cross-process calculation, and the calculation efficiency is improved.
Further, based on the first embodiment, a second embodiment of the present invention provides a column-oriented caching method. In this embodiment, after the step S30, the method further includes:
step A10, after performing determinant caching on the data to be cached to obtain columnar cache data, designating a target virtual row or a target column in the columnar cache data for a parallel process or a parallel thread;
and A20, calling the parallel process or the parallel thread to determine a virtual memory block corresponding to the target virtual row or the target column according to the metadata, and carrying out parallel processing on data in the virtual memory block.
The cache system performs determinant caching on the data to be cached according to the caching mode in the embodiment, and after obtaining the columnar cache data, various operations such as reading, writing or calculating can be performed on the columnar cache data.
The caching system may specify a range of column indexes, or an index set, creating a column-level view of the columnar cache data; virtual rows may also be supported on the columnar cache data, specifying a range of row indices or index set, creating a row-level view of the columnar cache data. Combining the two modes, the view split by rows and columns can be flexibly combined, and when the row or column index is used for accessing the column cache data, the real index is mapped by creating parameters of the view.
When there is a parallel computing task, the cache system may specify a target virtual row or target column in the columnar cache data for the parallel process or parallel thread. And calling a parallel process or a parallel thread to perform parallel computation on the data of the target virtual row or the target column. Specifically, the parallel process or the parallel thread determines a virtual memory block corresponding to a target virtual row or a target column according to the corresponding relation between the columnar format information in the metadata and the virtual memory, and then carries out parallel processing on data in the virtual memory block. Because the process or the thread can directly determine the memory block where the target virtual line or the target column is located through the mapping information in the metadata, the process or the thread can directly read the data in the memory block, and when writing, the non-overlapping virtual line or column can be designated for the parallel process or the parallel thread through the cache system, so that the parallel process or the thread can simultaneously write the column type cache data in the memory. Therefore, the problem that the existing cache system cannot support multi-process or multi-thread parallel computing is solved, and the computing capability of the chip is fully exerted.
Further, based on the first and second embodiments, a third embodiment of the present invention provides a column-oriented caching method. In this embodiment, after the step S30, the method further includes:
Step B10, after performing determinant caching on the data to be cached to obtain columnar cache data, determining a target memory block corresponding to the data to be written in the columnar cache data according to the metadata;
step B20, copying the content in the target memory block to the copied memory block;
and step B30, updating the metadata, wherein the data to be written in the updated metadata is correspondingly associated with the copied memory block.
In this embodiment, after the cache system performs determinant caching on the data to be cached based on the caching method in the above embodiment to obtain columnar cache data, based on the characteristics of the columnar cache data, multiple concurrent read-write or calculation strategies can be supported, and the processing efficiency can be improved, where the concurrent strategies include, but are not limited to, copy-on-write, concurrent write, fine-granularity lock, and instruction optimization.
Specifically, copy-on-write refers to that when a new process is created, not all resources are copied to the newly created process, but when data needs to be written, pages where the data needs to be written are copied, so that resource waste caused by a copy operation is avoided.
When the cache system reads data, the data in the memory block is not required to be copied, and the data is directly mapped into the memory block according to the metadata and is read from the memory block. When writing, the cache system can determine a target memory block corresponding to the data to be written in the column cache data according to the mapping relation in the metadata. For example, if the data to be written is column 1 in the column cache data, the cache system searches the metadata and determines the memory block corresponding to column 1. And then the cache system copies the content in the target memory block into a copy memory block, wherein the copy memory block is a memory block reassigned in the memory resource, and the size of the copy memory block is the same as that of the target memory block. After the cache system copies the content of the target memory block, updating metadata, and updating the corresponding relation between the data to be written in the original metadata and the target memory block into the corresponding relation between the data to be written and the copied memory block. The copy memory block corresponding to the data to be written can be directly indexed according to the metadata, and then the data in the copy memory block is directly written.
Further, after step S30, the method further includes:
step C10, after the data to be cached is subjected to determinant caching to obtain columnar cache data, determining a target memory block corresponding to the data to be modified in the columnar cache data according to the metadata;
and step C20, transmitting the data to be modified and the modified value into a concurrent written assembly instruction, and executing the concurrent written assembly instruction to update the data to be modified in the target memory block into the modified value.
When writing concurrently, the cache system can determine the target memory block corresponding to the data to be modified in the column cache data according to the mapping relation in the metadata. The cache system reads the data to be modified from the target memory block, transmits the data to be modified and the modified value into an assembly instruction which is written in concurrently, and executes the assembly instruction which is written in concurrently, so as to update the data to be modified in the target memory block into the modified value. It should be noted that, the concurrently written assembler instruction may be a cmpxchg instruction, which is an assembler instruction for solving the issue of concurrent writing. The cmpxchg instruction supports vectorized updating of multiple values, and the cmpxchg will sequentially write one or a batch of values (if supported by a chip) into memory, and then compile the values into a chip-level instruction, which is far higher than using thread or process lock performance.
Further, after step S30, the method further includes:
step D10, after the data to be cached is subjected to determinant caching to obtain columnar cache data, determining a target virtual block from the columnar cache data;
step D20, detecting whether the metadata has the locked virtual block overlapped with the target virtual block;
step D30, if not, adding the virtual row information or the column information corresponding to the target virtual block into the metadata to lock the target virtual block;
and D40, unlocking the target virtual block after the writing of the target virtual block is completed.
In this embodiment, the cache system may lock a virtual block for a process that concurrently operates the columnar cache data, the virtual block being part of the columnar cache data. Specifically, the cache system first determines a virtual row or column to be operated by the process, and a target virtual block is formed by the determined virtual row or column. The information of the locked virtual block is recorded in the metadata, and the cache system detects whether the locked virtual block overlapped with the target virtual block exists in the metadata. And if the virtual rows or columns of the target virtual block and the locked virtual block are overlapped, determining that the target virtual block and the locked virtual block are overlapped. If a locked virtual block overlapping with the target virtual block is not detected, the cache system may add virtual row information or column information corresponding to the target virtual block to the metadata to lock the target virtual block. When the process completes writing to the target virtual block, the locking of the target virtual block is released, and the information of the target virtual block in the metadata can be deleted. If the cache system detects the locked virtual block overlapped with the target virtual block, waiting until the locked virtual block which is not overlapped with the target virtual block in the metadata is not obtained, and locking the target virtual block; or directly returns failure information to the process.
In this embodiment, because the corresponding relationship between the columnar format information of the columnar cache data and the virtual memory is recorded in the metadata, and the mapping relationship between the virtual memory and the memory block is also recorded, the position of the data cached in the actual memory can be determined according to the metadata, so that the data in the memory can be directly read and written, and a fine-granularity lock can be added to the columnar cache data according to the columnar format information of the fine granularity recorded in the metadata, so that the read and write efficiency of the concurrent process is improved.
Further, after the step S30, the method further includes:
step E10, after the data to be cached is subjected to determinant caching to obtain columnar cache data, determining a target memory block corresponding to the data to be processed in the columnar cache data according to the metadata;
and E20, calling an instruction in the single-instruction multi-data-stream instruction set to process the target memory block.
Further, in this embodiment, the cache system may determine, according to the mapping relationship of the metadata, a target memory block corresponding to the data to be processed in the column cache data. And invoking a single instruction multiple data stream (Single Instruction Multiple Data, SIMD) instruction set to process the data to be processed in the target memory block according to the operation task. SIMD is a technique in which one controller is used to control a plurality of processors while performing the same operation on each of a set of data (also called "data vectors") separately, thereby achieving spatial parallelism. Because the column cache data is the data cached by column, a SIMD instruction set can be adopted to process a row of data, and the processing efficiency is improved.
The embodiment generally covers the requirements of different chips such as GPU/NPU, and the like, and the continuous small memory which can be randomly scheduled and mapped is beneficial to localization, instruction pipeline or register optimization.
In addition, an embodiment of the present invention further provides a column-oriented caching apparatus, referring to fig. 4, where the column-oriented caching apparatus includes:
an obtaining module 10, configured to obtain column format information of data to be cached;
the allocation module 20 is configured to allocate a memory block to the data to be cached in a memory resource according to the column format information;
the generating module 30 is configured to add the memory block to a continuous virtual memory, and generate metadata of the data to be cached, where the metadata includes a correspondence between the columnar format information and the virtual memory, and a mapping relationship between the virtual memory and the memory block.
Further, the column-oriented caching apparatus further includes:
the designating module is used for designating a target virtual row or a target column in the column cache data for a parallel process or a parallel thread after performing determinant cache on the data to be cached to obtain column cache data;
and the parallel processing module is used for calling the parallel process or the parallel thread to determine a virtual memory block corresponding to the target virtual row or the target column according to the metadata, and carrying out parallel processing on the data in the virtual memory block.
Further, the column-oriented caching apparatus further includes:
the first determining module is used for determining a target memory block corresponding to data to be written in the column-type cache data according to the metadata after the column-type cache is carried out on the data to be cached to obtain the column-type cache data;
the copying module is used for copying the content in the target memory block into the copied memory block;
and the updating module is used for updating the metadata, wherein the data to be written in the updated metadata is correspondingly associated with the copy memory block.
Further, after the step of adding the memory block to the continuous virtual memory and generating the metadata of the data to be cached, the method further includes:
the second determining module is used for determining a target memory block corresponding to the data to be modified in the column-type cache data according to the metadata after the column-type cache is carried out on the data to be cached to obtain the column-type cache data;
and the execution module is used for transmitting the data to be modified and the modified value into a concurrent written assembly instruction, and executing the concurrent written assembly instruction to update the data to be modified in the target memory block into the modified value.
Further, after the step of adding the memory block to the continuous virtual memory and generating the metadata of the data to be cached, the method further includes:
the third determining module is used for determining a target virtual block from the column cache data after column cache data is obtained by performing column cache on the data to be cached;
a detection module, configured to detect whether there is a locked virtual block overlapping the target virtual block in the metadata;
the locking module is used for adding the virtual row information or the column information corresponding to the target virtual block into the metadata to lock the target virtual block if the virtual row information or the column information does not exist;
and the releasing module is used for releasing the locking of the target virtual block after the writing of the target virtual block is completed.
Further, after the step of adding the memory block to the continuous virtual memory and generating the metadata of the data to be cached, the method further includes:
a fourth determining module, configured to determine, according to the metadata, a target memory block corresponding to data to be processed in the columnar cache data after performing determinant caching on the data to be cached to obtain columnar cache data;
And the calling module is used for calling the instructions in the single-instruction multi-data-stream instruction set to process the target memory block.
Further, the step S20 includes:
a first allocation unit, configured to calculate, when a column in the column format information is a column with a fixed element size, an actual size of the column according to the number of elements in the column and a data type, and allocate, in a memory resource, a memory block of the actual size for the column;
and the second allocation unit is used for allocating memory blocks with default sizes for the columns in the memory resource when the columns in the column format information are columns with non-fixed element sizes, and allocating the memory blocks with the default sizes for the columns when the default sizes are smaller than the actual sizes of the columns.
The expansion content of the specific implementation manner of the column-oriented caching device of the present invention is basically the same as that of each embodiment of the above-mentioned column-oriented caching method, and will not be described herein.
In addition, the embodiment of the invention also provides a computer readable storage medium, wherein the computer readable storage medium stores a cache program facing the array, and the cache program facing the array realizes the steps of the cache method facing the array when being executed by a processor.
The expansion content of the specific embodiments of the present invention for the column-oriented caching device and the computer-readable storage medium is substantially the same as the embodiments of the above-described column-oriented caching method, and will not be described herein.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or system. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or system that comprises the element.
The foregoing embodiment numbers of the present invention are merely for the purpose of description, and do not represent the advantages or disadvantages of the embodiments.
From the above description of the embodiments, it will be clear to those skilled in the art that the above-described embodiment method may be implemented by means of software plus a necessary general hardware platform, but of course may also be implemented by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a storage medium (e.g. ROM/RAM, magnetic disk, optical disk) as described above, comprising instructions for causing a terminal device (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to perform the method according to the embodiments of the present invention.
The foregoing description is only of the preferred embodiments of the present invention, and is not intended to limit the scope of the invention, but rather is intended to cover any equivalents of the structures or equivalent processes disclosed herein or in the alternative, which may be employed directly or indirectly in other related arts.
Claims (8)
1. The column-oriented caching method is characterized by comprising the following steps of:
obtaining column format information of data to be cached, wherein the column format information is used for describing information of the data to be cached organized in a column format, and the column format information comprises the number of columns, the index of each column, nesting information of columns and sub-columns, the number of cells of each column, the data type of each column, the number of storage batches or columns contained in each storage batch;
distributing memory blocks for the data to be cached in memory resources according to the column format information;
adding the memory blocks into a continuous virtual memory and generating metadata of the data to be cached, wherein the metadata comprises a corresponding relation between the column format information and the virtual memory and a mapping relation between the virtual memory and the memory blocks, and the virtual memory is a section of continuous memory formed by virtual addresses;
After the step of adding the memory block to the continuous virtual memory and generating the metadata of the data to be cached, the method further includes:
after the data to be cached is subjected to determinant caching to obtain columnar cache data, designating a target virtual row or a target column in the columnar cache data for a parallel process or a parallel thread;
invoking the parallel process or the parallel thread to determine a virtual memory block corresponding to the target virtual row or the target column according to the metadata, and carrying out parallel processing on data in the virtual memory block;
the step of allocating memory blocks for the data to be cached in the memory resource according to the column format information comprises the following steps:
when the column in the column format information is a column with fixed element size, calculating the actual size of the column according to the number of elements in the column and the data type, and distributing the memory blocks with the actual size for the column in a memory resource;
when the column in the column format information is a column with non-fixed element size, memory blocks with default size are allocated to the column in the memory resource, and when the default size is smaller than the actual size of the column, the memory blocks with default size are allocated to the column.
2. The column-oriented caching method as claimed in claim 1, wherein after the step of adding the memory blocks to a continuous virtual memory and generating metadata of the data to be cached, further comprises:
after the data to be cached is subjected to determinant caching to obtain columnar cache data, determining a target memory block corresponding to data to be written in the columnar cache data according to the metadata;
copying the content in the target memory block into a copied memory block;
and updating the metadata, wherein the data to be written in the updated metadata is correspondingly associated with the copy memory block.
3. The column-oriented caching method as claimed in claim 1, wherein after the step of adding the memory blocks to a continuous virtual memory and generating metadata of the data to be cached, further comprises:
after the data to be cached is subjected to determinant caching to obtain columnar cache data, determining a target memory block corresponding to the data to be modified in the columnar cache data according to the metadata;
and transmitting the data to be modified and the modified value into a concurrent written assembly instruction, and executing the concurrent written assembly instruction to update the data to be modified in the target memory block into the modified value.
4. The column-oriented caching method as claimed in claim 1, wherein after the step of adding the memory blocks to a continuous virtual memory and generating metadata of the data to be cached, further comprises:
after the data to be cached is subjected to determinant caching to obtain columnar cache data, determining a target virtual block from the columnar cache data;
detecting whether the metadata has a locked virtual block overlapped with the target virtual block or not;
if not, adding virtual row information or column information corresponding to the target virtual block into the metadata to lock the target virtual block;
and unlocking the target virtual block after the writing of the target virtual block is completed.
5. The column-oriented caching method as claimed in claim 1, wherein after the step of adding the memory blocks to a continuous virtual memory and generating metadata of the data to be cached, further comprises:
after the data to be cached is subjected to determinant caching to obtain columnar cache data, determining a target memory block corresponding to the data to be processed in the columnar cache data according to the metadata;
And calling an instruction in the single-instruction multi-data stream instruction set to process the target memory block.
6. A column-oriented caching apparatus, the column-oriented caching apparatus comprising:
the storage device comprises an acquisition module, a storage batch storage module and a storage batch storage module, wherein the acquisition module is used for acquiring column format information of data to be cached, wherein the column format information is used for describing information of the data to be cached organized in a column format, and the column format information comprises the number of columns, indexes of each column, nesting information of columns and sub-columns, the number of cells of each column, the data type of each column, the number of storage batches or columns contained in each storage batch;
the distribution module is used for distributing memory blocks for the data to be cached in the memory resources according to the column format information;
the generation module is used for adding the memory blocks into a continuous virtual memory and generating metadata of the data to be cached, wherein the metadata comprises a corresponding relation between the column format information and the virtual memory and a mapping relation between the virtual memory and the memory blocks, and the virtual memory is a section of continuous memory formed by virtual addresses;
after the step of adding the memory block to the continuous virtual memory and generating the metadata of the data to be cached, the column-oriented caching device further includes:
After the data to be cached is subjected to determinant caching to obtain columnar cache data, designating a target virtual row or a target column in the columnar cache data for a parallel process or a parallel thread;
invoking the parallel process or the parallel thread to determine a virtual memory block corresponding to the target virtual row or the target column according to the metadata, and carrying out parallel processing on data in the virtual memory block;
the distribution module comprises:
a first allocation unit, configured to calculate, when a column in the column format information is a column with a fixed element size, an actual size of the column according to the number of elements in the column and a data type, and allocate, in a memory resource, a memory block of the actual size for the column;
and the second allocation unit is used for allocating memory blocks with default sizes for the columns in the memory resource when the columns in the column format information are columns with non-fixed element sizes, and allocating the memory blocks with the default sizes for the columns when the default sizes are smaller than the actual sizes of the columns.
7. A rank-oriented caching device, characterized in that it comprises a memory, a processor and a rank-oriented caching program stored on the memory and executable on the processor, which when executed by the processor implements the steps of the rank-oriented caching method according to any one of claims 1 to 5.
8. A computer readable storage medium, characterized in that it has stored thereon a cache program oriented in columns, which when executed by a processor implements the steps of the cache method oriented in columns according to any of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911326754.5A CN111104067B (en) | 2019-12-20 | 2019-12-20 | Cache method, device, equipment and computer readable storage medium for alignment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911326754.5A CN111104067B (en) | 2019-12-20 | 2019-12-20 | Cache method, device, equipment and computer readable storage medium for alignment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111104067A CN111104067A (en) | 2020-05-05 |
| CN111104067B true CN111104067B (en) | 2024-01-12 |
Family
ID=70422737
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911326754.5A Active CN111104067B (en) | 2019-12-20 | 2019-12-20 | Cache method, device, equipment and computer readable storage medium for alignment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111104067B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113296961B (en) * | 2021-07-22 | 2021-11-19 | 广州中望龙腾软件股份有限公司 | GPU-based dynamic memory allocation method and device and memory linked list |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102609305A (en) * | 2012-02-07 | 2012-07-25 | 中山爱科数字科技股份有限公司 | Method for sharing internal memory in server cluster |
| CN103092905A (en) * | 2011-11-07 | 2013-05-08 | Sap股份公司 | Columnar database using virtual file data objects |
| CN104516912A (en) * | 2013-09-29 | 2015-04-15 | 中国移动通信集团黑龙江有限公司 | Dynamic data storage method and device |
| CN104765574A (en) * | 2015-04-23 | 2015-07-08 | 成都博元时代软件有限公司 | Data cloud storage method |
| CN105144160A (en) * | 2013-03-15 | 2015-12-09 | 甲骨文国际公司 | A method to accelerate queries using dynamically generated alternate data formats in flash cache |
| CN107408132A (en) * | 2015-03-20 | 2017-11-28 | 国际商业机器公司 | The effective performance of insertion and point inquiry operation in row store |
| CN109542889A (en) * | 2018-10-11 | 2019-03-29 | 平安科技（深圳）有限公司 | Stream data column storage method, device, equipment and storage medium |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10733688B2 (en) * | 2017-09-26 | 2020-08-04 | Intel Corpoation | Area-efficient implementations of graphics instructions |
-
2019
- 2019-12-20 CN CN201911326754.5A patent/CN111104067B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103092905A (en) * | 2011-11-07 | 2013-05-08 | Sap股份公司 | Columnar database using virtual file data objects |
| CN102609305A (en) * | 2012-02-07 | 2012-07-25 | 中山爱科数字科技股份有限公司 | Method for sharing internal memory in server cluster |
| CN105144160A (en) * | 2013-03-15 | 2015-12-09 | 甲骨文国际公司 | A method to accelerate queries using dynamically generated alternate data formats in flash cache |
| CN104516912A (en) * | 2013-09-29 | 2015-04-15 | 中国移动通信集团黑龙江有限公司 | Dynamic data storage method and device |
| CN107408132A (en) * | 2015-03-20 | 2017-11-28 | 国际商业机器公司 | The effective performance of insertion and point inquiry operation in row store |
| CN104765574A (en) * | 2015-04-23 | 2015-07-08 | 成都博元时代软件有限公司 | Data cloud storage method |
| CN109542889A (en) * | 2018-10-11 | 2019-03-29 | 平安科技（深圳）有限公司 | Stream data column storage method, device, equipment and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 刘庆，韩蓓蓓.数据库系统概论.电子科技大学出版社,2015,第185-201页. * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111104067A (en) | 2020-05-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP5147280B2 (en) | System and method for garbage collection in heterogeneous multiprocessor systems | |
| US4812981A (en) | Memory management system improving the efficiency of fork operations | |
| US11106795B2 (en) | Method and apparatus for updating shared data in a multi-core processor environment | |
| US9513886B2 (en) | Heap data management for limited local memory(LLM) multi-core processors | |
| RU2641244C2 (en) | Unified access to jointly used and controlled memory | |
| US8839215B2 (en) | String cache file for optimizing memory usage in a java virtual machine | |
| US8650537B2 (en) | Optimizing an object-oriented program by transforming invocations of synthetic accessor methods | |
| US9304898B2 (en) | Hardware-based array compression | |
| US11556468B2 (en) | Multi-ring shared, traversable, and dynamic advanced database | |
| KR20200060245A (en) | Data accessing method, apparatus, device, and storage medium | |
| CN111104067B (en) | Cache method, device, equipment and computer readable storage medium for alignment | |
| US20140379739A1 (en) | System and method for marshaling massive database data from native layer to java using linear array | |
| US8656120B2 (en) | Device, method and computer-readable medium relocating remote procedure call data in heterogeneous multiprocessor system on chip | |
| US10558502B2 (en) | Software application runtime having dynamic evaluation functions and parameters | |
| CN104714792A (en) | Multi-process shared data processing method and device | |
| JP2019523480A (en) | Preemptive decompression scheduling for NAND storage devices | |
| US20100268921A1 (en) | Data collection prefetch device and methods thereof | |
| CN113243008A (en) | Distributed VFS with shared page cache | |
| US20100299672A1 (en) | Memory management device, computer system, and memory management method | |
| CN113961485A (en) | Memory optimization method, device, terminal and storage medium | |
| US20150278089A1 (en) | Execution control method and execution control apparatus | |
| US7757053B2 (en) | Apparatus and method for managing stacks for efficient memory usage | |
| US20130166887A1 (en) | Data processing apparatus and data processing method | |
| JP5048072B2 (en) | Information search system, information search method and program | |
| CN113986555B (en) | Cache optimization method, device, equipment and readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |