CN113177045A

CN113177045A - Data extraction method and device, computable storage equipment and data request equipment

Info

Publication number: CN113177045A
Application number: CN202110389430.7A
Authority: CN
Inventors: 郑宁; 许树堃
Original assignee: Ruizhe Hangzhou Technology Co ltd
Current assignee: Ruizhe Hangzhou Technology Co ltd
Priority date: 2021-04-12
Filing date: 2021-04-12
Publication date: 2021-07-27

Abstract

The embodiment of the invention provides a data extraction method, a data extraction device, a computable storage device and a data request device, wherein the method comprises the following steps: receiving data request information sent by a data request end, wherein the data request information comprises storage address information and data selection conditions of a target data block; acquiring a target data block from all data blocks according to the storage address information, wherein the target data block comprises at least one data group, index information of each data group and data block meta information; determining a data group contained in the data block according to the data block meta information, and extracting target data according to the data selection condition and index information of the data group; the target data is data meeting data selection conditions; and the target data is sent to the data request terminal, so that the data which meets the data selection condition in the data block is screened out by the computable storage device, the computable function of the storage device is fully exerted, and the transmitted data amount is reduced.

Description

Data extraction method and device, computable storage equipment and data request equipment

Technical Field

The embodiment of the invention relates to the technical field of computer data processing, in particular to a data extraction method, a data extraction device, a computable storage device and a data request device.

Background

In the field of data analysis, the storage of original data is generally performed in a column storage mode, and a column storage mode rather than a row storage mode is adopted, so that the reading of target column data is facilitated, and IO (input/output) operation is reduced; and secondly, the data compression is facilitated, and the occupation of the storage space is reduced.

In order to further improve the data reading efficiency, indexes of column data are added in many databases, and before data are written into the storage device in columns, statistical information corresponding to data blocks of each column is counted, so that unnecessary data reading operation can be filtered out in a data request end as much as possible, and the data reading process is accelerated. However, existing data indexes are based on column-stored data blocks, which are often large, typically tens or hundreds of KB. This results in the storage device still sending the complete block of columnar data to the data requestor when the data requestor needs only a small amount of data in the block of columnar data. Even if the size of the columnar data block is reduced (less than 4KB), the storage device will return at least 4KB of data, limited by the size of its physical data blocks (i.e., sectors, typically 4 KB).

However, in the process of implementing the present invention, the inventor finds that the calculable function of the storage device is not fully utilized in the above method, and the data transmitted between the storage device and the data request end is large.

Disclosure of Invention

The embodiment of the invention provides a data extraction method, a data extraction device, a calculable storage device and a data request device, which can fully play the calculable function of the storage device and reduce the data volume transmitted between the storage device and a data request terminal.

In a first aspect, an embodiment of the present invention provides a data extraction method applied to a computable storage device, including:

receiving data request information sent by a data request end, wherein the data request information comprises storage address information and data selection conditions of a target data block;

acquiring a target data block from all data blocks according to the storage address information, wherein the target data block comprises at least one data group, index information of each data group and data block meta information;

determining a data group contained in the data block according to the data block meta information, and extracting target data according to the data selection condition and index information of the data group; the target data is data meeting data selection conditions;

and sending the target data to the data request terminal.

Optionally, extracting the target data according to the data selection condition and the index information of the data group includes:

judging whether the data group is overlapped with the data interval determined by the data selection condition according to the index information of the data group, and if so, determining that the data group is a target data group;

and determining target data according to the target data group.

Optionally, judging whether the data group overlaps with the data interval determined by the data selection condition according to the index information of the data group includes:

when the index information comprises maximum value index information and minimum value index information, if the maximum value index information and/or the minimum value index information of the data group is/are overlapped with the data interval determined by the data selection condition, determining that the data group is overlapped with the data interval determined by the data selection condition;

or when the index information includes bit index information, if a bucket interval corresponding to a preset numerical value in the bit information of the data group overlaps with a data interval determined by the data selection condition, determining that the data group overlaps with the data interval determined by the data selection condition; wherein the bucket interval is determined according to the number of bucket intervals in the data block meta-information and a bucket boundary value.

Optionally, the data request information further includes: extracting granularity level information; determining target data from the target data set, comprising:

when the extracted granularity level information is a data group level, determining the target data group as target data, and storing the target data to a cache position;

or, when the extracted granularity level information is an element level, determining a data element meeting a data selection condition in the target data group as target data, and storing the target data in a cache position.

Optionally, the data block meta information includes start offset information of each data group; the determining a data group contained in the data block according to the data block meta information includes:

determining the position information of each data group according to the initial offset information of each data group;

and acquiring each data group according to the position information of each data group.

Optionally, extracting the target data according to the data selection condition and the index information of the data group, further comprising:

judging whether alarm information appears when target data is extracted according to the data selection condition and the index information of the data group;

and when the alarm information appears, returning an error code corresponding to the alarm information so that the data request terminal executes data extraction operation corresponding to the error code.

Optionally, sending the target data to the data request end includes:

and packaging the target data according to a preset rule, and sending the packaged data to the data request terminal so that the data request terminal analyzes the data target data according to the packaged data.

Optionally, encapsulating the target data according to a preset rule, including:

when the target data is a data group, packaging data length information, the data group, data group offset information, data group number information and check code information;

and when the target data is a data element, packaging the data length information, the data element number information and the check code information.

Optionally, before receiving the data request information sent by the data request end, the method further includes:

receiving a data block and storage address information sent by a data request end, wherein the data block comprises at least one data group, index information of each data group and data block meta information;

and writing the data block into a storage medium according to the storage address information.

In a second aspect, an embodiment of the present invention provides a data extraction method, which is applied to a data request device, and includes:

sending data request information to a computable storage device, so that the computable storage device acquires a target data block from all data blocks according to storage address information of the target data block in the data request information, determines a data group contained in the data block according to data block meta information, and extracts target data according to the data selection condition and index information of the data group; the target data is data meeting data selection conditions; the data request information comprises storage address information and data selection conditions of the target data block; the target data block comprises at least one data group, index information of each data group and data block meta information;

receiving the target data sent by the computable storage device.

Optionally, before sending the data request information to the computable storage device, the method further includes:

and generating and sending data blocks and storage address information to the computable storage device so that the computable storage device writes the data blocks into a storage medium according to the storage address information, wherein the data blocks comprise at least one data group, index information of each data group and data block meta information.

In a third aspect, an embodiment of the present invention provides a data extraction apparatus, including:

the receiving module is used for receiving data request information sent by a data request end, wherein the data request information comprises storage address information and data selection conditions of a target data block;

the acquisition module is used for acquiring a target data block from all data blocks according to the storage address information, wherein the target data block comprises at least one data group, index information of each data group and data block meta information;

the extraction module is used for determining a data group contained in the data block according to the data block meta information and extracting target data according to the data selection condition and the index information of the data group; the target data is data meeting data selection conditions;

and the sending module is used for sending the target data to the data request terminal.

In a fourth aspect, an embodiment of the present invention provides a data extraction apparatus, including:

the sending module is used for sending data request information to the calculable storage equipment so that the calculable storage equipment can acquire target data blocks from all data blocks according to storage address information of the target data blocks in the data request information, determine data groups contained in the data blocks according to data block meta information and extract target data according to the data selection conditions and index information of the data groups; the target data is data meeting data selection conditions; the data request information comprises storage address information and data selection conditions of the target data block; the target data block comprises at least one data group, index information of each data group and data block meta information;

and the receiving module is used for receiving the target data sent by the computable storage device.

In a fifth aspect, an embodiment of the present invention provides a computable storage device, including: at least one processor and memory;

the memory stores computer-executable instructions;

the at least one processor executing the computer-executable instructions stored by the memory causes the at least one processor to perform the data extraction method of any one of the first aspects.

In a sixth aspect, an embodiment of the present invention provides a data requesting device, including: at least one processor and memory;

the memory stores computer-executable instructions;

the at least one processor executing the computer-executable instructions stored by the memory causes the at least one processor to perform the data extraction method of any one of the second aspects.

In a seventh aspect, an embodiment of the present invention provides a computer-readable storage medium, where computer-executable instructions are stored, and when a processor executes the computer-executable instructions, the data extraction method according to any one of the first aspect and the second aspect is implemented.

In an eighth aspect, an embodiment of the present invention provides a program product, which includes a computer program that, when executed by a processor, implements the data extraction method according to any one of the first and second aspects.

The data extraction method and the device provided by the embodiment of the invention receive data request information sent by a data request end, wherein the data request information comprises storage address information and data selection conditions of a target data block; acquiring a target data block from all data blocks according to the storage address information, wherein the target data block comprises at least one data group, index information of each data group and data block meta information; determining a data group contained in the data block according to the data block meta information, and extracting target data according to the data selection condition and index information of the data group; the target data is data meeting data selection conditions; sending the target data to the data request terminal; the data which meets the data selection condition in the data block are screened out by the calculable storage device, the calculable function of the storage device can be fully exerted, the transmitted data volume is reduced, and further the occupancy rate of the bus bandwidth during data transmission and the occupancy rate of the memory bandwidth of the data request end are reduced; in addition, the processing procedure of the data request end to the data can be reduced.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.

Fig. 1 is a schematic view of an application scenario of data extraction according to an embodiment of the present invention;

fig. 2 is a schematic flow chart of a data extraction method according to an embodiment of the present invention;

FIG. 3 is a diagram illustrating a data block format according to an embodiment of the present invention;

fig. 4 is a schematic diagram of data extraction performed by using maximum value index information and minimum value index information according to an embodiment of the present invention;

fig. 5 is a schematic diagram of data extraction using bit indexes according to an embodiment of the present invention;

FIG. 6 is a diagram illustrating a package format of a data set according to an embodiment of the present invention;

FIG. 7 is a diagram illustrating a packaging format of a data element according to an embodiment of the present invention;

FIG. 8 is a schematic flow chart of another data extraction method according to an embodiment of the present invention;

fig. 9 is a schematic structural diagram of a data extraction apparatus according to an embodiment of the present invention;

FIG. 10 is a schematic structural diagram of another data extraction apparatus according to an embodiment of the present invention;

FIG. 11 is a diagram illustrating a hardware configuration of a computing storage device according to an embodiment of the present invention;

fig. 12 is a schematic diagram of a hardware structure of a data request device according to an embodiment of the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

The terms "first," "second," "third," "fourth," and the like in the description and in the claims, as well as in the drawings, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are, for example, capable of operation in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

Fig. 1 is a schematic view of an application scenario of data extraction according to an embodiment of the present invention. As shown in fig. 1, the data extraction process involves devices including: the data requestor 101 is coupled to the computing storage device 102. The data request end 101 may be any device that needs to obtain target data from the computing storage device 102, such as a host of a computer. The computing storage device 102 is a device having data storage and computing functions, such as a hard disk or a flash memory.

Wherein the data is stored in a storage medium of the computable storage device in a columnar manner. The data extraction process mainly comprises the following steps: the data request terminal 101 generates data request information, sends the data request information to the computable storage device 102, and the computable storage device 102 may obtain target data according to the data request information and send the screened target data to the data request terminal 101.

In some technologies, the minimum unit of data stored in the storage device may be calculated as a data block, when a data request end requests data, the storage device may be calculated to send a data block meeting a certain condition to the data request end, and the data request end then screens the acquired data block for the required data. However, a data block stores more data, the size of a data block is generally about 4KB, and the data requesting end may only need a part of the data in a certain data block. That is, the current computing storage device does not fully perform its computing function, or sends a large amount of unnecessary data to the data request side, which causes a problem that the amount of data transmitted between the computing storage device and the data request side is large.

With the development of the computing function of the computing storage device, in the data extraction process, if the computing storage device can replace a data request terminal to perform a part of data screening work, the data in one data block is screened to obtain data meeting the conditions, and the data meeting the conditions is sent to the data request terminal, the data amount transmitted between the computing storage device and the data request terminal can be greatly reduced.

According to the data extraction method provided by the embodiment of the invention, the data selection condition is set in the data request information, the data blocks are set into the plurality of data groups in advance, and the index information and the data block meta information of each data group, so that after the computable storage device acquires the data request information, the data meeting the data selection condition can be obtained according to the data selection condition, the index information and the data block meta information of the data groups, and the data meeting the data selection condition is sent to the data request terminal, further data extraction of data at the data block level is realized through the computable storage device, the transmitted data is converted into the dimensionality of the current data group or data element from the data block dimensionality, and the transmitted data amount is reduced.

The technical solution of the present invention will be described in detail below with specific examples. The following several specific embodiments may be combined with each other, and details of the same or similar concepts or processes may not be repeated in some embodiments.

Fig. 2 is a schematic flowchart of a data extraction method according to an embodiment of the present invention, where the method according to this embodiment may be executed by a data extraction apparatus, and the data extraction apparatus may be arranged in the computable storage device of fig. 1 in the form of software and/or hardware. As shown in fig. 2, the method of this embodiment may include:

s201, receiving data request information sent by a data request end, wherein the data request information comprises storage address information and data selection conditions of a target data block.

In this embodiment, the data request side needs to obtain the required data from the computable storage device when calling the data. The data request information comprises storage address information of the target data block, and the storage position of the target data block can be determined through the storage address information of the target data block. Wherein the storage address information includes: the logical block address of the target data block, and the offset and length of the target data block within the logical segment address.

The data is stored in the computable storage device in columns, when a column of data is large, the column of data can be divided into a plurality of data blocks for storage, one data request corresponds to one or a plurality of target data blocks, when extraction of the column of data is to be completed, the extraction can be completed through one or a plurality of interactions, and each time the extraction of the column of data is completed, the extraction of the column of data is completed corresponding to different target data blocks.

S202, acquiring a target data block from all data blocks according to the storage address information, wherein the target data block comprises at least one data group, index information of each data group and data block meta information.

Fig. 3 is a schematic diagram of a data block format according to an embodiment of the present invention, and as shown in fig. 3, a data block includes a data group 1, a data group 2, and up to a data group N, where index information corresponding to the data group is stored behind each data group, and the index information is index information 1, index information 2, and up to index information N, respectively; the data block further includes data block meta-information, and the storage location of the data block meta-information is not limited, and may be at the initial location of the data block, or may be at the end location of the data block. The figure shows the situation where the data block meta-information is at the end of the data block.

The data block meta-information includes the maximum value and the minimum value of the data block, so that the data block satisfying the condition can be preliminarily screened out according to the data selection condition and the data block meta-information, and the data block satisfying the condition is the target data block. For example, data block 1 and data block 2 may be preliminarily screened out as eligible data according to the data block meta information.

After the target data block is obtained, the target data block can be determined according to the storage address information of the target data block, specifically, the physical address of the target data block can be obtained according to the logical block address of the target data block, and the target data block can be accurately obtained according to the offset and the length of the target data block in the logical segment address. For example, there are three data blocks stored in the storage medium with physical addresses 0, 1, 2, 3, etc., each in 4 k. When the second data block is to be read, the physical addresses of the second data block are 1 and 2 according to the logical address of the second data block (where there is a mapping relationship between the logical address and the physical address). When the offset of the second data block is 2k and the length is 6k, the data block can be accurately read, that is, the data of the second storage unit and the data of the third storage unit are read, and the data of the second half of the data of the second storage unit and the data of the third storage unit are intercepted to be used as a target data block.

S203, determining a data group contained in the data block according to the data block meta information, and extracting target data according to the data selection condition and the index information of the data group; the target data is data satisfying a data selection condition.

In this embodiment, after the target data block is obtained, a plurality of data groups may be obtained from the target data block, and target data satisfying the data selection condition may be obtained according to the data selection condition and the index information of the data group, where the target data may be a data group or a data element. Wherein a data group comprises a plurality of data elements. The data selection condition is not limited, and may be data in a certain data interval. The index information of the data group is fine-grained index information, and whether the data group contains data meeting the data selection condition can be determined according to the index information.

And S204, sending the target data to the data request terminal.

In this embodiment, after the target data is acquired, the target data needs to be sent to the data request end, where the acquired data request information further includes a memory address for storing the returned target data. After the target data is obtained by the computable storage device, the target data can be sent to the data request terminal and stored in the memory address.

In addition, the obtained data request information also includes a data type, and the data request end can also determine the size of the target data according to the data type of the target data, the logical block address in the computable storage device, and the offset and the length of the target data block in the logical segment address, and applies for a storage space with a proper size in advance according to the size of the target data, so as to efficiently store the received target data.

According to the data selection method and device, the data selection conditions are set in the data request information, the data blocks are set into the multiple data groups in advance, the index information and the data block meta information of each data group, after the data request information is obtained by the computable storage device, the data meeting the data selection conditions can be obtained according to the data selection conditions, the index information and the data block meta information of the data groups, the data meeting the data selection conditions are sent to the data request end, further data extraction of the data at the data block level is achieved through the computable storage device, the transmitted data are converted into the dimensionality of the current data groups or data elements through the data block dimensionality, and the transmitted data amount is reduced.

judging whether the data group is overlapped with the data interval determined by the data selection condition according to the index information of the data group, and if so, determining that the data group is a target data group; and determining target data according to the target data group.

In this embodiment, when determining whether the data groups satisfy the data selection condition, the determination may be directly performed according to the index information corresponding to each data group. Specifically, a data interval may also be determined according to the data selection condition, the value range of the data group may be determined according to the index information of the data group, and if the value range of the data group overlaps with the data interval, it indicates that data meeting the data selection condition exists in the data group, that is, the data group is the target data group.

The target data may be further determined after the target data set is acquired. Specifically, it may be determined according to the data selection condition.

According to the method, whether the data groups determined by the index information and the data intervals determined by the data selection conditions are overlapped or not is judged, and whether one data group contains data meeting the data selection conditions or not can be accurately determined.

In this embodiment, the index information may include two kinds, one is maximum value index information and minimum value index information, and one is bit index information. According to both index information, the range of each data element in the data group can be obtained.

And comparing the maximum index information and/or the minimum index information with the data interval determined by the data selection condition to judge whether the data group is overlapped with the data interval determined by the data selection condition. Fig. 4 is a schematic diagram of data extraction performed by using maximum value index information and minimum value index information according to an embodiment of the present invention. As shown in fig. 4, the data block includes a data group 1 and a data group 2, and an index information 1 corresponding to the data group 1 and an index information 2 corresponding to the data group 2, and data block meta information, where the data selection condition is greater than 10 and less than 13, the determined data interval is [11,12], and it is known from the index information 1 that the minimum value of the data group 1 is 1 and the maximum value is 8 that the value range of the data group can be determined to be [1,8], and it is known that the value range is not overlapped with the data interval, and the data group 1 does not satisfy the data selection condition. Similarly, according to the index information corresponding to the data group 2, it can be known that the value range of the data group 2 is [9,16], and that the value range overlaps with the data interval, and the data group 2 meets the data selection condition, so that the data group can be initially extracted.

When the index information is a bit index, the bit index may indicate whether the data group overlaps with the bucket section by 0 and 1, respectively. And further, according to the numerical value in the index information, whether the data group is overlapped with the data interval determined by the data selection condition can be determined. Fig. 5 is a schematic diagram of data extraction using bit indexes according to an embodiment of the present invention; as shown in fig. 5, first, according to the bucket boundary values in the data block meta-information, four bucket intervals are determined to be [1,4], [5,8], [9,12] and [13,16], where bit information respectively corresponds to the four bucket intervals, where bit information from right to left in the bit information sequentially corresponds to the four bucket intervals, and 1 indicates that there is an intersection with the bucket interval, and 0 indicates that there is no intersection with the bucket interval. Then, it can be determined that data elements in the bucket interval [1,4] and the bucket interval [9,12] exist in the data group 1 according to the bit index 0101 in the index information 1, and then the data group can be determined to satisfy the data selection condition by comparing the bucket interval with the data interval [11,12] corresponding to the data selection condition, and the data group can be extracted preliminarily. Likewise, for data set 2, it may be determined from the bit index 1010 that data set 2 does not satisfy the data selection condition.

Compared with the maximum index information and the minimum index information, the data selection method has the advantages that more data element information in the data group can be obtained in a bit index mode, the number of bit information can be flexibly set, and whether the data elements meeting data selection conditions exist in the data group or not can be accurately judged.

In this embodiment, further, information of an extraction granularity level may be obtained from the data requesting end, where the extraction granularity is a data group level or a data element level. Different data meeting the data selection condition can be sent to the data request terminal according to different extraction granularity levels.

Specifically, when the extraction granularity level is a data group level, the determined target data group is directly determined as target data, and the target data is the data group satisfying the data selection condition. As shown in fig. 4, at this time, the extracted data set, that is, the data set 2, may be transmitted to the data requester. As shown in fig. 5, the extracted data set, i.e., data set 1, may be sent to the data requesting side at this time.

In addition, when the extraction granularity level is a data element level, data elements meeting the data selection condition are screened from the extracted target data group, the screened data elements meeting the data selection condition are used as target data, and the target data is the data elements meeting the data selection condition. As shown in fig. 4 and 5, at this time, the extracted

data elements

11 and 12 may be transmitted to the data requester.

By setting the extraction granularity level information, data of different levels can be sent to the data request terminal, so that the transmitted data level can be more flexible.

determining the position information of each data group according to the initial offset information of each data group; and acquiring each data group according to the position information of each data group.

In this embodiment, the data block meta information stores start offset information of each data group, and position information of each data group can be determined according to the start offset information of each data group, as shown in fig. 4, it can be known that the position of the 0 th byte in the data block is the index information of the data group 1 and the data group 1 from the start offset information 1 in the data block meta information is 0. According to the fact that the start offset information 2 in the data block meta information is 100, the data block starting from the 100 th byte position is the index information of the data group 2 and the data group 2.

The size of the storage space occupied by the index information can be set to be a fixed value, so that after the data group and the index information corresponding to the data group are obtained, the storage positions of the index information and the data group can be determined. Or, the actual size of the storage space occupied by the index information may be acquired and stored in the data block meta information, so that the index information of each data group and each data group may be determined according to the initial offset information of the data group and the actual size of the storage space occupied by the index information.

In addition, the data block meta-information may further include data group number information, a maximum value and a minimum value of the data block, and the like.

By setting the start offset information of each data group in the data block meta information, each data group can be accurately obtained.

judging whether alarm information appears when target data is extracted according to the data selection condition and the index information of the data group; and when the alarm information appears, returning an error code corresponding to the alarm information so that the data request terminal executes data extraction operation corresponding to the error code.

In this embodiment, an abnormal condition may occur in the process of extracting the target data, such as a request timeout, that is, after receiving the data request information of the data request end, the target data is not sent over a preset time, or an abnormal condition may occur in a cache space for storing the target data in the storage device. Therefore, in the process of extracting data, when the abnormal condition occurs, the alarm information can be sent out. Each kind of alarm information may correspond to an error code, and when the data request end receives the error code, an abnormal condition may be determined.

When alarm information occurs, the computing storage device stops the data extraction operation, empties target data stored in a cache space, returns an error code corresponding to the alarm information to the data request terminal, and enables the data request terminal to execute the data extraction operation according to the error code, wherein the data extraction operations corresponding to different error codes can be the same or different.

By judging whether an abnormal condition occurs in the data extraction process and executing data extraction operation through the data request terminal when the abnormal condition occurs, the situation that data meeting data selection conditions cannot be extracted when the calculable storage device is abnormal can be prevented, and smooth completion of the data extraction operation is ensured.

Optionally, sending the target data to the data request end includes:

In this embodiment, after the target data is obtained, the target data needs to be encapsulated according to a preset rule, so that the data requesting end can correctly decapsulate the target data after receiving the encapsulated data.

In this embodiment, after the target data is obtained, the target data needs to be encapsulated, and then the encapsulated data is sent to the data request terminal. FIG. 6 is a diagram illustrating a package format of a data set according to an embodiment of the present invention; fig. 7 is a schematic diagram of a packaging format of a data element according to an embodiment of the present invention. As shown in fig. 6, when the acquired target data is a data group, it is necessary to encapsulate data length information, the data group, offset information of the data group, number information of the data group, and a check code. As shown in fig. 7, when the obtained target data is a data element, the data length information, the data element number information, and the check code need to be encapsulated.

And whether the obtained target data has errors or not can be preliminarily judged through the data length information. Whether errors occur in the target data in the process of transmitting the target data from the computable storage device to the data request end can be judged through the check code.

After the data request end receives the returned encapsulated data, the data can be analyzed firstly, assuming that the data length information, the data group offset information, the data group number information and the check code all occupy 4 bytes, the check code information can be obtained through the last four bytes, and whether the data has errors is checked according to the check code information. The number of the data groups can be determined according to the data of the first 4 bytes of the check code, if the number of the data groups is two, the data group offset information of the two data groups can be respectively obtained according to the data of the first 4 bytes of the number of the data groups, the storage positions of the data group 1 and the data group 2 can be determined according to the data group offset information, and finally the data group 1 and the data group 2 are obtained. The data length information may be used to check whether the encapsulated data is correct. The process of analyzing the encapsulated data when the target data is a data element is similar to the above process, and is not described herein again.

When the extracted data is in a data group level, the processor of the data request end can perform further extraction of the data element level after the data request end acquires the data group. When the extracted data is at the data element level, the data requesting end can directly use the extracted data elements.

And the acquired data is packaged according to a preset packaging format, so that the packaged data can be conveniently analyzed by a data request terminal.

receiving a data block and storage address information sent by a data request end, wherein the data block comprises at least one data group, index information of each data group and data block meta information; and writing the data block into a storage medium according to the storage address information.

In this embodiment, before receiving the data request information sent by the data request end, it is necessary to receive the data sent by the data request end and store the data in the storage medium. The data request end may divide the data block into a plurality of data groups according to a preset format, and obtain index information of each data group and meta information of the data block, which may also be a generation stage of the data block. Specifically, the data in the data block may be divided into a plurality of data groups according to the byte size, for example, the data in the data block is sequentially read, data of each preset number of bytes is used as one data group (for example, 128 bytes is used as one data group), and the index information of the data group, such as the maximum index information and the minimum index information of the data group, or the bit index information of the data group, is obtained. After the index information of each data group is obtained, data block meta information of the data block, such as start offset information of each data group in the data block, may also be obtained, and the number of data groups, the maximum value and the minimum value of the data block, and the like may also be included.

And combining the data group, the index information and the data block meta information according to a preset rule to obtain a data block, and sending the data block and the storage address information of the data block to the computable storage device. Specifically, when the data blocks are combined according to the preset rule, each data group and the index information of the data group may be sequentially stored, and finally, the data block meta information of the data block may be stored.

After receiving the data block and the storage address information, the computable storage device stores the data block in the storage medium according to the storage address information.

In addition, it should be noted that the purpose of filtering data through the computing storage device is to reduce the amount of data transmitted between the computing storage device and the data request end on the basis of not reducing the data processing speed, and therefore, an accelerator may be disposed in the computing storage device to increase the data processing speed.

Fig. 8 is a schematic flow chart of another data extraction method according to an embodiment of the present invention, where the method of this embodiment may be executed by a data extraction device, and the data extraction device may be disposed in the data request end of fig. 1 in the form of software and/or hardware. As shown in fig. 8, the method of this embodiment may include:

s801, sending data request information to a calculable storage device so that the calculable storage device can acquire a target data block from all data blocks according to storage address information of the target data block in the data request information, determine a data group contained in the data block according to data block meta information, and extract target data according to the data selection condition and index information of the data group; the target data is data meeting data selection conditions; the data request information comprises storage address information and data selection conditions of the target data block; the target data block comprises at least one data group, index information of each data group and data block meta information;

s802, receiving the target data sent by the computable storage device.

In this embodiment, the operation performed by the data request end corresponds to the operation performed by the computable storage device, and the implementation principle and the technical effect thereof are similar and will not be described herein again.

In this embodiment, the data request end needs to divide the data block into a plurality of data groups in advance, and generate index information and data block meta information of each data group, where when dividing the data group, the data with a preset byte length is divided into one data group according to the storage sequence of the original data.

In addition, in this embodiment, operations performed by the data request end when the packaged data is received and an abnormal condition occurs in the process of extracting data by the computable storage device are the same as those performed in the above embodiment, and the implementation principle and the technical effect thereof are similar and are not described herein again.

Fig. 9 is a schematic structural diagram of a data extraction apparatus according to an embodiment of the present invention; the data extraction device 90 includes:

a receiving module 901, configured to receive data request information sent by a data request end, where the data request information includes storage address information of a target data block and a data selection condition;

an obtaining module 902, configured to obtain a target data block from all data blocks according to the storage address information, where the target data block includes at least one data group, index information of each data group, and data block meta information;

an extracting module 903, configured to determine a data group included in the data block according to data block meta information, and extract target data according to the data selection condition and index information of the data group; the target data is data meeting data selection conditions;

a sending module 904, configured to send the target data to the data request end.

Optionally, the extracting module 903 includes a target data group determining unit and a target data determining unit:

the target data group determining unit is used for judging whether the data group is overlapped with the data interval determined by the data selection condition according to the index information of the data group, and if so, determining the data group as a target data group;

and the target data determining unit is used for determining target data according to the target data group.

Optionally, the target data group determining unit is specifically configured to:

Optionally, the data request information further includes: extracting granularity level information; the target data determination unit is specifically configured to:

Optionally, the data block meta information includes start offset information of each data group; when the data group included in the data block is determined according to the data block meta information, the extraction module is specifically configured to:

Optionally, the extracting module further includes an exception handling unit, where the exception handling unit is specifically configured to:

Optionally, the sending module 904 is specifically configured to:

Optionally, when the sending module 904 encapsulates the target data according to a preset rule, the sending module is specifically configured to:

Optionally, the apparatus further includes a data writing module, where the data writing module is configured to:

receiving a data block and storage address information sent by a data request end, wherein the data block comprises at least one data group and meta information of an index information data block of each data group;

The data extraction device provided in the embodiment of the present invention can implement the data extraction method in the embodiment shown in fig. 2, and the implementation principle and the technical effect are similar, which are not described herein again.

FIG. 10 is a schematic structural diagram of another data extraction apparatus according to an embodiment of the present invention; the data extraction apparatus 100 includes:

a sending module 1001, configured to send data request information to a storage device capable of computing, so that the storage device capable of computing obtains a target data block from all data blocks according to storage address information of the target data block in the data request information, determines a data group included in the data block according to data block metadata, and extracts target data according to the data selection condition and index information of the data group; the target data is data meeting data selection conditions; the data request information comprises storage address information and data selection conditions of the target data block; the target data block comprises at least one data group, index information of each data group and data block meta information;

a receiving module 1002, configured to receive the target data sent by the computing storage device.

Optionally, the apparatus further includes a generation module, where the generation module is configured to:

The data extraction device provided in the embodiment of the present invention can implement the data extraction method in the embodiment shown in fig. 7, and the implementation principle and the technical effect are similar, which are not described herein again.

FIG. 11 is a diagram illustrating a hardware configuration of a computing storage device according to an embodiment of the present invention; as shown in fig. 11, the present embodiment provides a computing storage device 110 including: at least one processor 1101 and memory 1102. The processor 1101 and the memory 1102 are connected by a bus 1103.

In a specific implementation process, the at least one processor 1101 executes the computer-executable instructions stored in the memory 1102, so that the at least one processor 1101 executes the data extraction method in the above-described method embodiment.

For a specific implementation process of the processor 1101, reference may be made to the above method embodiments, which implement similar principles and technical effects, and details of this embodiment are not described herein again.

Fig. 12 is a schematic hardware structure diagram of a data request device according to an embodiment of the present invention; as shown in fig. 12, the present embodiment provides a computable storage device 120 including: at least one processor 1201 and memory 1202. The processor 1201 and the memory 1202 are connected by a bus 1203.

In a specific implementation process, the at least one processor 1201 executes computer-executable instructions stored in the memory 1202, so that the at least one processor 1201 executes the data extraction method in the foregoing method embodiment.

For a specific implementation process of the processor 1201, reference may be made to the above method embodiments, which have similar implementation principles and technical effects, and details are not described herein again.

In the embodiments shown in fig. 11 and fig. 12, it should be understood that the Processor may be a Central Processing Unit (CPU), other general purpose processors, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of a method disclosed in connection with the present invention may be embodied directly in a hardware processor, or in a combination of the hardware and software modules within the processor.

The memory may comprise high speed RAM memory and may also include non-volatile storage NVM, such as at least one disk memory.

The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended ISA (EISA) bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, the buses in the figures of the present application are not limited to only one bus or one type of bus.

The embodiment of the present invention further provides a computer-readable storage medium, where a computer execution instruction is stored in the computer-readable storage medium, and when a processor executes the computer execution instruction, the data extraction method of the above method embodiment is implemented.

The computer-readable storage medium may be implemented by any type of volatile or non-volatile memory device or combination thereof, such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disk. Readable storage media can be any available media that can be accessed by a general purpose or special purpose computer.

An exemplary readable storage medium is coupled to the processor such the processor can read information from, and write information to, the readable storage medium. Of course, the readable storage medium may also be an integral part of the processor. The processor and the readable storage medium may reside in an Application Specific Integrated Circuits (ASIC). Of course, the processor and the readable storage medium may also reside as discrete components in the apparatus.

Those of ordinary skill in the art will understand that: all or a portion of the steps of implementing the above-described method embodiments may be performed by hardware associated with program instructions. The program may be stored in a computer-readable storage medium. When executed, the program performs steps comprising the method embodiments described above; and the aforementioned storage medium includes: various media that can store program codes, such as ROM, RAM, magnetic or optical disks.

Finally, it should be noted that: the above embodiments are only used to illustrate the technical solution of the present invention, and not to limit the same; while the invention has been described in detail and with reference to the foregoing embodiments, it will be understood by those skilled in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data extraction method applied to a computable storage device comprises the following steps:

and sending the target data to the data request terminal.

2. The method according to claim 1, wherein extracting the target data according to the data selection condition and the index information of the data group comprises:

and determining target data according to the target data group.

3. The method according to claim 2, wherein determining whether the data group overlaps with the data section determined by the data selection condition according to the index information of the data group comprises:

4. The method of claim 2, wherein the data request information further comprises: extracting granularity level information; determining target data from the target data set, comprising:

5. The method of claim 1, wherein the data block meta information includes start offset information of each data group; the determining a data group contained in the data block according to the data block meta information includes:

6. The method according to claim 2, wherein extracting the target data according to the data selection condition and the index information of the data group further comprises:

7. The method of claim 1, wherein sending the target data to the data requestor comprises:

8. The method of claim 7, wherein encapsulating the target data according to a preset rule comprises:

9. The method according to any one of claims 1 to 8, wherein before receiving the data request information sent by the data request side, the method further comprises:

10. A data extraction method is applied to a data request device, and comprises the following steps:

receiving the target data sent by the computable storage device.

11. The method of claim 10, wherein prior to sending the data request information to the computable storage device, further comprising:

12. A data extraction apparatus, comprising:

13. A data extraction apparatus, comprising:

14. A computable storage device, comprising: at least one processor and memory;

the memory stores computer-executable instructions;

the at least one processor executing the computer-executable instructions stored by the memory causes the at least one processor to perform the data extraction method of any of claims 1 to 9.

15. A data requesting device, comprising: at least one processor and memory;

the memory stores computer-executable instructions;

the at least one processor executing the computer-executable instructions stored by the memory causes the at least one processor to perform the data extraction method of claim 10 or 11.

16. A computer-readable storage medium having computer-executable instructions stored therein, which when executed by a processor, implement the data extraction method of any one of claims 1 to 11.

17. A program product, comprising a computer program, characterized in that the computer program, when being executed by a processor, implements the data extraction method according to any one of claims 1-11.