CN112540984A

CN112540984A - Data storage method, query method, device, electronic equipment and storage medium

Info

Publication number: CN112540984A
Application number: CN202011327056.XA
Authority: CN
Inventors: 付威; 廖强; 黄志龙; 孙春; 王晓健
Original assignee: Chengdu Jiahua Chain Cloud Technology Co ltd
Current assignee: Chengdu Jiahua Chain Cloud Technology Co ltd
Priority date: 2020-11-23
Filing date: 2020-11-23
Publication date: 2021-03-23
Anticipated expiration: 2040-11-23
Also published as: CN112540984B

Abstract

The application provides a data storage method, an inquiry method, a device, an electronic device and a storage medium, and relates to the technical field of computers. The data query method comprises the following steps: acquiring target data written by an upper application from a disk, and writing the target data into a target memory space which is distributed for the target data and is in an active state, wherein the active state represents that data are allowed to be written into the memory space; when the storage state conversion condition is determined to be met, setting the storage state of the target memory space to be an unstable state, wherein the unstable state represents that data are not allowed to be written into the memory space; and compressing and storing the target data in the target memory space with the storage state being an unstable state. According to the scheme, the data in the disk are read into the memory for storage, so that the data can be queried from the memory when being queried, and the data can be efficiently queried due to the fact that only a small amount of data are stored in the memory, and the efficiency of data query is improved.

Description

Data storage method, query method, device, electronic equipment and storage medium

Technical Field

The present application relates to the field of computer technologies, and in particular, to a data storage method, an inquiry method, an apparatus, an electronic device, and a storage medium.

Background

The existing data storage mainly uses a memory as a cache and a hard disk as a persistent storage medium, after data receives a write request of a user, the data is usually written into the cache, and when the data amount in the cache meets a certain condition, the data is persisted into the disk. When data query is performed, data is generally queried from a disk file in a mmap or other modes, and then a query result is returned to a user. However, since the disk stores massive data, when data is queried in the disk, corresponding data needs to be searched from the massive data, and the query efficiency is low.

Disclosure of Invention

An embodiment of the present application provides a data storage method, an inquiry method, an apparatus, an electronic device, and a storage medium, so as to solve the problem of low inquiry efficiency in the prior art.

In a first aspect, an embodiment of the present application provides a data storage method, where the method includes: acquiring target data written by an upper application from a disk, and writing the target data into a target memory space which is distributed for the target data and is in an active state, wherein the active state represents that data are allowed to be written into the memory space; when the storage state conversion condition is determined to be met, setting the storage state of the target memory space to be an unstable state, wherein the unstable state represents that data are not allowed to be written into the memory space; and compressing and storing the target data in the target memory space with the storage state being an unstable state.

In the implementation process, the data in the disk is read into the memory for storage, so that the data can be queried from the memory when the data is queried, and because only a small amount of data is stored in the memory, the data can be queried efficiently, and the efficiency of data query is improved. And the memory space is marked in an active state and an unstable state, so that the writing of data is allowed when the memory space is in the active state, the writing of the data is not allowed when the memory space is in the unstable state, and the data is compressed and stored in time, so that the memory consumption is reduced while the quick writing is realized, and the storage requirement of more data is met.

Alternatively, it is determined that the storage state transition condition is satisfied by:

judging whether the data volume in the target memory space in the active state exceeds a set threshold value or not;

and if so, determining that the storage state transition condition is met.

In the implementation process, when the data amount in the target memory space exceeds the set threshold, the storage state of the target memory space is set to be an unstable state, so that the data in the target memory space can be compressed in time, and the memory consumption is reduced.

Optionally, the storage state transition condition is satisfied by:

judging whether new data are written in a target memory space with the storage state being an active state within a preset time period;

if not, determining that the storage state transition condition is met.

In the implementation process, when no new data is written in the target memory space within a preset time period, the storage state of the target memory space is set to be an unstable state, so that the data in the target memory space can be compressed in time, and the memory consumption is reduced.

Optionally, the compressing and storing the target data in the memory space with the storage state being an unstable state includes:

acquiring stable data in a memory space for storing compressed data;

judging whether the time ranges of the stable data and unstable data in a target memory space with an unstable storage state are overlapped or not, wherein the unstable data comprise the target data;

if so, decompressing the stable data to obtain decompressed data;

merging the decompressed data and the unstable data to obtain merged data;

and compressing the merged data to form compressed data for storage.

In the implementation process, when the time ranges of the stable data and the unstable data are overlapped, the stable data is decompressed, then merged with the unstable data, and then compressed, so that the integrity and the orderliness of the data during compression can be ensured.

Optionally, the compressing and storing the target data in the target memory space with the storage state being an unstable state includes:

acquiring stable data in a memory space for storing compressed data;

if not, compressing the unstable data to form new compressed data;

and writing the new compressed data into the memory space for storage.

In the implementation process, when the time ranges of the stable data and the unstable data are not overlapped, the unstable data can be directly compressed without recompressing the stable data, so that the data compression amount can be reduced, and the compression time can be saved.

Optionally, after compressing and storing the target data in the target memory space whose storage state is an unstable state, the method further includes:

the storage state of the memory space is set to be a stable state, so that stable data can be conveniently searched according to the stable state when data compression is carried out.

Optionally, the method further comprises:

and after the data in the target memory space with the storage state being the unstable state is determined to be compressed and stored in the memory space, releasing the target memory space, thereby timely recovering the memory space and reducing the space occupation.

Optionally, the data in the memory space is divided into a plurality of data blocks according to a time sequence for storage, each data block includes a preset number of data and a checksum of the data, the memory space further stores index data of the plurality of data blocks, and the index data includes an offset of each data block, a size of each data block, and a start time of the data.

In the implementation process, the stable data is stored in a data block mode, so that when data query is carried out, the data can be rapidly queried through index data.

Optionally, the target memory space is managed by a memory storage engine, where the memory space managed by the memory storage engine includes a plurality of memory spaces, each memory space is used to store data in a corresponding time period, and the writing the target data into the target memory space in an active state allocated to the target memory space includes:

acquiring a time point of the target data;

determining a corresponding target memory space in an active state according to the time point;

and writing the target data into the target memory space.

In the implementation process, the data is written into the corresponding memory space according to the time point of the data, so that the data is stored in a segmented manner, the data compression amount can be reduced during data compression, and the compression efficiency is improved.

Optionally, the data storage in the disk is managed by a disk storage engine, and the obtaining target data written by an upper application from the disk includes:

and reading the target data in the preset time period from the magnetic disk through the magnetic disk storage engine, and managing the data storage of the magnetic disk through the magnetic disk storage engine, so that the effective writing of the data can be realized when the data is written into the memory.

In a second aspect, an embodiment of the present application provides a data query method, where the method includes:

receiving a data query request;

determining a time range needing to be queried according to the data query request;

determining a target memory space in a plurality of memory spaces managed by a memory storage engine according to the time range, wherein data of a corresponding time period is stored in each memory space according to the method provided by the first aspect;

and querying the data in the target memory space to obtain a corresponding query result.

In the implementation process, by performing data query in the memory space managed by the memory storage engine, compared with a mode of performing data query in a disk, the query efficiency can be effectively improved.

Optionally, when the query result does not meet the preset requirement or the memory storage engine fails, the disk storage engine is instructed to query according to the query request, so that data query can be realized when the memory storage engine cannot query.

Optionally, the memory storage engine and the disk storage engine are respectively arranged in different devices, so that when any one of the engines fails, service availability is not affected, and data query can be realized.

In a third aspect, an embodiment of the present application provides a data storage device, including:

the data writing module is used for acquiring target data written by upper-layer application from a disk and writing the target data into a target memory space which is distributed for the target data and is in an active state, wherein the active state represents that data are allowed to be written into the memory space;

the state conversion module is used for setting the storage state of the target memory space to be an unstable state when the storage state conversion condition is determined to be met, and the unstable state represents that data are not allowed to be written into the memory space;

and the data compression module is used for compressing and storing the target data in the target memory space with the storage state being an unstable state.

In a fourth aspect, an embodiment of the present application provides a data query apparatus, including:

the request receiving module is used for receiving a data query request;

the time determining module is used for determining a time range needing to be queried according to the data query request;

a memory determination module, configured to determine a target memory space in a plurality of memory spaces managed by a memory storage engine according to the time range, where data of a corresponding time period is stored in each memory space according to the method according to any one of claims 1 to 10;

and the data query module is used for querying the data in the target memory space to obtain a corresponding query result.

In a fifth aspect, an embodiment of the present application provides an electronic device, including a processor and a memory, where the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the method of the first or second aspect is executed.

In a sixth aspect, embodiments of the present application provide a readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, performs the steps in the method as provided in the first or second aspect.

Additional features and advantages of the present application will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by the practice of the embodiments of the present application. The objectives and other advantages of the application may be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are required to be used in the embodiments of the present application will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present application and therefore should not be considered as limiting the scope, and that those skilled in the art can also obtain other related drawings based on the drawings without inventive efforts.

Fig. 1 is a schematic structural diagram of an electronic device for executing a data storage method or a data query method according to an embodiment of the present application;

fig. 2 is a flowchart of a data storage method according to an embodiment of the present application;

FIG. 3 is a schematic diagram illustrating overlapping of time ranges of stable data and unstable data according to an embodiment of the present disclosure;

FIG. 4 is a schematic diagram of a stable data storage structure according to an embodiment of the present application;

FIG. 5 is a flowchart of a data query method provided in an embodiment of the present application;

FIG. 6 is a diagram illustrating a data query process provided by an embodiment of the present application;

fig. 7 is a block diagram of a data storage device according to an embodiment of the present application;

fig. 8 is a block diagram of a data query device according to an embodiment of the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application.

The embodiment of the application provides a data storage method, which reads data in a disk into a memory for storage, so that when data is queried, the data can be queried from the memory. And the memory space is marked in an active state and an unstable state, so that the writing of data is allowed when the memory space is in the active state, the writing of the data is not allowed when the memory space is in the unstable state, the data is compressed and stored in time, the writing of the data is not allowed when the data is compressed, the integrity in the data compression process can be ensured, the fast writing can be realized, the memory consumption is reduced, and the storage requirement of more data is met.

Referring to fig. 1, fig. 1 is a schematic structural diagram of an electronic device for executing a data storage method or a data query method according to an embodiment of the present application, where the electronic device may include: at least one processor 110, such as a CPU, at least one communication interface 120, at least one memory 130, and at least one communication bus 140. Wherein the communication bus 140 is used for realizing direct connection communication of these components. The communication interface 120 of the device in the embodiment of the present application is used for performing signaling or data communication with other node devices. The memory 130 may be a high-speed RAM memory or a non-volatile memory (e.g., at least one disk memory). Memory 130 may optionally be at least one memory device located remotely from the aforementioned processor. The memory 130 stores computer readable instructions, which when executed by the processor 110, cause the electronic device to perform the method processes of fig. 2 or 5 described below.

It will be appreciated that the configuration shown in fig. 1 is merely illustrative and that the electronic device may also include more or fewer components than shown in fig. 1 or have a different configuration than shown in fig. 1. The components shown in fig. 1 may be implemented in hardware, software, or a combination thereof.

Referring to fig. 2, fig. 2 is a flowchart of a data storage method according to an embodiment of the present application, where the method includes the following steps:

step S110: and acquiring target data written by the upper application from a disk, and writing the target data into a target memory space which is distributed for the target data and is in an active state.

The upper layer application may refer to an application installed in the terminal device, and the upper layer application may store data input by a user or data obtained from another place, and when storing the data, the upper layer application writes the data into a magnetic disk for storage.

In the embodiment of the present application, the target data may refer to time series data, and the time series data refers to time series data with a time tag. The data collected or generated in the real-time monitoring process is generally realized in the fields of electric power, chemical industry and the like, and the data has the characteristics of high generation frequency, dependence on collection time and large information quantity of the ceiling lamp, so that dozens of GB data quantity can be generated every day. In order to support the query of high-throughput time series data, the data can be read from the disk and stored in the memory, so that the data in the memory can be queried first when the query is performed, and the fast query can be realized because a small amount of data is stored in the memory.

The acquired target data may be data of a latest period of time, that is, the latest stored data may be acquired from the disk in real time, or the latest stored data may be acquired from the disk at regular time, and in order to reduce the memory usage, the historical data in the memory may be deleted at regular time.

It can be understood that, the specific form of obtaining the target data may be to copy the target data from a disk, that is, the disk still stores all original data, in order to improve the query efficiency, and since the query of the time series data may mostly involve data in the latest time period, the data written in the disk in the latest time period may be copied to the memory as the target data, thereby implementing the query of the data in the memory.

When the target data acquired from the disk is stored, in order to implement fast writing of the data, the data may be written into a target memory space allocated to the target memory space in an active state, where the active state represents that the data is allowed to be written into the memory space.

For example, when obtaining the target data, a memory space is applied for the target data, assuming that the address is 0x0000-0x0fff, the memory space corresponding to the address is the target memory space. When a target memory space is applied, the target memory space is in an active state. In some embodiments, after applying for the target memory space, a flag bit may be set for the target memory space to mark an active state, and then target data may be written into the target memory space; or, after applying for the target memory space, the target data may be written into the target memory space first, and then the target memory space is set to be in an active state.

The memory space marked as active may allow data to continue to be written. If the target memory space is in an active state, the subsequently acquired data may be continuously written into the target memory space, that is, after other data is subsequently acquired, if the target memory space is still in an active state, the data may be continuously written into the target memory space without applying for another memory space.

Step S120: and when the storage state conversion condition is determined to be met, setting the storage state of the target memory space to be an unstable state.

Since the space of the target memory space is limited and data cannot be written all the time, the storage state of the target memory space is set to an unstable state when the target memory space satisfies the storage state transition condition. The specific implementation manner may be to modify the flag bit corresponding to the target memory space into a flag representing an unstable state, where if the flag bit corresponding to the target memory space is "0", the flag bit represents an active state, and when the storage state of the target memory space is set to an unstable state, the flag bit may be modified into "1", that is, the flag represents an unstable state.

The unstable state represents that data is not allowed to be written into the memory space, that is, after the storage state of the target memory space is set to the unstable state, new data cannot be written into the target memory space, and after new data is obtained, the memory space can be reapplied to write the new data into another memory space, for example, a memory space with an address of 0x1000-0x1fff is reapplied.

Step S130: and compressing and storing the target data in the target memory space with the storage state being an unstable state.

After the storage state of the target memory space is set to be an unstable state, the data in the target memory can be compressed, at this time, the data is not allowed to be written into the target memory space, and the data is time sequence data, so that the compression process involves sequencing and can not be written and compressed simultaneously, and the integrity of the data in the compression process can be ensured.

The occupied memory space can be reduced by compression, so that the data in the target memory space can be compressed, the occupied memory space of the compressed data can be effectively reduced compared with the data before compression, and more data can be acquired from the disk for storage, so that the subsequent data query can be more comprehensive.

There are various methods for compressing data, such as a lossless compression method, a lossy compression method, etc., and reference may be made to related implementations in the prior art, which are not illustrated herein.

In addition, when data is stored in the memory, the data is not limited to be stored in the manner described above, and when other data is stored in the memory, the data may be stored by using the method described above.

In some embodiments, whether the storage state transition condition is satisfied may be determined by: and judging whether the data volume in the target memory space in the active state exceeds a set threshold value, and if so, determining that a storage state conversion condition is met.

The data amount may be a data memory occupation amount or a data amount. In some embodiments, the data storage in the memory may be managed by a memory storage engine, that is, when the data is stored, the memory storage engine applies for a memory space, and then sets a corresponding storage state for the memory space. The memory storage engine may be disposed in the electronic device, and may monitor a data amount in the memory space, determine that a storage state transition condition is satisfied when the data amount in the target memory space exceeds a set threshold, and then set a storage state of the target memory space to an unstable state.

The set threshold can be flexibly set according to actual requirements, and it can be understood that the set threshold may be different in different application scenarios.

When the data is time-series data, the time-series data has the latest and hottest characteristic, the query and the write are concentrated in the latest data segment, and for the data segment which has no data written for a long time, no new data is written in the future. Therefore, in other embodiments, the manner of determining whether the storage state transition condition is satisfied may also be: and judging whether new data are written in the target memory space in which the storage state is the active state within a preset time period, and if not, determining that the storage state conversion condition is met.

For example, the target memory space is detected by the memory storage engine, the time of the latest written data in the target memory space and the current time are acquired in real time, if the time interval between the time of the latest written data and the current time exceeds a preset time period, it is determined that no new data is written in the preset time, the target memory space is determined to meet the storage state conversion condition, so that the storage state of the target memory space can be set to be an unstable state, and then the process of data compression is started.

The preset time period can also be flexibly set according to actual requirements, and the numerical values of the preset time period may also be different in different application scenes.

It can be understood that whether the memory space meets the storage state transition condition may also be determined in other manners, for example, the determination may be performed based on a time length for creating the memory space, for example, after the target memory space is created, a timer for the target memory space is started, and after the time length counted by the timer reaches a preset time length, it is determined that the storage state transition condition is met. The time length may be set according to a requirement, the same time length may be set for each memory space, or the time lengths may be set to be different, for example, when it is determined that the time length of the target memory space reaches 1 hour, and it is determined that the storage state transition condition is satisfied, the storage state of the target memory space is set to be an unstable state. Other determination methods are not listed here, and it should be understood that other determination methods are also included in the scope of the present application.

In some embodiments, to ensure ordered writing, the data in the memory space in the active state generally adopts a skip list storage structure, and the data is not compressed and stores the original value of the data to support rapid ordering and insertion of the data. However, in the skip list storage structure, since the index of the data node is maintained to support fast query, there is a certain expansion in space, and meanwhile, the structure supporting fast insertion of the linked list cannot utilize the correlation between adjacent data points to perform data compression coding, so that the data needs to be compressed after a certain time. In addition, the memory space in the active state usually sacrifices space efficiency to realize rapid insertion and query of data, so that the memory space can be recovered without reducing writing performance by compressing the data.

In the process of data compression, because compressed data may be stored in advance, and data may need to be reordered during compression, in the compression process, stable data in a memory space for storing the compressed data may be obtained first, then whether a time range of the stable data and unstable data in a target memory space with an unstable storage state overlaps or not is judged, if yes, the stable data is decompressed to obtain decompressed data, the decompressed data and the unstable data are combined to obtain combined data, and the combined data is compressed to form compressed data for storage, wherein the unstable data includes the target data.

The stable data refers to other compressed data except data in the memory space in the active state, and the other data form stable data after being subjected to compression coding processing.

That is, when compressing, the stable data may be read in a stream form, and it is determined whether the time range of the stable data overlaps with the time range of the unstable data in the target memory space currently in an unstable state, and when there is an overlapping portion (a schematic diagram of the overlapping portion is shown in fig. 3), it indicates that the two data may need to be reordered and then compressed. The stable data is compressed binary coding and cannot be directly inserted into the middle position, so the stable data is decompressed first, and then the decompressed data obtained by decompression and the unstable data are merged and then compressed.

In addition, the memory space for storing the compressed data and the target memory space are not the same memory space, so after decompressing the stable data in the memory space and merging the unstable data in the target memory space, in order to reduce the memory occupation or avoid the original memory space being smaller, the original memory space can be released, and then the compressed data formed by compressing the merged data is stored in another memory space again.

And if stable data does not exist in the initial stage, the unstable data can be directly compressed and stored without the process.

In other embodiments, if the time ranges of the stable data and the unstable data are not overlapped, the unstable data may be directly compressed to form new compressed data, and the new compressed data is written into the memory space for storage.

It can be understood that if the time ranges of the stable data and the unstable data do not overlap, it means that the stable data does not need to be reordered when data is compressed, so that the unstable data can be directly compressed to obtain new compressed data, and the new compressed data can be appended to the original stable data in the form of additional writing.

In some embodiments, in order to facilitate searching for stable data, after the target data is compressed, the storage state of the memory space for storing the compressed data may be set to be a stable state, so that the data stored in the memory space may be known to be compressed data by searching for the memory space in the stable state.

In some embodiments, in order to reduce memory consumption, after determining to compress and store data in the target memory space with the storage state being an unstable state into the memory space, the target memory space may be released. Thus, the target memory space can be recycled and continuously used for storing new data.

In some embodiments, to facilitate data query, data in a memory space storing compressed data is divided into a plurality of data blocks in a time sequence for storage, each database includes data of preset data and a checksum of the data, the memory space further stores index data of the plurality of data blocks, and the index data includes an offset of each data block, a size of each data block, and a start time of the data. The schematic diagram of the storage structure is shown in fig. 4.

When data query is carried out, the corresponding data block can be rapidly queried through the index data, so that the rapid query of the data is realized.

In some embodiments, to facilitate data storage management, the memory space may be managed by a memory storage engine, where the memory space managed by the memory storage engine includes a plurality of memory spaces, and each memory space is used for storing data in a corresponding time period.

For example, for time series data, the number of memory spaces managed by the memory storage engine may include 24, so that each memory space may store data for one hour, for example, data of 0-1 point is stored in the memory space 1, data of 1-2 points is stored in the memory space 2, and the storage state of data in each memory space may be classified into an active state, an unstable state, a stable state, and the like.

The reason why the memory space is divided into 24 spaces and each space corresponds to one hour of data is to reduce the amount of data to be merged and further reduce the amount of data to be compressed each time data compression is performed. Of course, in this practical application, the time period of the data stored in each memory space may also be flexibly set according to the time division requirement.

In the process of writing the target data into the target memory space, the time point of the target data may be obtained first, then the corresponding active target memory space is determined according to the time point, and then the target data is written into the target memory space.

For example, the time point of the target data is 11:30, at this time, the memory storage engine may apply for storing a memory space corresponding to the time period of 11-12 points, where the memory space is the target memory space, and then write the target data into the memory space, thereby implementing data writing.

In addition, since data is stored according to a time period, in order to reduce a data compression amount, each memory space for storing unstable data corresponds to a memory space for storing compressed data, for example, an initial storage state of the memory space 1 is an active state, after a storage state transition condition is satisfied, a storage state of the memory space 1 is changed into an unstable state, the memory space 1 corresponds to another memory space 2, and the memory space 2 is used for storing compressed data formed by compressing the memory space 1. After data in the memory space 1 is compressed to form compressed data and the compressed data is stored in the memory space 2, the memory space 1 is released, and then when data in a time period corresponding to the memory space 1 is received, the memory storage engine reapplies the memory space 1 to store newly received data, and then when a storage state conversion condition is met, the data in the memory space 1 is compressed and stored in the memory space 2.

In order to reduce memory consumption, the memory space for storing the compressed data may be released after a certain time, for example, the stable data stored in the memory space 2 is deleted after a certain time period, and the memory space 2 is released, so that the memory space 2 may be used to continue to store the newly compressed data.

In some embodiments, in order to facilitate storage management of a large amount of data, data storage in the disk may be managed by a disk storage engine, so that when the memory storage engine obtains target data, the disk storage engine may read the target data from the disk within a preset time period.

That is to say, data written by the upper application is stored in the disk first, because the amount of data stored in the disk is large, if data query is performed on the disk, it may take a long time, so that a part of data in the disk may be written into the memory, for example, the disk storage engine may copy data in the disk for storage, and then the memory storage engine copies corresponding data from the disk storage engine and stores the data in the above manner, so that when data query is performed, query may be performed in the memory first, and because the amount of data in the memory is small, fast query may be implemented.

Due to the characteristic that the time sequence data is the newest and hottest, the target data in the preset time period can be read from the disk and stored in the memory, and the reading mode can be copying, so that all data can be stored in the disk, and only the newest data can be stored in the memory. The preset time period may be a latest time period, that is, the memory storage engine may read data in the latest time period from the disk in real time as target data to be stored in the target memory space. In order to reduce the memory consumption, the data stored in the memory may be deleted after a certain period of time, so as to ensure that there is enough memory for storing the new data read from the disk.

In the embodiment, the target data is read from the disk for storage, so that when data is queried, the data in the memory can be directly queried, and the data can be rapidly queried, thereby supporting the query of high-throughput time series data. In addition, by storing the data in an active state, an unstable state or a stable state, the memory consumption can be reduced while fast writing is realized, so that the storage requirement of more data is met.

Referring to fig. 5, fig. 5 is a flowchart of a data query method according to an embodiment of the present application, where the method includes the following steps:

step S210: a data query request is received.

When data query is performed, a user may submit a data query request through a client, where the data query request may, for example, query data of a certain time period or certain data at a certain time point. After the client obtains the data query request, the client submits the data query request to the memory storage engine, and the memory storage engine receives the query request sent by the client.

Step S220: and determining the time range needing to be queried according to the data query request.

In order to realize efficient query, the data query request carries time data to be queried, so that the memory storage engine can acquire a time range to be queried from the data query request, such as certain data in a time period with the time range of 2-3:30 points.

Step S230: and determining a target memory space in the plurality of memory spaces managed by the memory storage engine according to the time range.

Each memory space stores data in a corresponding time period according to the data storage method described in the above embodiment, so that the memory storage engine can determine the memory space to be queried according to the time range to be queried, and if the time period of the data stored in the memory space 2 is 2-3 points and the time period of the data stored in the memory space 3 is 3-4 points, determining the corresponding target memory space according to the time range to be queried includes the memory space 2 and the memory space 3, which means that the corresponding data can be searched in the memory space 2 and the memory space 3.

Step S240: and querying the data in the target memory space to obtain a corresponding query result.

After determining the corresponding target memory space, such as the memory space 2 and the memory space 3, the memory storage engine may search data meeting a certain condition in the two memory spaces, that is, search and screen data meeting the query condition, and return the searched data as a corresponding query result to the client for the user to check.

When searching for data in the memory space, the data can be searched for in active data, unstable data and stable data in the memory space in an active state, certainly, if the time range required to be searched for is small, such as data in a time period of 2:15-2:30, the data required to be searched for can be further determined according to the time ranges of the active data, the unstable data and the stable data, and if the time range corresponding to the unstable data is 2:10-2:30, the data can be searched for in the unstable data, so that the search range can be shortened, and the data can be quickly searched for.

The data in the memory space is acquired from the disk by the memory storage engine, so that when data is queried, the data is queried in the memory first instead of being directly queried in the disk, and the query efficiency can be improved.

In some embodiments, if the query result does not satisfy the preset requirement or the memory storage engine fails, the disk storage engine may be instructed to perform the query according to the query request.

The preset requirement may be that the memory storage engine does not find the data meeting the query condition in the target memory space, or the data amount of the data meeting the query condition does not exceed a set amount.

Or, when the memory storage engine fails and cannot perform data query, the disk storage engine may be instructed to perform query according to the query request. The disk storage engine is used for managing data storage in the disk, the memory storage engine can read data from the disk in real time through the disk storage engine, for example, the disk storage engine can generate a replication log according to data input by a user, and the memory storage engine can pull the replication log from the disk storage engine and store the replication log locally. In consideration of cost and other factors, the memory space managed by the memory storage engine usually cannot store the same amount of data on the disk, and in order to fully utilize the memory space, the memory storage engine may only store the latest data for a period of time, and the disk stores all the data. When the memory storage engine fails or the query does not meet the preset requirement, the memory storage engine can directly instruct the disk storage engine to perform the query, or can output corresponding indication information to the client to instruct the client to resubmit the query request to the disk storage engine, so that the query can be performed again through the disk storage engine.

When the disk storage engine obtains the query request, data query can be performed in the disk, and then a query result obtained by the query can be returned to the client.

In addition, the query result may include data obtained by querying by the memory storage engine or no query result, when no corresponding data is queried, the query result returned by the memory storage engine to the client may be prompt information of "no query result", and if corresponding data is queried, the returned query result includes data obtained by querying. After the client obtains the corresponding query result, if the user is not satisfied with the query result, the client can also directly submit the query request to the disk storage engine, and the disk storage engine performs the query.

In some embodiments, in order to enable the memory space and the disk space not to be affected, the memory storage engine and the disk storage engine may be respectively disposed in different devices, for example, in different servers, so that both the memory space and the disk space may not be considered by the two servers. Therefore, when the memory storage engine fails, the disk storage engine performs service degradation to realize the query of the disk data.

The query process is as shown in fig. 6, and in such a heterogeneous system, the memory storage engine can implement high-performance data reading based on the memory and provide the horizontal expansion capability of the system read load. In addition, after the memory storage engine is restarted due to a failure, all queries can be executed by the disk storage engine before the data is completely copied from the disk.

That is, when the memory storage engine is not in use or fails, the disk storage engine performs the query to ensure that the query request of the user can be satisfied.

In addition, after the memory storage engine is recovered to be normal, the memory storage engine can continue to perform data query. In some cases, the client may not be able to sense whether the memory storage engine fails, so the client may submit the query request to the memory storage engine, and if the memory storage engine fails, the client may not obtain the query result, and the client may submit the query request to the disk storage engine. If the memory storage engine recovers to be normal, the memory storage engine can send prompt information that the memory storage engine has recovered to be normal to the client, so that the client can continue to submit the query request to the memory storage engine.

In the above embodiment, the data query is implemented through a heterogeneous system of the memory storage engine and the disk storage engine, so that when any engine fails, the service availability is not affected, and the data query can be implemented.

Referring to fig. 7, fig. 7 is a block diagram of a data storage device 200 according to an embodiment of the present disclosure, where the data storage device 200 may be a module, a program segment, or code on an electronic device. It should be understood that the apparatus 200 corresponds to the above-mentioned embodiment of the method of fig. 2, and can perform various steps related to the embodiment of the method of fig. 2, and the specific functions of the apparatus 200 can be referred to the above description, and the detailed description is appropriately omitted here to avoid redundancy.

Optionally, the apparatus 200 comprises:

a data writing module 210, configured to obtain target data written by an upper application from a disk, and write the target data into a target memory space in an active state allocated to the target data, where the active state represents that data is allowed to be written into the memory space;

a state transition module 220, configured to set a storage state of the target memory space to an unstable state when it is determined that a storage state transition condition is met, where the unstable state represents that data is not allowed to be written into the memory space;

and the data compression module 230 is configured to compress and store the target data in the target memory space with the storage state being an unstable state.

Optionally, the state transition module 220 is configured to determine whether a data amount in the target memory space in the active state exceeds a set threshold; and if so, determining that the storage state transition condition is met.

Optionally, the state transition module 220 is configured to determine whether new data is written in a target memory space in which the storage state is an active state within a preset time period; if not, determining that the storage state transition condition is met.

Optionally, the data compression module 230 is configured to obtain stable data in a memory space for storing compressed data; judging whether the time ranges of the stable data and unstable data in a target memory space with an unstable storage state are overlapped or not, wherein the unstable data comprise the target data; if so, decompressing the stable data to obtain decompressed data; merging the decompressed data and the unstable data to obtain merged data; and compressing the merged data to form compressed data for storage.

Optionally, the data compression module 230 is configured to obtain stable data in a memory space for storing compressed data; judging whether the time ranges of the stable data and unstable data in a target memory space with an unstable storage state are overlapped or not, wherein the unstable data comprise the target data; if not, compressing the unstable data to form new compressed data; and writing the new compressed data into the memory space for storage.

Optionally, the state transition module 220 is further configured to set the storage state of the memory space to a stable state.

Optionally, the apparatus 200 further comprises:

and the memory releasing module is used for releasing the target memory space after determining that the data in the target memory space with the storage state being the unstable state is compressed and stored in the memory space.

Optionally, the target memory space is managed by a memory storage engine, the memory space managed by the memory storage engine includes a plurality of memory spaces, each memory space is used for storing data of a corresponding time period, and the data writing module 210 is used for acquiring a time point of the target data; determining a corresponding target memory space in an active state according to the time point; and writing the target data into the target memory space.

Optionally, the data storage in the disk is managed by a disk storage engine, and the data writing module 210 is configured to read target data in a preset time period from the disk by the disk storage engine.

Referring to fig. 8, fig. 8 is a block diagram of a data query apparatus 300 according to an embodiment of the present application, where the apparatus 300 may be a module, a program segment, or a code on an electronic device. It should be understood that the apparatus 300 corresponds to the above-mentioned embodiment of the method of fig. 5, and can perform various steps related to the embodiment of the method of fig. 5, and the specific functions of the apparatus 300 can be referred to the above description, and the detailed description is appropriately omitted here to avoid redundancy.

Optionally, the apparatus 300 comprises:

a request receiving module 310, configured to receive a data query request;

a time determining module 320, configured to determine a time range to be queried according to the data query request;

a memory determining module 330, configured to determine, according to the time range, a target memory space in a plurality of memory spaces managed by a memory storage engine, where data of a corresponding time period is stored in each memory space according to the method of any one of claims 1 to 10;

the data query module 340 is configured to query the data in the target memory space to obtain a corresponding query result.

Optionally, when the query result does not meet a preset requirement or the memory storage engine fails, instructing the disk storage engine to query according to the query request.

Optionally, the memory storage engine and the disk storage engine are respectively disposed in different devices.

It should be noted that, for the convenience and brevity of description, the specific working procedure of the above-described apparatus may refer to the corresponding procedure in the foregoing method embodiment, and the description is not repeated herein.

Embodiments of the present application provide a readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, performs the method processes performed by an electronic device in the method embodiments shown in fig. 1 or fig. 5.

The present embodiments disclose a computer program product comprising a computer program stored on a non-transitory computer readable storage medium, the computer program comprising program instructions which, when executed by a computer, enable the computer to perform the methods provided by the above-described method embodiments, for example, comprising: acquiring target data written by an upper application from a disk, and writing the target data into a target memory space which is distributed for the target data and is in an active state, wherein the active state represents that data are allowed to be written into the memory space; when the storage state conversion condition is determined to be met, setting the storage state of the target memory space to be an unstable state, wherein the unstable state represents that data are not allowed to be written into the memory space; and compressing and storing the target data in the target memory space with the storage state being an unstable state.

In summary, the embodiments of the present application provide a data storage method, an inquiry method, an apparatus, an electronic device, and a storage medium, in which data in a disk is read into a memory for storage, so that when data is inquired, the data can be inquired from the memory, and since only a small amount of data is stored in the memory, efficient inquiry of the data can be achieved, and efficiency of data inquiry is improved. And the memory space is marked in an active state and an unstable state, so that the writing of data is allowed when the memory space is in the active state, the writing of the data is not allowed when the memory space is in the unstable state, and the data is compressed and stored in time, so that the memory consumption is reduced while the quick writing is realized, and the storage requirement of more data is met.

In the embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units is only one logical division, and there may be other divisions when actually implemented, and for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection of devices or units through some communication interfaces, and may be in an electrical, mechanical or other form.

In addition, units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

Furthermore, the functional modules in the embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

In this document, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions.

The above description is only an example of the present application and is not intended to limit the scope of the present application, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method of data storage, the method comprising:

acquiring target data written by an upper application from a disk, and writing the target data into a target memory space which is distributed for the target data and is in an active state, wherein the active state represents that data are allowed to be written into the memory space;

when the storage state conversion condition is determined to be met, setting the storage state of the target memory space to be an unstable state, wherein the unstable state represents that data are not allowed to be written into the memory space;

and compressing and storing the target data in the target memory space with the storage state being an unstable state.

2. The method of claim 1, wherein the storage state transition condition is determined to be satisfied by:

and if so, determining that the storage state transition condition is met.

3. The method of claim 1, wherein the storage state transition condition is satisfied by:

if not, determining that the storage state transition condition is met.

4. The method according to claim 1, wherein compressing and storing the target data in the memory space with the storage state being the unstable state comprises:

acquiring stable data in a memory space for storing compressed data;

if so, decompressing the stable data to obtain decompressed data;

merging the decompressed data and the unstable data to obtain merged data;

and compressing the merged data to form compressed data for storage.

5. The method according to claim 1, wherein compressing and storing the target data in the target memory space with the storage state being the unstable state comprises:

acquiring stable data in a memory space for storing compressed data;

if not, compressing the unstable data to form new compressed data;

and writing the new compressed data into the memory space for storage.

6. The method according to claim 4 or 5, wherein after compressing and storing the target data in the target memory space with the storage status being the unstable status, the method further comprises:

and setting the storage state of the memory space to be a stable state.

7. The method according to claim 4 or 5, characterized in that the method further comprises:

and after determining that the data in the target memory space with the storage state being the unstable state is compressed and stored in the memory space, releasing the target memory space.

8. The method according to claim 4 or 5, wherein the data in the memory space is divided into a plurality of data blocks according to a time sequence for storage, each data block comprises a preset amount of data and a checksum of the data, and the memory space further stores index data of the plurality of data blocks, wherein the index data comprises an offset of each data block, a size of each data block and a start time of the data.

9. The method according to claim 1, wherein the target memory space is managed by a memory storage engine, the memory space managed by the memory storage engine includes a plurality of memory spaces, each memory space is used for storing data of a corresponding time period, and the writing the target data into the target memory space allocated to the memory storage engine in an active state includes:

acquiring a time point of the target data;

and writing the target data into the target memory space.

10. The method of claim 1, wherein the data storage in the disk is managed by a disk storage engine, and the obtaining target data written by the upper layer application from the disk comprises:

and reading target data in a preset time period from the disk through the disk storage engine.

11. A method for data query, the method comprising:

receiving a data query request;

determining a target memory space in a plurality of memory spaces managed by a memory storage engine according to the time range, wherein data of a corresponding time period is stored in each memory space according to the method of any one of claims 1 to 10;

12. The method according to claim 11, wherein when the query result does not satisfy a preset requirement or the memory storage engine fails, instructing a disk storage engine to query according to the query request.

13. The method of claim 12, wherein the memory storage engine and the disk storage engine are respectively disposed in different devices.

14. A data storage device, characterized in that the device comprises:

15. A data query apparatus, characterized in that the apparatus comprises:

the request receiving module is used for receiving a data query request;

16. An electronic device comprising a processor and a memory, the memory storing computer readable instructions that, when executed by the processor, perform the method of any of claims 1-10.

17. A readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1-10.