CN117667595A - Data processing method, device, equipment and storage medium - Google Patents
Data processing method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN117667595A CN117667595A CN202311701443.9A CN202311701443A CN117667595A CN 117667595 A CN117667595 A CN 117667595A CN 202311701443 A CN202311701443 A CN 202311701443A CN 117667595 A CN117667595 A CN 117667595A
- Authority
- CN
- China
- Prior art keywords
- data
- data monitoring
- monitoring points
- target
- query
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 24
- 238000012544 monitoring process Methods 0.000 claims abstract description 154
- 230000006835 compression Effects 0.000 claims abstract description 74
- 238000007906 compression Methods 0.000 claims abstract description 74
- 238000000034 method Methods 0.000 claims abstract description 22
- 238000005070 sampling Methods 0.000 claims description 42
- 238000012545 processing Methods 0.000 claims description 17
- 238000013480 data collection Methods 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000004891 communication Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a data processing method, a device, equipment and a storage medium. The method comprises the following steps: acquiring first data acquired by each of a plurality of data monitoring points; compressing the first data acquired by each of the plurality of data monitoring points to obtain compression blocks corresponding to the plurality of data monitoring points respectively; configuring a primary index based on a plurality of data monitoring points; the first-level index comprises address information of the second-level index; the secondary index is used for distributing storage space for the information of the compression block corresponding to each of the plurality of data monitoring points; the information of the compression blocks corresponding to the data monitoring points respectively comprises start-stop time information and address information of each compression block; and storing compression blocks respectively corresponding to the plurality of data monitoring points based on the primary index and the secondary index to obtain first storage data. The first data is stored structurally through the primary index and the secondary index, so that the required data can be queried rapidly when the large-scale data is faced, the query performance is improved, the time consumption of query is reduced, and the user experience is improved.
Description
Technical Field
The present disclosure relates to the field of computer technologies, and in particular, to a data processing method, apparatus, device, and storage medium.
Background
With the development of emerging technologies such as big data and cloud computing, in order to meet the requirements of large-scale data collection and storage, a data system for data collection, storage and query services has been developed. For example, in a decentralized control system (Distributed Control System, DCS), a plurality of data monitoring points collect a large amount of data, which is stored and queried by a data system. When the data system stores a large amount of data for a long time, it takes a long time to query the data from the data system, so that the query performance is poor and the user experience is poor.
Disclosure of Invention
The embodiment of the application provides a data processing method, a device, equipment and a storage medium, so as to improve query performance, reduce query time consumption and improve user experience.
In a first aspect, an embodiment of the present application provides a data processing method, including:
acquiring first data acquired by each of a plurality of data monitoring points;
compressing the first data acquired by each of the plurality of data monitoring points to obtain compression blocks corresponding to the plurality of data monitoring points;
configuring a primary index based on the plurality of data monitoring points; the first-level index comprises address information of the second-level index; the secondary index is used for distributing storage space for the information of the compression blocks corresponding to the data monitoring points respectively; the information of the compression blocks corresponding to the data monitoring points respectively comprises start-stop time information and address information of each compression block;
and storing the compressed blocks respectively corresponding to the plurality of data monitoring points based on the primary index and the secondary index to obtain first storage data.
Optionally, before compressing the first data collected by each of the plurality of data monitoring points to obtain the compressed blocks corresponding to each of the plurality of data monitoring points, the method further includes:
allocating caches for the plurality of data monitoring points from the shared memory; the cache is used for storing first data acquired by the corresponding data monitoring point;
the compressing the first data collected by each of the plurality of data monitoring points to obtain compressed blocks corresponding to each of the plurality of data monitoring points, including:
and when the cache of any data monitoring point is full, compressing the first data in the cache of any data monitoring point to obtain a compression block corresponding to any data monitoring point.
Optionally, the method further comprises:
sampling the compressed blocks respectively carried out by the plurality of data monitoring points according to a preset sampling interval to obtain sampling compressed blocks respectively corresponding to the plurality of data monitoring points; the secondary index is also used for distributing storage space for the information of the sampling compression blocks corresponding to the data monitoring points respectively; the information of the compression blocks corresponding to the data monitoring points respectively comprises start-stop time information and address information of each sampling compression block;
and storing the sampling compression block to obtain second storage data.
Optionally, the method further comprises:
acquiring a data query instruction; the data query instruction comprises a query time interval;
determining a target query location from the first stored data or the second stored data based on the query time interval and the preset sampling interval;
and executing the data query instruction in the target query position.
Optionally, the determining, based on the query time interval and the preset sampling interval, a target query location from the first stored data or the second stored data includes:
when the query time interval is smaller than the preset sampling interval, determining that the first stored data is the target query position;
and when the query time interval is greater than or equal to the preset sampling interval, determining the second stored data as the target query position.
Optionally, the data query instruction further includes a target data monitoring point to be queried and target time span information; the executing the data query instruction in the target query location includes:
determining a target primary index corresponding to the target data monitoring point in the target query position based on the target data monitoring point; the target primary index comprises address information of a target secondary index;
and acquiring a target compressed block from the plurality of compressed blocks based on the target time span information and the address information of the target secondary index.
In a second aspect, an embodiment of the present application provides a data processing apparatus, including:
the acquisition module is used for acquiring the first data acquired by each of the plurality of data monitoring points;
the compression module is used for compressing the first data acquired by each of the plurality of data monitoring points to obtain compression blocks corresponding to the plurality of data monitoring points respectively;
the configuration module is used for configuring a primary index based on the plurality of data monitoring points; the first-level index comprises address information of the second-level index; the secondary index is used for distributing storage space for the information of the compression blocks corresponding to the data monitoring points respectively; the information of the compression blocks corresponding to the data monitoring points respectively comprises start-stop time information and address information of each compression block;
and the storage module is used for storing the compression blocks respectively corresponding to the data monitoring points based on the primary index and the secondary index to obtain first storage data.
Optionally, the data processing apparatus further includes:
the distribution module is used for distributing caches for the plurality of data monitoring points from the shared memory; the cache is used for storing first data acquired by the corresponding data monitoring point;
the compression module comprises:
and the compression sub-module is used for compressing the first data in the cache of any data monitoring point when the cache of any data monitoring point is full, so as to obtain a compression block corresponding to any data monitoring point.
In a third aspect, an embodiment of the present application provides an electronic device, including: a processor, memory, system bus;
the processor and the memory are connected through the system bus;
the memory is for storing one or more programs, the one or more programs comprising instructions, which when executed by the processor, cause the processor to perform any of the implementations of the data processing methods described above.
In a fourth aspect, embodiments of the present application provide a computer-readable storage medium having instructions stored therein that, when executed on an electronic device, cause the electronic device to perform any implementation of the above-described data processing method.
From the above technical solutions, the embodiments of the present application have the following advantages:
in this embodiment of the present application, after the first data collected by each of the plurality of data monitoring points is obtained, the first data collected by each of the plurality of data monitoring points may be compressed first, so as to obtain compression blocks corresponding to each of the plurality of data monitoring points. Next, a primary index is configured based on the plurality of data monitoring points. Because the primary index comprises address information of the secondary index; the second-level index can allocate storage space for the information of the compressed blocks corresponding to the data monitoring points respectively, and the information of the compressed blocks corresponding to the data monitoring points respectively comprises start-stop time information and address information of each compressed block, so that after the compressed blocks corresponding to the data monitoring points respectively are stored based on the first-level index and the second-level index to obtain first storage data, the corresponding data monitoring points can be quickly and accurately positioned through the first-level index, and then the compressed blocks under the required query time can be quickly positioned by combining the second-level index. In this way, the first data collected by each of the plurality of data monitoring points is stored in a structured manner by configuring the primary index and the secondary index, which is helpful for quickly inquiring the required data when facing large-scale data, so that the inquiry performance is improved, the inquiry time is reduced, and the user experience is improved.
Drawings
FIG. 1 is a flowchart of a data processing method according to an embodiment of the present application;
fig. 2 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present application.
Detailed Description
As described above, in the DCS system, a plurality of data monitoring points collect a large amount of data, and store and query the data through the data system. For example, there are approximately 2 ten thousand data monitoring points in a conventional DCS system, and these data monitoring points can collect and store data at a millisecond frequency, so that the data volume of one day needs to occupy at least 3GB of storage space, and the data volume of one year needs to occupy up to 1095GB of storage space. Therefore, when the data system stores a large amount of data for a long time, it takes a long time to perform data query from the data system, and thus the query performance is poor and the user experience is poor.
To solve the above problems, an embodiment of the present application provides a data processing method, which may include: after the first data collected by each of the plurality of data monitoring points is obtained, the first data collected by each of the plurality of data monitoring points can be compressed first, so that compression blocks corresponding to each of the plurality of data monitoring points are obtained. Next, a primary index is configured based on the plurality of data monitoring points. Because the primary index comprises address information of the secondary index; the second-level index can allocate storage space for the information of the compressed blocks corresponding to the data monitoring points respectively, and the information of the compressed blocks corresponding to the data monitoring points respectively comprises start-stop time information and address information of each compressed block, so that after the compressed blocks corresponding to the data monitoring points respectively are stored based on the first-level index and the second-level index to obtain first storage data, the corresponding data monitoring points can be quickly and accurately positioned through the first-level index, and then the compressed blocks under the required query time can be quickly positioned by combining the second-level index. In this way, the first data collected by each of the plurality of data monitoring points is stored in a structured manner by configuring the primary index and the secondary index, which is helpful for quickly inquiring the required data when facing large-scale data, so that the inquiry performance is improved, the inquiry time is reduced, and the user experience is improved.
It should be noted that, the embodiment of the present application may also not limit the execution body of the data processing method, for example, the data processing method of the embodiment of the present application may be applied to a data processing device such as a terminal device or a server. The terminal device may be an electronic device such as a smart phone, a computer, a personal digital assistant (Personal Digital Assistant, PDA), a tablet computer, etc. The servers may be stand alone servers, clustered servers, or cloud servers.
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present application more clear, the technical solutions of the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is apparent that the described embodiments are only some embodiments of the present application, but not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are within the scope of the present disclosure.
Fig. 1 is a flowchart of a data processing method according to an embodiment of the present application. Referring to fig. 1, the data processing method provided in the embodiment of the present application may include:
s101: and acquiring the first data acquired by each of the plurality of data monitoring points.
S102: and compressing the first data acquired by each of the plurality of data monitoring points to obtain compression blocks corresponding to the plurality of data monitoring points.
In this embodiment of the present application, for a data acquisition speed of millisecond level, for example, data is recorded every 100 milliseconds, when a plurality of data monitoring points perform data writing operation frequently, a large number of frequent IO (input/output) operations are continued, so that the occupation of the read/write capability of the magnetic disk is too high, and therefore, when data query is performed, the query performance is rapidly reduced, and the problem of long query time consumption is caused. Based on this, the embodiments of the present application can provide a countermeasure, which is described in detail below.
In one possible implementation manner, before the first data is compressed, that is, in step S102, a buffer may be allocated to a plurality of data monitoring points from the shared memory, where the buffer may be used to store the first data collected by the corresponding data monitoring points.
Correspondingly, the step S102 may include: and when the cache of any data monitoring point is full, compressing the first data in the cache of any data monitoring point to obtain a compression block corresponding to any data monitoring point. In practical application, a buffer memory for 10 minutes can be allocated to each data monitoring point from the shared memory, and correspondingly, after the buffer memory of any data monitoring point is full of 10 minutes, the buffer memory is compressed into a corresponding compressed block at one time. Therefore, the original IO operation on the disk can be changed into the storage to be stored in the shared memory, and the IO operation is compressed into the disk at a time which is proper instead of being directly written into the disk, so that the problem that the read-write capacity of the disk is too high can be solved, the time consumption for inquiring is further reduced, and the inquiring performance is further improved.
S103: the primary index is configured based on a plurality of data monitoring points.
Wherein the primary index includes address information of the secondary index. The secondary index can be used for distributing storage space for the information of the compression block corresponding to each of the plurality of data monitoring points; the information of the compression blocks corresponding to the data monitoring points respectively comprises start-stop time information and address information of each compression block. The address information of the secondary index can be embodied as the first address of the secondary index; the address information of the compressed block is used to point to the corresponding compressed block data. For ease of understanding, the structured storage formed by configuring the primary index and the secondary index is illustrated in table 1 below.
TABLE 1
In table 1 above, n is used to represent the number of data monitoring points, and m is used to represent the number of compressed blocks. The number of the compression blocks can be calculated by dividing the start-stop time information of the first data whole by the start-stop time information of each compression block. Therefore, the corresponding data monitoring points can be quickly and accurately positioned through the primary index, and the compression blocks under the required query time can be quickly positioned by combining the secondary index, so that the query performance is improved, the query time consumption is reduced, and the user experience is improved.
S104: and storing compression blocks respectively corresponding to the plurality of data monitoring points based on the primary index and the secondary index to obtain first storage data.
In addition, in the embodiment of the application, because the data volume to be queried is larger due to the larger time span of the query, and the query efficiency is affected, in order to further shorten the query time consumption, the data can be sampled twice, so that the query time consumption is shortened by reducing the data volume of the query.
Based on this, in the embodiment of the present application, the data processing may further include: sampling the compression blocks respectively carried out by the data monitoring points according to a preset sampling interval to obtain sampling compression blocks respectively corresponding to the data monitoring points; the secondary index is also used for distributing storage space for the information of the sampling compression block corresponding to the data monitoring points respectively; the information of the compression blocks corresponding to the data monitoring points respectively comprises start-stop time information and address information of each sampling compression block; and storing the sampling compression block to obtain second storage data. In practical application, the preset sampling interval may be 1 minute, that is, the compressed block is sampled twice every 1 minute, and then stored to obtain second storage data, so that the number of the second storage data is reduced, and the data amount of querying from the second storage data is reduced, thereby reducing the time consumption of querying. It should be noted that, the configuration of the secondary index and the storage of the sample compression block may be referred to the relevant content of step S103 and step S104, which are not described herein.
Further, based on the first stored data and the second stored data obtained by the secondary sampling, in the embodiment of the application, a data query method may also be provided, so as to improve query efficiency. Corresponding to this, the above data processing method may further include: acquiring a data query instruction; the data query instruction includes a query time interval; determining a target query position from the first stored data or the second stored data based on the query time interval and the preset sampling interval; a data query instruction is executed in the target query location. Therefore, by executing the data query instruction from different query positions, the data query performance is improved, so that the time consumption of query is reduced, and the user experience is improved.
Specifically, the process of determining the target query location may include: when the query time interval is smaller than the preset sampling interval, determining the first stored data as a target query position; and when the query time interval is greater than or equal to the preset sampling interval, determining the second storage data as a target query position. As mentioned above, the preset sampling interval is 1 minute, so when the inquiry time interval is less than 1 minute, the first stored data with a larger data amount can be used as the target inquiry position and inquired from the target inquiry position, and when the inquiry time interval is greater than or equal to 1 minute, the second stored data with a smaller data amount can be used as the target inquiry position and inquired from the target inquiry position, and because the data amount of the second stored data is small, the efficiency of inquiring the data from the second stored data is high and the time consumption is low.
In addition, the data query instruction may further include target data monitoring points to be queried and target time span information. In this regard, the above-described process of executing the data query instruction may include: determining a target primary index corresponding to the target data monitoring point in the target query position based on the target data monitoring point; the target primary index comprises address information of the target secondary index; the target compressed block is acquired from the plurality of compressed blocks based on the target time span information and the address information of the target secondary index. Because the primary index is configured based on a plurality of data monitoring points, the corresponding target primary index can be queried through the target data monitoring points in the data query instruction. Then, the target primary index comprises address information of the target secondary index, and the secondary index can be used for distributing storage space for information of compression blocks corresponding to the data monitoring points respectively, wherein the information of the compression blocks comprises start-stop time information and address information of each compression block, so that the target compression block can be queried correspondingly based on the target time span information and the address information of the target secondary index, and a data query instruction is completed.
Based on the above description of steps S101 to S104, in this embodiment of the present application, after the first data collected by each of the plurality of data monitoring points is obtained, the first data collected by each of the plurality of data monitoring points may be compressed first, so as to obtain compressed blocks corresponding to each of the plurality of data monitoring points. Next, a primary index is configured based on the plurality of data monitoring points. Because the primary index comprises address information of the secondary index; the second-level index can allocate storage space for the information of the compressed blocks corresponding to the data monitoring points respectively, and the information of the compressed blocks corresponding to the data monitoring points respectively comprises start-stop time information and address information of each compressed block, so that after the compressed blocks corresponding to the data monitoring points respectively are stored based on the first-level index and the second-level index to obtain first storage data, the corresponding data monitoring points can be quickly and accurately positioned through the first-level index, and then the compressed blocks under the required query time can be quickly positioned by combining the second-level index. In this way, the first data collected by each of the plurality of data monitoring points is stored in a structured manner by configuring the primary index and the secondary index, which is helpful for quickly inquiring the required data when facing large-scale data, so that the inquiry performance is improved, the inquiry time is reduced, and the user experience is improved.
Further, based on the data processing method provided in the above embodiment, the embodiment of the present application may also provide a data processing apparatus. The data processing apparatus will be described below with reference to the embodiments and drawings, respectively.
Fig. 2 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present application. Referring to fig. 2, a data processing apparatus 200 provided in an embodiment of the present application includes:
a first obtaining module 201, configured to obtain first data collected by each of a plurality of data monitoring points;
the compression module 202 is configured to compress the first data collected by each of the plurality of data monitoring points to obtain compression blocks corresponding to each of the plurality of data monitoring points;
a configuration module 203, configured to configure a primary index based on the plurality of data monitoring points; the first-level index comprises address information of the second-level index; the secondary index is used for distributing storage space for the information of the compression blocks corresponding to the data monitoring points respectively; the information of the compression blocks corresponding to the data monitoring points respectively comprises start-stop time information and address information of each compression block;
the first storage module 204 is configured to store compressed blocks corresponding to the plurality of data monitoring points respectively based on the primary index and the secondary index, so as to obtain first storage data.
As an embodiment, the data processing apparatus 200 further includes:
the distribution module is used for distributing caches for the plurality of data monitoring points from the shared memory; the cache is used for storing first data acquired by the corresponding data monitoring point;
the compression module 202 includes:
and the compression sub-module is used for compressing the first data in the cache of any data monitoring point when the cache of any data monitoring point is full, so as to obtain a compression block corresponding to any data monitoring point.
As an embodiment, the data processing apparatus 200 further includes:
the sampling module is used for sampling the compressed blocks respectively carried out by the plurality of data monitoring points according to a preset sampling interval to obtain sampling compressed blocks respectively corresponding to the plurality of data monitoring points; the secondary index is also used for distributing storage space for the information of the sampling compression blocks corresponding to the data monitoring points respectively; the information of the compression blocks corresponding to the data monitoring points respectively comprises start-stop time information and address information of each sampling compression block;
and the second storage module is used for storing the sampling compression block to obtain second storage data.
As an embodiment, the data processing apparatus 200 further includes:
the second acquisition module is used for acquiring a data query instruction; the data query instruction comprises a query time interval;
the first determining module is used for determining a target query position from the first stored data or the second stored data based on the query time interval and the preset sampling interval;
and the execution module is used for executing the data query instruction in the target query position.
As an embodiment, the first determining module includes:
a first determining sub-module, configured to determine, when the query time interval is smaller than the preset sampling interval, that the first stored data is the target query location;
and the second determining submodule is used for determining the second stored data as the target query position when the query time interval is greater than or equal to the preset sampling interval.
As one implementation mode, the data query instruction further comprises a target data monitoring point to be queried and target time span information; the execution module comprises:
the second determining module is used for determining a target primary index corresponding to the target data monitoring point in the target query position based on the target data monitoring point; the target primary index comprises address information of a target secondary index;
and a third acquisition module, configured to acquire a target compressed block from the plurality of compressed blocks based on the target time span information and the address information of the target secondary index.
Further, the embodiment of the application also provides an electronic device, which comprises: a processor, memory, system bus;
the processor and the memory are connected through the system bus;
the memory is for storing one or more programs, the one or more programs comprising instructions, which when executed by the processor, cause the processor to perform any of the implementations of the data processing methods described above.
Further, the embodiment of the application also provides a computer readable storage medium, in which instructions are stored, which when executed on an electronic device, cause the terminal device to execute any implementation manner of the data processing method.
From the above description of embodiments, it will be apparent to those skilled in the art that all or part of the steps of the above described example methods may be implemented in software plus necessary general purpose hardware platforms. Based on such understanding, the technical solutions of the present application may be embodied essentially or in a part contributing to the prior art in the form of a software product, which may be stored in a storage medium, such as a ROM/RAM, a magnetic disk, an optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, a server, or a network communication device such as a media gateway, etc.) to perform the methods described in the embodiments or some parts of the embodiments of the present application. It should be noted that, in the present description, each embodiment is described in a progressive manner, and each embodiment is mainly described in a different manner from other embodiments, and identical and similar parts between the embodiments are all enough to refer to each other.
For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant points refer to the description of the method section.
It is further noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (10)
1. A method of data processing, comprising:
acquiring first data acquired by each of a plurality of data monitoring points;
compressing the first data acquired by each of the plurality of data monitoring points to obtain compression blocks corresponding to the plurality of data monitoring points;
configuring a primary index based on the plurality of data monitoring points; the first-level index comprises address information of the second-level index; the secondary index is used for distributing storage space for the information of the compression blocks corresponding to the data monitoring points respectively; the information of the compression blocks corresponding to the data monitoring points respectively comprises start-stop time information and address information of each compression block;
and storing the compressed blocks respectively corresponding to the plurality of data monitoring points based on the primary index and the secondary index to obtain first storage data.
2. The method for processing data according to claim 1, wherein before compressing the first data collected by each of the plurality of data monitoring points to obtain compressed blocks corresponding to each of the plurality of data monitoring points, the method further comprises:
allocating caches for the plurality of data monitoring points from the shared memory; the cache is used for storing first data acquired by the corresponding data monitoring point;
the compressing the first data collected by each of the plurality of data monitoring points to obtain compressed blocks corresponding to each of the plurality of data monitoring points, including:
and when the cache of any data monitoring point is full, compressing the first data in the cache of any data monitoring point to obtain a compression block corresponding to any data monitoring point.
3. The data processing method of claim 1, wherein the method further comprises:
sampling the compressed blocks respectively carried out by the plurality of data monitoring points according to a preset sampling interval to obtain sampling compressed blocks respectively corresponding to the plurality of data monitoring points; the secondary index is also used for distributing storage space for the information of the sampling compression blocks corresponding to the data monitoring points respectively; the information of the compression blocks corresponding to the data monitoring points respectively comprises start-stop time information and address information of each sampling compression block;
and storing the sampling compression block to obtain second storage data.
4. A data processing method according to claim 3, characterized in that the method further comprises:
acquiring a data query instruction; the data query instruction comprises a query time interval;
determining a target query location from the first stored data or the second stored data based on the query time interval and the preset sampling interval;
and executing the data query instruction in the target query position.
5. The data processing method of claim 4, wherein the determining a target query location from the first stored data or the second stored data based on the query time interval and the preset sampling interval comprises:
when the query time interval is smaller than the preset sampling interval, determining that the first stored data is the target query position;
and when the query time interval is greater than or equal to the preset sampling interval, determining the second stored data as the target query position.
6. The data processing method of claim 4, wherein the data query instruction further comprises target data monitoring points to be queried and target time span information; the executing the data query instruction in the target query location includes:
determining a target primary index corresponding to the target data monitoring point in the target query position based on the target data monitoring point; the target primary index comprises address information of a target secondary index;
and acquiring a target compressed block from the plurality of compressed blocks based on the target time span information and the address information of the target secondary index.
7. A data processing apparatus, comprising:
the first acquisition module is used for acquiring first data acquired by each of the plurality of data monitoring points;
the compression module is used for compressing the first data acquired by each of the plurality of data monitoring points to obtain compression blocks corresponding to the plurality of data monitoring points respectively;
the configuration module is used for configuring a primary index based on the plurality of data monitoring points; the first-level index comprises address information of the second-level index; the secondary index is used for distributing storage space for the information of the compression blocks corresponding to the data monitoring points respectively; the information of the compression blocks corresponding to the data monitoring points respectively comprises start-stop time information and address information of each compression block;
and the first storage module is used for storing the compressed blocks respectively corresponding to the plurality of data monitoring points based on the primary index and the secondary index to obtain first storage data.
8. The data processing apparatus of claim 7, wherein the data processing apparatus further comprises:
the distribution module is used for distributing caches for the plurality of data monitoring points from the shared memory; the cache is used for storing first data acquired by the corresponding data monitoring point;
the compression module comprises:
and the compression sub-module is used for compressing the first data in the cache of any data monitoring point when the cache of any data monitoring point is full, so as to obtain a compression block corresponding to any data monitoring point.
9. An electronic device, the device comprising: a processor, memory, system bus;
the processor and the memory are connected through the system bus;
the memory is for storing one or more programs, the one or more programs comprising instructions, which when executed by the processor, cause the processor to perform the data processing method of any of claims 1 to 6.
10. A computer readable storage medium, characterized in that the computer readable storage medium has stored therein instructions, which when run on an electronic device, cause the electronic device to perform the data processing method of any of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311701443.9A CN117667595A (en) | 2023-12-12 | 2023-12-12 | Data processing method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311701443.9A CN117667595A (en) | 2023-12-12 | 2023-12-12 | Data processing method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117667595A true CN117667595A (en) | 2024-03-08 |
Family
ID=90082405
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311701443.9A Pending CN117667595A (en) | 2023-12-12 | 2023-12-12 | Data processing method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117667595A (en) |
-
2023
- 2023-12-12 CN CN202311701443.9A patent/CN117667595A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110149803B (en) | Data storage method, system and terminal equipment | |
| CN106407207B (en) | Real-time newly-added data updating method and device | |
| CN110555001B (en) | Data processing method, device, terminal and medium | |
| CN111382182A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN111143158A (en) | Monitoring data real-time storage method and system, electronic equipment and storage medium | |
| CN110851474A (en) | Data query method, database middleware, data query device and storage medium | |
| CN107391769B (en) | Index query method and device | |
| CN111083008A (en) | Nginx-based traffic collection and analysis method | |
| CN113297269A (en) | Data query method and device | |
| CN115048053A (en) | Data storage method and device and electronic equipment | |
| CN114398520A (en) | Data retrieval method, system, device, electronic equipment and storage medium | |
| CN113849499A (en) | Data query method and device, storage medium and electronic device | |
| CN111625600B (en) | Data storage processing method, system, computer equipment and storage medium | |
| CN113297245A (en) | Method and device for acquiring execution information | |
| CN117667595A (en) | Data processing method, device, equipment and storage medium | |
| CN108616583B (en) | Storage space allocation method based on computer cloud | |
| CN115292280A (en) | Cross-region data scheduling method, device, equipment and storage medium | |
| CN117171209A (en) | Cache data cleaning method and device, storage medium and electronic equipment | |
| CN114416731A (en) | Data storage method, data reading method, data storage device, electronic device and medium | |
| CN112817987A (en) | Method, device, equipment and storage medium for accessing distributed storage cluster | |
| CN107679093B (en) | Data query method and device | |
| CN116150212B (en) | Data processing method and device | |
| CN111143711A (en) | Object searching method and system | |
| CN117891865A (en) | Multi-source heterogeneous data acquisition method, system, computer equipment and storage medium | |
| CN113722286A (en) | Space-time data compression method, device, server and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |