CN114416731A - Data storage method, data reading method, data storage device, electronic device and medium - Google Patents
Data storage method, data reading method, data storage device, electronic device and medium Download PDFInfo
- Publication number
- CN114416731A CN114416731A CN202111643360.XA CN202111643360A CN114416731A CN 114416731 A CN114416731 A CN 114416731A CN 202111643360 A CN202111643360 A CN 202111643360A CN 114416731 A CN114416731 A CN 114416731A
- Authority
- CN
- China
- Prior art keywords
- data
- storage
- information
- index information
- address
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the application discloses a data storage method, a data reading device, electronic equipment and a storage medium. The data storage method comprises the following steps: determining the variation amplitude of the data; when the change amplitude exceeds a set threshold value, storing the data; establishing index information of the data according to the identification and the storage position of the data, wherein the index information at least comprises: the identification and location information of the storage location. Therefore, data with substantial changes can be selectively recorded, complicated repeated data recording is replaced, and storage space is saved; and the index information can realize the quick retrieval of specific data, and improve the utilization and processing efficiency of the data.
Description
Technical Field
The present application relates to the field of software engineering technologies, and in particular, to a data storage method, a data reading method, an apparatus, an electronic device, and a storage medium.
Background
The industrial Control field relates to communication and data storage of a large number of devices, the data change frequency of most devices (typically, Control devices such as a DCS (Distributed Control System), a PLC (Programmable Logic Controller), and the like) is relatively slow and is approximately at a second level, and the System data storage generally adopts a time and characteristic data recording mode one by one, so that the data read-write quantity is large, the memory space is occupied, the required storage space is large, and the waste of the storage space can be caused; and the repeated data records are many, which brings calculation consumption when the data is used or utilized.
Disclosure of Invention
In view of the above, embodiments of the present application are intended to provide a data storage method, a data reading method, a device, an electronic device, and a storage medium.
The technical scheme of the application is realized as follows:
in a first aspect, an embodiment of the present application provides a data storage method, including:
determining the variation amplitude of the data;
when the change amplitude exceeds a set threshold value, storing the data;
establishing index information of the data according to the identification and the storage position of the data, wherein the index information at least comprises: the identification and location information of the storage location.
Based on the above scheme, the method further comprises:
determining a time of change of the data;
determining a storage node according to the change amplitude and the change time, wherein the storage node is used for storing the data and the change time; wherein the location information indicates the storage node.
Based on the above scheme, the method further comprises:
storing the index information by adopting a linked list, wherein the linked list comprises a plurality of nodes; each node comprises at least two parts: a node address area and an index information area;
the node address area is used for storing the address information of the node; the index information area is used for storing the index information of the data; the node address area stores a first address and a second address; the first address is the address of the previous adjacent node of the current node; the second address is the address of the next adjacent node of the current node; the index information area contains index information of a plurality of data. Based on the above scheme, when the variation amplitude exceeds a set threshold, the storing the data includes:
storing the data by adopting a data group, wherein the data group at least comprises two data; and storing index information of each data in one data group by adopting the same linked list.
In a second aspect, an embodiment of the present application provides a data reading method, including:
receiving a data reading request, wherein the data reading request at least comprises an identifier of data to be read;
inquiring index information according to the identification to obtain position information corresponding to the identification,
and reading the data to be read according to the storage position indicated by the position information.
Based on the above scheme, the querying index information according to the identifier to obtain location information corresponding to the identifier includes:
the index information is stored by adopting a linked list, wherein the linked list comprises a plurality of nodes; each node comprises at least two parts: a node address area and an index information area;
the node address area is used for storing the address information of the node; the index information area is used for storing the index information of the data to be read; the node address area stores a first address and a second address; the first address is the address of the previous adjacent node of the current node; the second address is the address of the next adjacent node of the current node; the index information area contains index information of a plurality of data to be read.
Based on the above scheme, the data to be read includes: the data to be read is stored by adopting a data group, wherein the data group at least comprises two data to be read; and the index information of each data to be read included in one data group is stored by adopting the same linked list.
Based on the above scheme, the method further comprises:
querying the linked list according to the data reading request to obtain a linked list corresponding to the data to be read, wherein the data reading request comprises type information, and the type information indicates the data type of the data to be read;
inquiring a corresponding linked list according to the identification to obtain index information corresponding to the data to be read;
and obtaining a storage position corresponding to the data to be read according to the corresponding index information.
In a third aspect, an embodiment of the present application provides a data storage device, including:
the first determining module is used for determining the variation amplitude of the data;
the first storage module is used for storing the data when the change amplitude exceeds a set threshold;
an establishing module, configured to establish index information of the data according to the identifier and the storage location of the data, where the index information at least includes: the identification and location information of the storage location.
In a fourth aspect, an embodiment of the present application provides a data reading apparatus, including:
the device comprises a receiving module, a processing module and a processing module, wherein the receiving module is used for receiving a data reading request, and the data reading request at least comprises an identifier of data to be read;
the query module is used for querying index information according to the identification to obtain position information corresponding to the identification;
and the reading module is used for reading the data to be read according to the storage position indicated by the position information.
In a fifth aspect, an embodiment of the present application provides an electronic device, including:
a memory storing computer readable instructions;
and the processor is connected with the memory and used for realizing the method provided by any technical scheme of the first aspect or the second aspect by executing the computer-executable instructions stored on the memory.
In a sixth aspect, an embodiment of the present application provides a computer storage medium, where the computer storage medium stores computer-executable instructions, and after the computer-executable instructions are executed, the method provided in any technical solution of the first aspect or the second aspect can be implemented.
According to the data storage method, the data reading method and device, the electronic equipment and the storage medium, data needing to be stored can be determined according to the variation amplitude of the data, and when the variation amplitude exceeds the set threshold value, the data are stored, so that the data with enough variation amplitude can be selectively stored, repeated data or data with small amplitude difference can be reduced, and the storage space can be saved; establishing index information of the data according to the identification and the storage position of the data, wherein the index information at least comprises: the identifier and the position information of the storage position can quickly retrieve the specific data through the index information, so that the data utilization and processing efficiency is improved.
Drawings
Fig. 1 is a schematic flowchart of a data storage method according to an embodiment of the present application;
fig. 2 is a schematic flowchart of a data reading method according to an embodiment of the present application;
fig. 3 is a schematic diagram of a format of storage information of a file header of an index file according to an embodiment of the present application;
fig. 4 is a schematic diagram of a storage information format of an index segment of an index file according to an embodiment of the present application;
fig. 5 is a schematic diagram of a storage information format of a data file header according to an embodiment of the present application;
fig. 6 is a schematic diagram illustrating a storage information format of a data segment of a data file according to an embodiment of the present application;
fig. 7 is a schematic diagram illustrating a space allocation scheme and storage rules of related information items of a file header of an index file according to an embodiment of the present application;
fig. 8 is a schematic diagram illustrating a space allocation scheme and storage rules of related information items of an index segment of an index file according to an embodiment of the present application;
fig. 9 is a schematic diagram illustrating a space allocation scheme and storage rules of related information items of a data file header according to an embodiment of the present application;
FIG. 10 is a schematic diagram illustrating a space allocation scheme and storage rules of related information items of a data segment of a data file according to an embodiment of the present application;
fig. 11 is a schematic structural diagram of a data storage device according to an embodiment of the present application;
fig. 12 is a schematic structural diagram of a data reading apparatus according to an embodiment of the present application;
fig. 13 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
So that the manner in which the features and aspects of the present application can be understood in detail, a more particular description of the invention, briefly summarized above, may be had by reference to embodiments, some of which are illustrated in the appended drawings.
As shown in fig. 1, an embodiment of the present application provides a data storage method, including:
step S110: determining the variation amplitude of the data;
step S120: when the change amplitude exceeds a set threshold value, storing the data;
step S130: establishing index information of the data according to the identification and the storage position of the data, wherein the index information at least comprises: the identification and location information of the storage location.
The data storage method can be realized by the electronic equipment which is provided with the database, and the database can be distributed at the cloud end, so that the data storage method can be executed by a cloud end server.
The data may be results obtained by experiments, calculations, measurements, and the like. In the embodiment of the application, data can be acquired through control equipment (such as DCS, PLC and the like). For example, in the field of industrial control, a large amount of equipment communication and data storage are involved, and a system performs real-time data acquisition through control equipment such as a DCS (distributed control system), a PLC (programmable logic controller) and the like to feed back the running state of the equipment.
In some embodiments, the data includes, but is not limited to: status data of various monitored devices, illustratively, operational status data of industrial devices,
in other embodiments, the data may further include: control parameter values of the control device. Illustratively, control values of the PLC device.
The variation amplitude may be: the rate at which the data is increased or decreased compared to the last data stored can indicate the degree to which the data has changed.
The set threshold may be a critical value at which the data substantially changes, that is, when the change amplitude exceeds the set threshold, the data is considered to substantially change from the stored data, and the data needs to be stored and recorded. When the change amplitude does not exceed the set threshold, the data is considered to have no substantial change relative to the stored data, and the data does not need to be stored and recorded.
The set threshold value can be manually set according to actual conditions. The threshold is set too small so that even data that changes only slightly is stored and recorded, and the data does not reflect a sufficient influence. For example, the current running state of the device is not changed, so that the storage space is wasted, and the effects of reducing repeated recording data and saving the storage space are difficult to realize; an excessively large set threshold value may not store and record some critical data, for example, the current operation state of the device changes, so that it is difficult to monitor and store the change of the recorded data in real time.
The identification is used for distinguishing different data, and the identification may include: name, and/or serial number.
In practical application, a plurality of data need to be monitored and recorded in real time, so that a lot of data may change in the same time period, and it is difficult to quickly and accurately determine the required data according to information such as storage time or change time of the data. Therefore, the required data can be determined quickly and accurately according to the data identification in the application. The storage location may be a location where the data is stored.
The index information is used to read the data, including but not limited to a query index of stored data. For example, the index information may include an identifier and a storage location of the data, and the data to be read can be quickly located through the index information when the data is read. The index information may further include other information capable of indicating the data, such as the type and amount of the data, and information related to the index information itself, such as the creation time, name, and amount of the data of the index information.
The location information of the storage location is used to indicate the storage location. In this embodiment of the present application, the location information may be an absolute location, a relative location, or a customized location format, and the storage location may be obtained according to the location information.
In some embodiments, the index information of the data may also be established according to the storage time or the change time of the data.
The storage time may be a time when the data storage is completed.
The change time may be a time when the data is changed.
In some embodiments, the index information may further include time information of a storage time or time information of a variation time.
The time information of the storage time is used for indicating the storage time, and the time information of the change time is used for indicating the change time. In this embodiment of the application, the time information of the storage time and the time information of the change time may be absolute time, relative time, or a custom time format, and the storage time may be obtained according to the time information of the storage time, or the change time may be obtained according to the time information of the change time.
In an embodiment of the present application, the method further includes:
determining a time of change of the data;
determining a storage node according to the change amplitude and the change time, wherein the storage node is used for storing the data and the change time; wherein the location information indicates the storage node.
The change time may be a time when the data is changed.
The storage node may be a server or a device for storing data.
In an embodiment of the present application, the method further includes:
storing the index information by adopting a linked list, wherein the linked list comprises a plurality of nodes; each node comprises at least two parts: a node address area and an index information area;
the node address area is used for storing the address information of the node; the index information area is used for storing the index information of the data; the node address area stores a first address and a second address; the first address is the address of the previous adjacent node of the current node; the second address is the address of the next adjacent node of the current node; the index information area contains index information of a plurality of data.
The linked list may be a single linked list, a double linked list, and a circular linked list. In the embodiment of the application, the data is selectively stored and recorded according to the variation amplitude of the data, so that the storage time and the variation amplitude of the data are uncertain, unlike the continuous storage according to a general sequence (time sequence), and the index information is stored in a linked list form, so that the storage space is saved, and the data is convenient to store and read.
The first address may be an uplink address.
The second address may be a downlink address.
The nodes may be dynamically generated while the aforementioned devices are running. The node is configured to record information, such as information of the data or the index information, and may be configured to describe attributes of the data or the index information, such as an address, a type, a data amount, a name, and the like.
In this embodiment of the present application, when the variation amplitude exceeds a set threshold, the storing the data includes:
storing the data by adopting a data group, wherein the data group at least comprises two data; and storing index information of each data in one data group by adopting the same linked list.
As shown in fig. 2, an embodiment of the present application provides a data reading method, including:
step S210: receiving a data reading request, wherein the data reading request at least comprises an identifier of data to be read;
step S220: inquiring index information according to the identification to obtain position information corresponding to the identification;
step S230: and reading the data to be read according to the storage position indicated by the position information.
The data reading method can be realized by a database, and the database can be distributed in the cloud.
The data reading request includes, but is not limited to, a computer instruction or a message, and the data reading request includes information of the data to be read, such as information of storage time, data type, data amount or name of the data to be read.
The data to be read may be results obtained and stored through experiments, calculations, measurements, and the like. In the embodiment of the application, the data to be read can be acquired and stored through control equipment (such as DCS, PLC and the like). For example, in the field of industrial control, a large amount of equipment communication and data storage are involved, and a system performs real-time data acquisition through control equipment such as a DCS (distributed control system), a PLC (programmable logic controller) and the like to feed back the running state of the equipment.
The identifier is used for distinguishing different data to be read, and the identifier may include: name, and/or serial number.
In some embodiments, the data to be read includes, but is not limited to: status data of various monitored devices, illustratively, operational status data of industrial devices,
in other embodiments, the data to be read may further include: control parameter values of the control device. Illustratively, control values of the PLC device.
In some embodiments, the data read request may further include time information of the storage time or time information of the variation time.
The storage time may be a time for completing the storage of the data to be read.
The change time may be a time when the data to be read changes. The time information of the storage time is used for indicating the storage time of the data to be read, and the time information of the change time is used for indicating the change time of the data to be read. In this embodiment of the application, the time information of the storage time or the time information of the change time may be in an absolute time format, a relative time format, or a user-defined time format, and the storage time of the data to be read may be obtained according to the time information of the storage time, or the change time of the data to be read may be obtained according to the time information of the change time.
The index information is used for reading the data to be read, including but not limited to a query index of the stored data. For example, the index information may include a name and a storage location of the data to be read, and the data to be read can be quickly located through the index information when data reading is performed. The index information may further include other information capable of indicating the data to be read, such as information about the type and the amount of the data to be read, and information related to the index information itself, such as information about the creation time, the name, and the amount of the data of the index information.
In some embodiments, the index information may further include time information of a storage time or time information of a variation time.
The position information is used for indicating the storage position of the data to be read. In this embodiment of the application, the location information may be an absolute location, a relative location, or a user-defined location format, and the storage location of the data to be read can be obtained according to the location information.
The storage location may be a location where the data to be read is stored.
In this embodiment of the present application, the querying index information according to the identifier to obtain location information corresponding to the identifier includes:
the index information is stored by adopting a linked list, wherein the linked list comprises a plurality of nodes; each node comprises at least two parts: a node address area and an index information area;
the node address area is used for storing the address information of the node; the index information area is used for storing the index information of the data to be read; the node address area stores a first address and a second address; the first address is the address of the previous adjacent node of the current node; the second address is the address of the next adjacent node of the current node; the index information area contains index information of a plurality of data to be read.
The linked list may be a single linked list, a double linked list, and a circular linked list. In the embodiment of the application, the data is selectively stored and recorded according to the variation amplitude of the data, so that the storage time and the variation amplitude of the data are uncertain, unlike the continuous storage according to a general sequence (time sequence), and the index information is stored in a linked list form, so that the storage space is saved, and the data is convenient to store and read.
The first address may be an uplink address.
The second address may be a downlink address.
The nodes may be dynamically generated while the aforementioned devices are running. The node is configured to record information, such as information of the data or the index information, and may be configured to describe attributes of the data or the index information, such as a type, a data amount, a name, and the like.
In this embodiment of the present application, the data to be read includes:
the data to be read is stored by adopting a data group, wherein the data group at least comprises two data to be read; and the index information of each data to be read included in one data group is stored by adopting the same linked list.
In an embodiment of the present application, the method further includes:
querying the linked list according to the data reading request to obtain a linked list corresponding to the data to be read, wherein the data reading request comprises type information, and the type information indicates the data type of the data to be read;
inquiring a corresponding linked list according to the identification to obtain index information corresponding to the data to be read;
and obtaining a storage position corresponding to the data to be read according to the corresponding index information.
The type information is used for indicating the data type of the data to be read. In this embodiment of the application, the type information may be an attribute of data or a custom type format, and the data type of the data to be read can be obtained according to the type information.
Based on the foregoing embodiments, embodiments of the present application provide a data storage structure, where the storage structure can perform characteristic storage according to a data change state, thereby avoiding waste of storage space and improving data utilization and processing speed. The storage structure is composed of a data file and an index file, effective data recording can be achieved, the data volume is remarkably reduced, and rapid data retrieval can be carried out through the index file. The data file is used for storing numerical information of a variable, the variable is one of the data, and the index file is one of the index information. Specifically, as shown in fig. 3 and 4, the index file is composed of a file header and a plurality of index segments, each index segment corresponds to one of the nodes, and includes at least uplink and downlink addresses and basic index information: name and storage location of the data, the name being one of the aforementioned identities.
The index file mainly has the functions of establishing a variable data position index and associating the storage position information of the variable in the data file. The index file format is defined as an index file header and index segments, the number of which is determined by the amount of recorded data. Each index segment does not store actual data, and only stores data position information of related variables in the data file, so that rapid positioning is realized. When data reading and utilization are carried out, forward or backward data searching is realized through a linked list method, wherein the linked list is one of the linked lists.
In some embodiments, the index file header and the index segment may be in a fixed-length format or a variable-length format. When the index file header and the index segment are in a fixed-length format, the length is fixed, and data analysis is facilitated; when the index file header and the index segment are in a variable length format, the length is not fixed, and data analysis is not convenient.
As shown in fig. 3, a storage information format of an index file header is provided for the embodiment of the present application. The index file header stores the basic information of the index file, such as file name, creation time, file size, time format, total number of storage variables, recording time of the first data and the like.
The storage information format of the index file header shown in fig. 3 is only for illustration and is not intended to limit the present application. The storage information format of the index file header can be a custom format.
As shown in fig. 4, a storage information format of an index segment of an index file is provided in an embodiment of the present application. Each index segment is provided with a segment head with a fixed length and used for storing basic storage rule information as a basis for callback analysis of the data file. The basic storage rule information includes but is not limited to key information such as variable type, variable quantity, linked list address, name and the like. The attribute mark can be established by defining in advance to fix the preset storage rule of data classification. The index segment data part records the variable names in the segments and the index information of the corresponding storage addresses of the variable names in the associated data files.
The variable quantity item is arranged in the index segment head, and the variable quantity item is used for counting the variable quantity of the data file index information contained in the index segment and can be used for effectively analyzing the subsequent name and index address information. The name and its index data segment number may be selected in the form of an appropriate interclass separation, typically a fixed length and a characteristic separation flag. When stored at a fixed length, the name unit storage space allocation is determined by the actual storage name length requirement. The index data segment number unit storage space allocation is determined by the maximum data segment number in the corresponding data file. When the data is stored in the form of a feature separation mark, the data can be separated by adding the feature mark after the number of a single name/index data segment. In order to facilitate data storage and call back, the names in the segments and the segment numbers of the index data are preferably stored in a classified and continuous manner. The variable data address index should be determined by appropriate calculation rules.
The storage information format of the index segment of the index file shown in fig. 4 is only for illustration and is not intended to limit the present application. The storage information format of the index segment of the index file can be a custom format.
The data file mainly stores numerical information of variables. The data file format is defined as a data file header and data segments, and the number of the data segments is determined by the amount of recorded data.
In some embodiments, the data file header and the data segment may be in a fixed-length format or a variable-length format. When the data file header and the data segment are in a fixed length format, the length is fixed, and data analysis is convenient; when the data file header and the data segment are in a variable length format, the length is not fixed, and data analysis is not convenient.
As shown in fig. 5, a storage information format of a data file header provided in an embodiment of the present application is shown. The data file header part stores basic information of the data file, such as file name, creation time, file data amount, time format, total data segment number, first data recording time and the like.
The format of the storage information of the header of the data file shown in fig. 5 is only for illustration and is not intended to limit the present application. The storage information format of the data file header can be a custom format.
As shown in fig. 6, a storage information format of a data segment of a data file is provided for the embodiment of the present application. Each data segment has a segment head with fixed length for recording the variable type, data quantity, name and data recording start time information stored in the segment memory. The data section stores variable value information on a time scale.
The storage information format of the data file data segment shown in fig. 6 is only for illustration and is not intended to limit the present application. The storage information format of the data file data segment can be a custom format.
The compact data storage structure and the method provided by the application carry out data recording through a storage mode of searching necessary data by a linked list, and do not carry out indifference recording one by one according to a specific time period. The linked list storage mode carries out data storage necessity judgment before recording: calculating the data change amplitude, and storing the data change amplitude when the data change exceeds a limit condition; otherwise, the data is not changed, and the storage record is not carried out.
The linked list mode records the variable information of the effective quantitative change one by one according to the quantitative change amplitude, and because the variable value quantitative change and the storage time are uncertain, the data storage position is not regular any more, and the variable quantitative change time and the data information need to be linked by means of the inter-segment linked list. The upper node and the lower node of the data linked list respectively correspond to the position information of the front and the back adjacent effective index paragraphs of the current index information segment, namely the last effective data index paragraph of the current variable group can be found through the upper node of the current index information segment linked list; and searching the next effective data index paragraph of the current variable group through the lower node of the current index information paragraph linked list. The linked list address information may add translation rules as appropriate.
When data is analyzed, a target variable is found in an index file (the index file is one of the index information), and the storage position of the variable value in the data file is accurately positioned through the data segment number index of the corresponding variable stored in the index file, so that the quick retrieval of the specific variable data information is realized, the data file with larger volume does not need to be traversed for retrieval, and the data utilization and processing efficiency can be obviously improved.
The data acquisition and storage mode of the undifferentiated record has large data read-write quantity, occupies large memory space and requires large storage space. According to the method and the device, the data to be stored are subjected to characteristic analysis, characteristic screening is carried out according to the change frequency and the control requirement, on the premise that the data recording precision is met, the optimal storage mode can be adopted to meet the system function requirement, the variable number value with substantial change is selectively recorded, complicated and repeated data recording is replaced, invalid memory occupation is released, the storage space is saved, and the data utilization and processing speed is increased. Meanwhile, discrimination of data state transformation is facilitated, and effectiveness and practicability of mass data storage of the process system are obviously improved.
Based on the foregoing embodiments, the embodiments of the present application provide a space allocation scheme and a storage rule for information.
As shown in fig. 7, a space allocation scheme and a storage rule for related information items of a file header of an index file are provided in an embodiment of the present application. The file header part of the index file sequentially stores basic information of the index file: time format, variable quantity, first data recording time, creation time, file data quantity and file name. The fixed-length storage space is allocated according to the content length of each information item.
The space allocation scheme and storage rule of related information items of the index file header shown in fig. 7 are only for illustration and are not intended to limit the present application. The space allocation scheme and the storage rule of the related information items of the index file header can be in a custom format.
As shown in fig. 8, a space allocation scheme and storage rules of related information items of an index segment of an index file are provided according to an embodiment of the present application. Each index segment is provided with a segment header, the length of each index segment is fixed, and the index segment is used for recording basic storage rule information: the variable type, the variable quantity, the linked list address and the like are used as the basis for callback and analysis of the data file. And establishing attribute marks by pre-defining so as to fix the data classification preset storage rule.
And storing the position index information of the name and the numerical value item thereof in the data file next to the index segment head. The name and its index data segment number may be selected in the form of an appropriate interclass separation, typically a fixed length and a characteristic separation flag. When stored at a fixed length, the name unit storage space allocation is determined by the actual storage name length requirement. The index data segment number unit storage space allocation is determined by the maximum data segment number in the corresponding data file. When the data is stored in the form of a feature separation mark, the data can be separated by adding the feature mark after the number of a single name/index data segment. In order to facilitate data storage and call back, the names in the segments and the segment numbers of the index data are preferably stored in a classified and continuous manner. The variable data address index should be determined by appropriate calculation rules.
The space allocation scheme and storage rule of the related information items of the index segments of the index file shown in fig. 8 are only for illustration and are not intended to limit the present application. The space allocation scheme and the storage rule of the related information items of the index segments of the index file can be in a custom format.
As shown in fig. 9, a space allocation scheme and storage rules for related information items of a data file header are provided in an embodiment of the present application. The data file mainly stores numerical information of variables. The header part of the data file sequentially stores basic information of the data file: time format, data segment number, first data recording time, creation time, file data volume and file name. The fixed-length storage space is allocated according to the content length of each information item.
The space allocation scheme and storage rule of the related information items of the data file header shown in fig. 9 are only for illustration and are not intended to limit the present application. The space allocation scheme and the storage rule of the related information items of the data file header can be in a custom format.
Fig. 10 is a space allocation scheme and storage rules of related information items of data segments of a data file according to an embodiment of the present application. Each data segment is provided with a segment head, the length is fixed, and the data segment is used for recording basic storage rule information: the variable type, the data quantity, the initial time of the data record in the section, the name and the like are used as the basis for callback and analysis of the data file. And establishing attribute marks by pre-defining so as to fix the data classification preset storage rule.
The data section next to the data segment header stores variable value information on a time scale. Each piece of data is fixed in length, is grouped with a corresponding time scale, and is continuously stored one by one. The system time storage format can be flexibly selected according to the requirement, such as absolute time, relative time and various customized time formats.
The space allocation scheme and storage rules for the related information items of the data file data segments shown in fig. 10 are only for illustration and are not intended to limit the present application. The space allocation scheme and the storage rule of the related information items of the data file data segments can be in a custom format.
As shown in fig. 11, an embodiment of the present application provides a data storage device, including:
a first determining module 110, configured to determine a variation amplitude of the data;
a first storage module 120, configured to store the data when the variation amplitude exceeds a set threshold;
an establishing module 130, configured to establish index information of the data according to the identifier and the storage location of the data, where the index information at least includes: the identification and location information of the storage location.
In some embodiments, the first determining module 110, the first storing module 120, and the establishing module 130 may be all program modules, and the program modules can realize the functions of the above modules after being executed by a processor.
In other embodiments, the first determining module 110, the first storing module 120, and the establishing module 130 may be a hardware-software module; the soft and hard combining module includes but is not limited to: various programmable arrays; the field programmable array includes, but is not limited to: field programmable arrays and/or complex programmable arrays.
In still other embodiments, the first determining module 110, the first storing module 120, and the establishing module 130 may all be pure hardware modules; the pure hardware modules include, but are not limited to: an application specific integrated circuit.
In some embodiments, the apparatus further comprises:
a second determining module for determining a change time of the data;
a third determining module, configured to determine a storage node according to the change amplitude and the change time, where the storage node is configured to store the data and the change time; wherein the location information indicates the storage node.
In some embodiments, the apparatus further comprises:
the second storage module is used for storing the index information by adopting a linked list, wherein the linked list comprises a plurality of nodes; each node comprises at least two parts: a node address area and an index information area; the node address area is used for storing the address information of the node; the index information area is used for storing the index information of the data; the node address area stores a first address and a second address; the first address is the address of the previous adjacent node of the current node; the second address is the address of the next adjacent node of the current node; the index information area contains index information of a plurality of data.
In some embodiments, the storing the data when the magnitude of the change exceeds a set threshold includes:
storing the data by adopting a data group, wherein the data group at least comprises two data; and storing index information of the data included in the data group by adopting the same linked list. .
As shown in fig. 12, an embodiment of the present application provides a data reading apparatus, including:
a receiving module 210, configured to receive a data read request, where the data read request at least includes an identifier of data to be read;
the query module 220 is configured to query the index information according to the identifier to obtain location information corresponding to the identifier;
the reading module 230 is configured to read the data to be read according to the storage location indicated by the location information.
In some embodiments, the receiving module 210, the querying module 220, and the reading module 230 may be all program modules, and the program modules can be executed by a processor to implement the functions of the above modules.
In other embodiments, the receiving module 210, the querying module 220, and the reading module 230 may be all hardware and software combined modules; the soft and hard combining module includes but is not limited to: various programmable arrays; the field programmable array includes, but is not limited to: field programmable arrays and/or complex programmable arrays.
In still other embodiments, the receiving module 210, the querying module 220, and the reading module 230 may all be purely hardware modules; the pure hardware modules include, but are not limited to: an application specific integrated circuit.
In some embodiments, the querying index information according to the identifier to obtain location information corresponding to the identifier includes:
the index information is stored by adopting a linked list, wherein the linked list comprises a plurality of nodes; each node comprises at least two parts: a node address area and an index information area;
the node address area is used for storing the address information of the node; the index information area is used for storing the index information of the data to be read; the node address area stores a first address and a second address; the first address is the address of the previous adjacent node of the current node; the second address is the address of the next adjacent node of the current node; the index information area contains index information of a plurality of data to be read.
In some embodiments, the data to be read includes:
the data to be read is stored by adopting a data group, wherein the data group at least comprises two data to be read; and the index information of each data to be read included in one data group is stored by adopting the same linked list.
In some embodiments, the apparatus further comprises:
the first obtaining module is used for querying the linked list according to the data reading request to obtain the linked list corresponding to the data to be read, wherein the data reading request comprises type information, and the type information indicates the data type of the data to be read;
a second obtaining module, configured to query a corresponding linked list according to the identifier, and obtain index information corresponding to the data to be read;
and the third obtaining module is used for obtaining a storage position corresponding to the data to be read according to the corresponding index information.
As shown in fig. 13, an embodiment of the present application provides an electronic device, including:
a memory for storing computer readable instructions;
and the processor is connected with the memory and used for realizing the method provided by any of the foregoing embodiments by executing the computer readable instructions, for example, the method shown in fig. 1 and/or fig. 2 can be executed.
The memory can be various types of memories, such as random access memory, read only memory, flash memory, and the like. The memory may be used for information storage, e.g., storing computer-executable instructions, etc. The computer-executable instructions may be various program instructions, such as object program instructions and/or source program instructions, and the like.
The processor may be various types of processors, such as a central processing unit, a microprocessor, a digital signal processor, a programmable array, a digital signal processor, an application specific integrated circuit, or an image processor, among others. The processor may be connected to the memory via a bus. The bus may be an integrated circuit bus or the like.
As shown in fig. 13, the electronic device may further include a network interface, which may be used for interacting with a peer device through a network.
Embodiments of the present application further provide a computer storage medium, where computer-executable instructions are stored, and when executed, the computer storage medium is capable of implementing the method provided in any of the foregoing embodiments, for example, the method shown in fig. 1 and/or fig. 2 may be executed.
The computer storage medium provided by the embodiment comprises: a mobile storage device, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
In the several embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. The above-described device embodiments are merely illustrative, for example, the division of the unit is only a logical functional division, and there may be other division ways in actual implementation, such as: multiple units or components may be combined, or may be integrated into another system, or some features may be omitted, or not implemented. In addition, the coupling, direct coupling or communication connection between the components shown or discussed may be through some interfaces, and the indirect coupling or communication connection between the devices or units may be electrical, mechanical or other forms.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed on a plurality of network units; some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, all functional units in the embodiments of the present application may be integrated into one processing module, or each unit may be separately regarded as one unit, or two or more units may be integrated into one unit; the integrated unit can be realized in a form of hardware, or in a form of hardware plus a software functional unit.
Those of ordinary skill in the art will understand that: all or part of the steps for implementing the method embodiments may be implemented by hardware related to program instructions, and the program may be stored in a computer readable storage medium, and when executed, the program performs the steps including the method embodiments; and the aforementioned storage medium includes: a mobile storage device, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
The above description is only for the specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present application, and shall be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
Claims (12)
1. A data storage method is applied to a storage system and comprises the following steps:
determining the variation amplitude of the data;
when the change amplitude exceeds a set threshold value, storing the data;
establishing index information of the data according to the identification and the storage position of the data, wherein the index information at least comprises: the identification and location information of the storage location.
2. The method of claim 1, further comprising:
determining a time of change of the data;
determining a storage node according to the change amplitude and the change time, wherein the storage node is used for storing the data and the change time; wherein the location information indicates the storage node.
3. The method of claim 1, further comprising:
storing the index information by adopting a linked list, wherein the linked list comprises a plurality of nodes; each node comprises at least two parts: a node address area and an index information area;
the node address area is used for storing the address information of the node; the index information area is used for storing the index information of the data; the node address area stores a first address and a second address; the first address is the address of the previous adjacent node of the current node; the second address is: an address of a next adjacent node of the current node; the index information area contains index information of a plurality of data.
4. The method of claim 3, wherein storing the data when the magnitude of change exceeds a set threshold comprises:
storing the data by adopting a data group, wherein the data group at least comprises two data; and storing index information of each data in one data group by adopting the same linked list.
5. A data reading method is applied to a storage system and comprises the following steps:
receiving a data reading request, wherein the data reading request at least comprises an identifier of data to be read;
inquiring index information according to the identification to obtain position information corresponding to the identification;
and reading the data to be read according to the storage position indicated by the position information.
6. The method according to claim 5, wherein said querying index information according to the identifier to obtain location information corresponding to the identifier comprises:
the index information is stored by adopting a linked list, wherein the linked list comprises a plurality of nodes; each node comprises at least two parts: a node address area and an index information area;
the node address area is used for storing the address information of the node; the index information area is used for storing the index information of the data to be read; the node address area stores a first address and a second address; the first address is the address of the previous adjacent node of the current node; the second address is the address of the next adjacent node of the current node; the index information area contains index information of a plurality of data to be read.
7. The method of claim 6, wherein the data to be read comprises:
the data to be read is stored by adopting a data group, wherein the data group at least comprises two data to be read; and the index information of each data to be read included in one data group is stored by adopting the same linked list.
8. The method of claim 6, further comprising:
querying the linked list according to the data reading request to obtain a linked list corresponding to the data to be read, wherein the data reading request comprises type information, and the type information indicates the data type of the data to be read;
inquiring a corresponding linked list according to the identification to obtain index information corresponding to the data to be read;
and obtaining a storage position corresponding to the data to be read according to the corresponding index information.
9. A data storage apparatus, for use in a storage system, the storage system including a plurality of storage nodes, the apparatus comprising:
the first determining module is used for determining the variation amplitude of the data;
the first storage module is used for storing the data when the change amplitude exceeds a set threshold;
an establishing module, configured to establish index information of the data according to the identifier and the storage location of the data, where the index information at least includes: the identification and location information of the storage location.
10. A data reading apparatus, applied to a storage system, the apparatus comprising:
the device comprises a receiving module, a processing module and a processing module, wherein the receiving module is used for receiving a data reading request, and the data reading request at least comprises an identifier of data to be read;
the query module is used for querying index information according to the identification to obtain position information corresponding to the identification;
and the reading module is used for reading the data to be read according to the storage position indicated by the position information.
11. An electronic device, comprising:
a memory storing computer readable instructions;
a processor coupled to the memory and configured to implement the method of claims 1-8 by executing the computer-readable instructions.
12. A computer storage medium having stored thereon computer-executable instructions; the computer-executable instructions, when executed by a processor, are capable of implementing the method of any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111643360.XA CN114416731A (en) | 2021-12-29 | 2021-12-29 | Data storage method, data reading method, data storage device, electronic device and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111643360.XA CN114416731A (en) | 2021-12-29 | 2021-12-29 | Data storage method, data reading method, data storage device, electronic device and medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114416731A true CN114416731A (en) | 2022-04-29 |
Family
ID=81270292
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111643360.XA Pending CN114416731A (en) | 2021-12-29 | 2021-12-29 | Data storage method, data reading method, data storage device, electronic device and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114416731A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115100757A (en) * | 2022-06-20 | 2022-09-23 | 重庆长安汽车股份有限公司 | Automobile data storage method and device, vehicle and storage medium |
-
2021
- 2021-12-29 CN CN202111643360.XA patent/CN114416731A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115100757A (en) * | 2022-06-20 | 2022-09-23 | 重庆长安汽车股份有限公司 | Automobile data storage method and device, vehicle and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107122126B (en) | Data migration method, device and system | |
| CN109783023B (en) | Method and related device for data scrubbing | |
| CN105468642A (en) | Data storage method and apparatus | |
| EP3796182A1 (en) | Data processing method and apparatus, and computer-readable storage medium | |
| CN105512283A (en) | Data quality management and control method and device | |
| CN111813756B (en) | Log retrieval system, method and device, electronic equipment and storage medium | |
| CN112148690A (en) | File caching method, file access request processing method and device | |
| CN111061758A (en) | Data storage method, device and storage medium | |
| CN114416731A (en) | Data storage method, data reading method, data storage device, electronic device and medium | |
| CN110516220B (en) | Report data input method, system and related equipment | |
| CN116340367A (en) | Data query method, device, equipment and storage medium | |
| CN111831691B (en) | Data reading and writing method and device, electronic equipment and storage medium | |
| CN109739854A (en) | A kind of date storage method and device | |
| CN117076341A (en) | Data storage method, device, computer equipment and computer readable storage medium | |
| CN113297245A (en) | Method and device for acquiring execution information | |
| CN109815351B (en) | Information query method and related product | |
| CN101515253A (en) | Device and method for writing file into storage medium and reading file from storage medium | |
| CN112162951A (en) | Information retrieval method, server and storage medium | |
| CN113220992A (en) | Information flow content recommendation method, system and medium | |
| CN113468379A (en) | Data source processing method and device and intelligent analysis platform | |
| CN111046249A (en) | Data storage, positioning and application method and related device | |
| CN117909301B (en) | Index-based object query method, device, equipment and medium | |
| CN117874296A (en) | Time sequence data storage method, processing equipment and computer readable medium | |
| CN104572643A (en) | Search method and search engine | |
| CN117555968B (en) | Data processing method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |