CN113239039B - Dynamic data storage method, query method, management method and management system - Google Patents
Dynamic data storage method, query method, management method and management system Download PDFInfo
- Publication number
- CN113239039B CN113239039B CN202110509412.8A CN202110509412A CN113239039B CN 113239039 B CN113239039 B CN 113239039B CN 202110509412 A CN202110509412 A CN 202110509412A CN 113239039 B CN113239039 B CN 113239039B
- Authority
- CN
- China
- Prior art keywords
- data
- message
- protocol
- specific key
- item
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 76
- 238000013500 data storage Methods 0.000 title claims abstract description 14
- 238000007726 management method Methods 0.000 title abstract description 15
- 238000012545 processing Methods 0.000 claims description 13
- 238000004806 packaging method and process Methods 0.000 claims description 9
- 238000013507 mapping Methods 0.000 claims description 6
- 230000008569 process Effects 0.000 abstract description 11
- 238000013523 data management Methods 0.000 abstract description 7
- 230000008859 change Effects 0.000 abstract description 4
- 238000005516 engineering process Methods 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 230000002085 persistent effect Effects 0.000 description 2
- 238000013439 planning Methods 0.000 description 2
- 230000001105 regulatory effect Effects 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 1
- 238000013479 data entry Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000009977 dual effect Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/22—Parsing or analysis of headers
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Computer Security & Cryptography (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to a storage method, a query method, a management method and a management system of dynamic data, wherein the storage method of the invention utilizes a predefined specific key value (specific key value) and an incoming timestamp corresponding to a data message to determine a storage path, utilizes a preset general data item sequence to carry out protocol on message data, stores the data message in the form of protocol data, only needs to modify the specific key and the preset general data item when the data item is required to be added or changed or the data storage sequence is required to be changed, does not need to change program codes, and improves the data expansion performance in the project data management process.
Description
Technical Field
The present invention relates to the field of big data management technologies, and in particular, to a method for storing dynamic data, a method for querying the dynamic data, a method for managing the dynamic data, and a system for managing the dynamic data.
Background
The main aim of big data research is to acquire, process and analyze big data of various application industries by effective information technology means and calculation methods, discover and extract the depth value of the data, and provide high added value application and service for the industries. Since big data problems come from industry application, big data research needs to be guided by industry application problems and requirements, and effective processing technologies and solutions are researched and solved starting from actual application requirements of industry and existing technical problems.
Under the principle of following the application requirement oriented, a method of accessing Kafka data through a jump in a practical project finds some problems: firstly, except for the determined data item, if the service is expanded, under the situation that the requirement of the data item is expanded along with the expansion, the data storage mode also needs to be changed, and the development code also needs to be redefined and expanded to conform to the change of the storage mode; and if the predefined data is changed, the code needs to be changed for configuration, and flexible configuration of the predefined items is not achieved. It can be seen that the existing project data management method has the technical defect of low data expansion performance.
Disclosure of Invention
The invention aims to provide a dynamic data storage method, a query method, a management method and a management system, so as to improve the data expansion performance in the project data management process.
In order to achieve the above object, the present invention provides the following solutions:
the invention provides a storage method of dynamic data, which comprises the following steps:
analyzing the data message transmitted in real time to obtain a specific key value, an incoming time stamp and data content of the data message;
determining a storage path of the data message according to the specific key value and the incoming timestamp;
based on a preset general data item sequence, carrying out structured protocol processing on the data message according to the data content to obtain protocol data of the data message;
and storing the data message according to the storage path and the specification data.
Optionally, the analyzing the data message incoming in real time to obtain a specific key value, an incoming timestamp and data content of the data message specifically includes:
analyzing the data message to obtain an incoming time stamp and data content of the data message;
and obtaining a specific key value of the data message according to the data content.
Optionally, obtaining the specific key value of the data message according to the data content specifically includes:
judging whether the data message contains a specific key according to the data content to obtain a first judging result;
if the first judgment result indicates yes, judging whether the value of the specific key mapping contained in the data message is empty or not, and obtaining a second judgment result;
if the second judgment result indicates no, outputting a specific key mapping value contained in the data message as a specific key value of the data message;
and if the first judging result indicates no or the second judging result indicates yes, outputting a filling value as a specific key value of the data message.
Optionally, the performing structured protocol processing on the data message according to the data content based on a preset general data item sequence to obtain protocol data of the data message specifically includes:
determining data items contained in the data message according to the data content;
generating a null sequence comprising X items; x is the number of general data items in a preset general data item sequence;
repeatedly executing the matching of the data items contained in the data message and the universal data items in the preset universal data item sequence, and storing the key values of the successfully matched data items to the positions corresponding to the universal data items matched with the data items in the null sequence until the matching is completed on all the data items contained in the data message, so as to obtain a matched data sequence;
judging whether unmatched universal data items exist in a preset universal data item sequence or not, and obtaining a third judging result;
if the third judgment result shows that the data message is a data message, adding the key value of the unmatched general data item to the next column of the matched data sequence to obtain the data sequence containing the X+1 item as the protocol data of the data message;
and if the third judgment result indicates no, outputting the matched data sequence as the protocol data of the data message.
A method of querying dynamic data, the method comprising the steps of:
determining a query path according to the query condition parameters; the query condition parameters comprise data input time and specific key values;
determining a plurality of storage paths corresponding to the query paths;
obtaining protocol data under each storage path corresponding to the query path to form a protocol data set;
the protocol data set is encapsulated into a message object and output.
Optionally, the determining the query path according to the query condition parameter specifically includes:
judging whether the query condition parameters are effective parameters or not, and obtaining a fourth judgment result;
if the fourth judgment result shows that the data input time in the query condition parameters is changed into a time word containing eight digits;
and determining a query path according to the length of the time word code, the time word code and a specific key value in the query condition parameters.
Optionally, the packaging the protocol data set into a message object and outputting the message object specifically includes:
judging whether the (X+1) th item of the ith protocol data in the protocol data set is empty or not, and obtaining a fifth judgment result;
if the fifth judging result is negative, splitting the (X+1) th item from the ith protocol data to be output as a general data item, and packaging the rest data items in the ith protocol data into a message object to be output;
if the fifth judgment result is yes, the ith protocol data is packaged into a message object to be output;
and (3) increasing the value of i by 1, and returning to the step of judging whether the X+1 item of the ith protocol data in the protocol data set is empty or not to obtain a fifth judgment result until the output of each protocol data in the protocol data set is completed.
A method of managing dynamic data, the method comprising the steps of:
storing the data message transmitted in real time by adopting a dynamic data storage method;
and adopting a dynamic data query method to query the data message stored in the form of the specification data.
A management system for dynamic data, the management system comprising:
the storage module is used for storing the data messages transmitted in real time by adopting a dynamic data storage method;
and the query module is used for querying the data message stored in the form of the protocol data by adopting a query method of dynamic data.
According to the specific embodiment provided by the invention, the invention discloses the following technical effects:
the invention discloses a storage method, a query method, a management method and a management system of dynamic data, wherein the storage method comprises the following steps: analyzing the data message transmitted in real time to obtain a specific key value, an incoming time stamp and data content of the data message; determining a storage path of the data message according to the specific key value and the incoming timestamp; based on a preset general data item sequence, carrying out structured protocol processing on the data message according to the data content to obtain protocol data of the data message; and storing the data message according to the storage path and the specification data. The invention determines the storage path by utilizing the value of the corresponding predefined specific key and the input timestamp in the data message, utilizes the message data to be regulated based on the preset general data item sequence, stores the data message in the form of the regulation data, only needs to modify the specific key and the preset general data item when the data item is required to be added or changed or the data storage sequence is required to be changed, does not need to change the program code, and improves the data expansion performance in the project data management process.
The invention stores, inquires and manages the data information in the form of the protocol data, and because the protocol data corresponds to the preset general data item sequence, the key corresponding to each data item can be obtained from the preset general data item sequence, only the key value of the data item is required to be stored, the key is not required to be stored, and the storage space is saved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions of the prior art, the drawings that are needed in the embodiments will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of a method for storing dynamic data according to the present invention;
FIG. 2 is a flow chart of determining a storage path according to the present invention;
FIG. 3 is a flow chart of a structured protocol process provided by the present invention;
FIG. 4 is a flow chart of a method for querying dynamic data provided by the invention;
fig. 5 is a schematic diagram of a method for managing dynamic data according to the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The invention aims to provide a dynamic data storage method, a query method, a management method and a management system, so as to improve the data expansion performance in the project data management process.
In order that the above-recited objects, features and advantages of the present invention will become more readily apparent, a more particular description of the invention will be rendered by reference to the appended drawings and appended detailed description.
The invention aims to solve the problems of elastic expansion and efficient storage query of actual items aiming at big data, and is a solution for dealing with dynamic data access under the uncertain scene of specific key values and data items, namely a double elastic mechanism is used for dynamically storing the data so as to solve the problems of uncertain specific key values and uncertain data items. Firstly, after real-time data is transmitted, analyzing the data, classifying according to a specific key value and an transmitted time stamp of a message, planning a storage path, and formatting and storing specific data according to specific data content and protocol data items; and then storing data according to the specific storage path and the regulated data item, and persisting the data into a file according to the planned storage path. And secondly, after two inquiry conditions of time and type are provided, replacing a data item according to a path template, acquiring a real path value, analyzing the data through the positioned specific data, and packaging an object as output for subsequent data processing.
As shown in fig. 1, the present invention provides a method for storing dynamic data, which includes the following steps:
and step 101, analyzing the data message which is transmitted in real time to obtain a specific key value, an incoming time stamp and data content of the data message.
Step 101 is preceded by setting a predefined variable:
for a specific key (which can be the type of data), a specific key value tableName of a predefined data message, extension fields extensions, path formats and the like, dynamic configuration and modification can be performed by matching a specific parsing tool class in an XML configuration file mode. Providing more flexibility to the overall process.
The analyzing the data message transmitted in real time to obtain a specific key value, an incoming time stamp and data content of the data message specifically comprises the following steps: analyzing the data message to obtain an incoming time stamp and data content of the data message; and obtaining a specific key value of the data message according to the data content.
The specific key value of the data message is obtained according to the data content, and the specific key value comprises the following specific steps: judging whether the data message contains a specific key according to the data content to obtain a first judging result; if the first judgment result indicates yes, judging whether the value of the specific key mapping contained in the data message is empty or not, and obtaining a second judgment result; if the second judgment result indicates no, outputting a specific key mapping value contained in the data message as a specific key value of the data message; and if the first judging result indicates no or the second judging result indicates yes, outputting a filling value as a specific key value of the data message. The padding value of the present invention is a predefined specific key value.
Step 102, determining a storage path of the data message according to the specific key value and the incoming time stamp.
As shown in fig. 2, steps 101 and 102 of the present invention specifically include: after the real-time data message is transmitted, the data is analyzed, the storage path is planned according to the specific key value of the message and the transmission time stamp classification, and the specific data is formatted and stored according to specific data content and protocol data items.
As illustrated in fig. 2, after the parsed data is obtained, the example: K1:V1, K2:V2, K3:V3 … …, firstly judging whether the message contains a specific key tableName, namely whether a certain K named tableName exists, then judging whether the mapped value of the specific key tableName is empty, namely whether the corresponding V of the K has a value, if the process has the value, taking out the corresponding value of the specific key tableName, otherwise filling according to a predefined value other. Based on the time of entry and the value retrieved, the value, i.e., the type, is used as a primary catalog by type and the time is used as a secondary catalog in a "type/year/month/day" format to identify the alternate storage path to the floor.
And 103, carrying out structured protocol processing on the data message according to the data content based on a preset general data item sequence to obtain protocol data of the data message.
Step 103, based on a preset general data item sequence, performing structured protocol processing on the data message according to the data content to obtain protocol data of the data message, which specifically includes: determining data items contained in the data message according to the data content; generating a null sequence comprising X items; x is the number of general data items in a preset general data item sequence; repeatedly executing the matching of the data items contained in the data message and the universal data items in the preset universal data item sequence, and storing the key values of the successfully matched data items to the positions corresponding to the universal data items matched with the data items in the null sequence until the matching is completed on all the data items contained in the data message, so as to obtain a matched data sequence; judging whether unmatched universal data items exist in a preset universal data item sequence or not, and obtaining a third judging result; if the third judgment result shows that the data message is a data message, adding the key value of the unmatched general data item to the next column of the matched data sequence to obtain the data sequence containing the X+1 item as the protocol data of the data message; and if the third judgment result indicates no, outputting the matched data sequence as the protocol data of the data message.
As shown in fig. 3, the specific steps of step 103 may be summarized as: after obtaining parsed data, example: K1:V1, K2:V2, K3:V3 … …, and filling according to a preset general data item sequence. The preset general data items are data items according to high service use frequency and data items commonly owned by multiple acceptance messages, and examples are as follows: oeTime/timestamp, primary ID/primary ID, tableName/data type, extensions/extensions … …. The filling function is to carry out structured protocol processing on the dynamically transmitted data, so that the K and the general data item are kept consistent in sequence and the V is stored. Then judging whether unmatched general data items exist, if so, using 'as separators' key values of all unmatched data items, performing character string splicing on the 'as separators' key value pairs, and adding double quotation marks as values of general data items to store, wherein the example is as follows: k is extensions and V is "K1: V1, K2: V2". And ordering the values of all the data items according to the data item template sequence by taking the values as separators. If the data item does not have the corresponding value, the data item is not stored, and only the separator is output; the value corresponding to the universal data item extensions is placed in the last column, example: k1, K2,, K6, … …, "K1: v1, K2: v2".
And 104, storing the data message according to the storage path and the specification data.
As shown in fig. 4, the present invention further provides a method for querying dynamic data, where the method for querying dynamic data includes the following steps:
determining a query path according to the query condition parameters; the query condition parameters include data entry time and specific key values. As a preferred embodiment, the specific key of the present invention is defined as the type of data message as shown in fig. 4. After two query conditions of data input time and type are provided, a data item is replaced according to a path template, a real path value is obtained, the data is analyzed through the positioned specific data, and an object is packaged as output for subsequent data processing.
After two parameters of data input time and type are obtained, firstly checking whether the input parameters are valid or not, and eliminating error parameters. The time is then converted to an eight-bit pure digital "year, month, day" format, i.e., an instant word, example: 20210208, 202102, 2021, … …. Determining a path for acquiring the data set, namely a query path, according to the length of the time word code: length 8, path "type/year/month/day"; length 6, path "type/year/month/"; length 4 and path "type/year/. Examples: if tbName is used as the type parameter and 202102 is used as the time parameter, the path is "tbName/2021/02/. And loading all the protocol data according to the obtained query path to form a protocol data set.
The determining the query path according to the query condition parameters specifically includes: judging whether the query condition parameters are effective parameters or not, and obtaining a fourth judgment result; if the fourth judgment result shows that the data input time in the query condition parameters is changed into a time word containing eight digits; and determining a query path according to the length of the time word code, the time word code and a specific key value in the query condition parameters.
And determining a plurality of storage paths corresponding to the query paths.
And obtaining the protocol data under each storage path corresponding to the query path to form a protocol data set.
The protocol data set is encapsulated into a message object and output.
And dividing the protocol data in the protocol data set according to the separator, and judging whether a last column has a value, namely whether the value corresponding to the universal data item extensions is null. If the data is not null, the data corresponding to the general data item extensions is analyzed, the header is mapped to Key, and the table item is mapped to Value, as shown in Table 1.
Table 1 protocol data set list
| K1 | K2 | …… | tableName | extends_K1 | extends_K2 | …… |
| V11 | V21 | …… | tbName1 | ev11 | ev12 | …… |
| V21 | V22 | …… | tbName2 | ev21 | ev22 | …… |
| …… | …… | …… | …… | …… | …… | …… |
The method for packaging the protocol data set into a message object and outputting the message object specifically comprises the following steps: judging whether the (X+1) th item of the ith protocol data in the protocol data set is empty or not, and obtaining a fifth judgment result; if the fifth judging result is negative, splitting the (X+1) th item from the ith protocol data to be output as a general data item, and packaging the rest data items in the ith protocol data into a message object to be output; if the fifth judgment result is yes, the ith protocol data is packaged into a message object to be output; and (3) increasing the value of i by 1, and returning to the step of judging whether the X+1 item of the ith protocol data in the protocol data set is empty or not to obtain a fifth judgment result until the output of each protocol data in the protocol data set is completed.
The invention also provides a method for managing dynamic data, which comprises the following steps:
and storing the data message which is transmitted in real time by adopting a dynamic data storage method.
And adopting a dynamic data query method to query the data message stored in the form of the specification data.
The following describes the data management method of the present invention with reference to fig. 5, and as shown in fig. 5, after real-time data is transferred, the flash component can be custom developed to implement the function of dynamic data storage, so that the dynamic data can be stored in the distributed file system in a persistent manner. The user provides two inquiry parameters to search data, and the data is returned to the user in combination with specific tool analysis, namely the whole process is transparent to the user and is a black box process. After receiving the message, there may be one or more data categories in terms of message category that are different from the preset category; in terms of data items, there may be one or more data items different from the preset data items, which are spliced and stored in a format as extension data items. The dual elastic expansion mechanism ensures that messages can be received dynamically, and in order to provide better flexibility, predefined data items can be changed through an XML configuration mode. The structured data item is provided for unstructured data in a structured storage mode, and the tool class performs targeted analysis operation.
The invention also provides a management system of dynamic data, which comprises:
the storage module is used for storing the data message transmitted in real time by adopting a dynamic data storage method, namely analyzing the data according to real-time data transmission (namely input), classifying according to a specific key value tableName and an input timestamp of the message, and planning a storage path; and according to the specific data content, the protocol data item is used for formatting and storing specific data, and then according to the specific storage path and the protocol data item, the data is stored. Finally, the data persists into the file (i.e., exports).
The query module is used for querying the data message stored in the form of the protocol data by adopting a query method of dynamic data, namely, performing data retrieval according to two query parameters provided by a user, (two query parameters are input) replacing data items according to a path template, obtaining a real path value, connecting to a storage module output result according to a specific path, positioning the specific data, analyzing the data, packaging the data into an object, and returning the object (namely, outputting) to the user for subsequent processing.
According to the specific embodiment provided by the invention, the invention discloses the following technical effects:
1. high scalability
In steps 101 and 102 of the storage method, a storage path is planned according to the message specific key value tableName classification; in step 103 of the storage method, according to the specific data content, the protocol data item is used for formatting and storing the specific data; by dynamically configuring the predefined content, an extension of the three aspects of data type, data item, predefined content can be achieved.
2. Flexibility of
In steps 101 and 102 of the storage method, there are compatible items other as storage types even if the type does not exist, classified according to the message specific key value tableName; in step 103 of the storing method, according to the specific data content, the protocol data item, even if the general data item does not exist, has the compatible item extensions stored as the data item; in the query method, for a user, only two necessary retrieval parameters are provided to obtain a packaged data object; the invention does not need to change codes by dynamically configuring the predefined content.
3. High storage efficiency
In step 103 of the storing method, only specific values V need to be stored in the order of the universal data item templates, and specific keys K need not to be stored, but all values are "cut" for the storage space that is not occupied by null values.
4. High retrieval efficiency
In step 103 of the storage method, all the values vbemory are stored in sequence, and the storage mode of the structured data can improve the retrieval efficiency.
5. Convenient to use
In the query method, the query module is transparent to the user, and the user does not care about or relate to a specific data processing process, so that the packaged data object can be obtained; under the support of the dynamic configuration supported by XML, the invention can support the service requirement by changing the configuration file without changing codes even if related services are changed.
In the present specification, each embodiment is described in a progressive manner, and each embodiment is mainly described in a different point from other embodiments, and identical and similar parts between the embodiments are all enough to refer to each other.
The principles and embodiments of the present invention have been described herein with reference to specific examples, the description of which is intended only to assist in understanding the methods of the present invention and the core ideas thereof; also, it is within the scope of the present invention to be modified by those of ordinary skill in the art in light of the present teachings. In view of the foregoing, this description should not be construed as limiting the invention.
Claims (7)
1. A method of storing dynamic data, the method comprising the steps of:
analyzing the data message transmitted in real time to obtain a specific key value, an incoming time stamp and data content of the data message;
determining a storage path of the data message according to the specific key value and the incoming timestamp;
based on a preset general data item sequence, carrying out structured protocol processing on the data message according to the data content to obtain protocol data of the data message;
storing the data message according to the storage path and the specification data;
the method comprises the steps of carrying out structured protocol processing on the data message according to the data content based on a preset general data item sequence to obtain protocol data of the data message, and specifically comprises the following steps:
determining data items contained in the data message according to the data content;
generating a null sequence comprising X items; x is the number of general data items in a preset general data item sequence;
repeatedly executing the matching of the data items contained in the data message and the universal data items in the preset universal data item sequence, and storing the key values of the successfully matched data items to the positions corresponding to the universal data items matched with the data items in the null sequence until the matching is completed on all the data items contained in the data message, so as to obtain a matched data sequence;
judging whether unmatched universal data items exist in a preset universal data item sequence or not, and obtaining a third judging result;
if the third judgment result shows that the data message is the data message, adding the key value of the unmatched general data item to the next column of the matched data sequence to obtain the data sequence containing the X+1 item as the protocol data of the data message;
and if the third judging result indicates no, outputting the matched data sequence as the protocol data of the data message.
2. The method for storing dynamic data according to claim 1, wherein the parsing of the data message incoming in real time obtains a specific key value, an incoming timestamp and data content of the data message, specifically comprising:
analyzing the data message to obtain an incoming time stamp and data content of the data message;
and obtaining a specific key value of the data message according to the data content.
3. The method for storing dynamic data according to claim 2, wherein obtaining the specific key value of the data message according to the data content specifically comprises:
judging whether the data message contains a specific key according to the data content to obtain a first judging result;
if the first judgment result indicates yes, judging whether the value of the specific key mapping contained in the data message is empty or not, and obtaining a second judgment result;
if the second judgment result indicates no, outputting a specific key mapping value contained in the data message as a specific key value of the data message;
and if the first judging result indicates no or the second judging result indicates yes, outputting a filling value as a specific key value of the data message.
4. A method for querying dynamic data, the method comprising the steps of:
determining a query path according to the query condition parameters; the query condition parameters comprise data input time and specific key values;
determining a plurality of storage paths corresponding to the query paths;
obtaining protocol data under each storage path corresponding to the query path to form a protocol data set;
packaging the protocol data set into a message object and outputting the message object;
the method for packaging the protocol data set into a message object and outputting the message object specifically comprises the following steps:
judging whether the (X+1) th item of the ith protocol data in the protocol data set is empty or not, and obtaining a fifth judgment result;
if the fifth judging result is negative, splitting the (X+1) th item from the ith protocol data to be output as a general data item, and packaging the rest data items in the ith protocol data into a message object to be output;
if the fifth judgment result is yes, the ith protocol data is packaged into a message object to be output;
and (3) increasing the value of i by 1, and returning to the step of judging whether the X+1 item of the ith protocol data in the protocol data set is empty or not to obtain a fifth judgment result until the output of each protocol data in the protocol data set is completed.
5. The method for querying dynamic data according to claim 4, wherein determining a query path according to the query condition parameter specifically comprises:
judging whether the query condition parameters are effective parameters or not, and obtaining a fourth judgment result;
if the fourth judgment result shows that the data input time in the query condition parameters is changed into a time word containing eight digits;
and determining a query path according to the length of the time word code, the time word code and a specific key value in the query condition parameters.
6. A method of managing dynamic data, the method comprising the steps of:
storing real-time incoming data messages using the dynamic data storage method of any one of claims 1-3;
querying a data message stored in the form of protocol data using the method for querying dynamic data according to any one of claims 4 to 5.
7. A system for managing dynamic data, the system comprising:
a storage module for storing real-time incoming data messages using the dynamic data storage method of any one of claims 1-3;
a query module for querying data messages stored in the form of protocol data using the method for querying dynamic data according to any one of claims 4 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110509412.8A CN113239039B (en) | 2021-05-11 | 2021-05-11 | Dynamic data storage method, query method, management method and management system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110509412.8A CN113239039B (en) | 2021-05-11 | 2021-05-11 | Dynamic data storage method, query method, management method and management system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113239039A CN113239039A (en) | 2021-08-10 |
| CN113239039B true CN113239039B (en) | 2023-11-14 |
Family
ID=77133273
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110509412.8A Active CN113239039B (en) | 2021-05-11 | 2021-05-11 | Dynamic data storage method, query method, management method and management system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113239039B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114764326B (en) * | 2022-03-30 | 2023-09-19 | 中国石油天然气集团有限公司 | Data layer expansion method and system of integrated software |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2013171826A1 (en) * | 2012-05-14 | 2013-11-21 | 富士機械製造株式会社 | Control system |
| CN108184170A (en) * | 2017-11-27 | 2018-06-19 | 北京奇艺世纪科技有限公司 | A kind of data processing method and device |
| CN108509834A (en) * | 2018-01-18 | 2018-09-07 | 杭州电子科技大学 | Graph structure stipulations method based on video features under polynary logarithm Gaussian Profile |
| CN110335507A (en) * | 2019-06-12 | 2019-10-15 | 中国电子科技集团公司第二十八研究所 | Flight operation situation law analytical method based on blank pipe track big data |
| CN111309677A (en) * | 2020-02-11 | 2020-06-19 | 西安奥卡云数据科技有限公司 | File management method and device of distributed file system |
| CN112632129A (en) * | 2020-12-31 | 2021-04-09 | 联想未来通信科技(重庆)有限公司 | Code stream data management method, device and storage medium |
-
2021
- 2021-05-11 CN CN202110509412.8A patent/CN113239039B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2013171826A1 (en) * | 2012-05-14 | 2013-11-21 | 富士機械製造株式会社 | Control system |
| CN108184170A (en) * | 2017-11-27 | 2018-06-19 | 北京奇艺世纪科技有限公司 | A kind of data processing method and device |
| CN108509834A (en) * | 2018-01-18 | 2018-09-07 | 杭州电子科技大学 | Graph structure stipulations method based on video features under polynary logarithm Gaussian Profile |
| CN110335507A (en) * | 2019-06-12 | 2019-10-15 | 中国电子科技集团公司第二十八研究所 | Flight operation situation law analytical method based on blank pipe track big data |
| CN111309677A (en) * | 2020-02-11 | 2020-06-19 | 西安奥卡云数据科技有限公司 | File management method and device of distributed file system |
| CN112632129A (en) * | 2020-12-31 | 2021-04-09 | 联想未来通信科技(重庆)有限公司 | Code stream data management method, device and storage medium |
Non-Patent Citations (2)
| Title |
|---|
| A relational database schema on the transaction key-value store scalaris;Nick Kruber等;《2014 IEEE international conference on big data》;第1-2页 * |
| 基于云计算的物联网数据挖掘;张海江 等;《微型电脑应用》;第28卷(第6期);第10-13页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113239039A (en) | 2021-08-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11068439B2 (en) | Unsupervised method for enriching RDF data sources from denormalized data | |
| CN109670081B (en) | Method and device for processing service request | |
| CN111008020B (en) | Method for analyzing logic expression into general query statement | |
| US20220004547A1 (en) | Method, apparatus, system, device, and storage medium for answering knowledge questions | |
| US11537448B1 (en) | Adapting application programming interfaces with schema mappings | |
| US20170270184A1 (en) | Methods and devices for processing objects to be searched | |
| CN113239039B (en) | Dynamic data storage method, query method, management method and management system | |
| CN115291967A (en) | Space data analysis method and device and electronic equipment | |
| CN111125199B (en) | Database access method and device and electronic equipment | |
| CN111198898A (en) | Big data query method and big data query device | |
| JP5044943B2 (en) | Method and system for high-speed encoding of data documents | |
| CN107729428A (en) | A kind of SQL query method based on Presto and Elasticsearch | |
| CN113760961A (en) | Data query method and device | |
| CN101719162A (en) | Multi-version open geographic information service access method and system based on fragment pattern matching | |
| CN109697234B (en) | Multi-attribute information query method, device, server and medium for entity | |
| CN116501761A (en) | Query optimization method, device and storage medium | |
| CN115936880A (en) | Real-time stock information transaction data processing system | |
| CN116010461A (en) | Data blood relationship analysis method and device, storage medium and electronic equipment | |
| CN110968594A (en) | Database query optimization method, engine and storage medium | |
| CN111352963A (en) | Data statistical method and device | |
| JP2006221655A (en) | Method and system for compiling schema | |
| CN107220327A (en) | Data query method and system based on MongoDB, service terminal, memory | |
| US20040015780A1 (en) | Position-independent access to data elements in an electronic document | |
| CN111695031A (en) | Label-based searching method, device, server and storage medium | |
| CN111538796A (en) | Address normalization processing method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |