CN113806364A - Big data storage system and method - Google Patents
Big data storage system and method Download PDFInfo
- Publication number
- CN113806364A CN113806364A CN202110998811.5A CN202110998811A CN113806364A CN 113806364 A CN113806364 A CN 113806364A CN 202110998811 A CN202110998811 A CN 202110998811A CN 113806364 A CN113806364 A CN 113806364A
- Authority
- CN
- China
- Prior art keywords
- data
- data storage
- module
- storage structure
- control module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000013500 data storage Methods 0.000 title claims abstract description 305
- 238000000034 method Methods 0.000 title claims abstract description 37
- 230000005540 biological transmission Effects 0.000 claims abstract description 15
- 238000013524 data verification Methods 0.000 claims abstract description 12
- 210000001503 joint Anatomy 0.000 claims abstract description 3
- 239000013589 supplement Substances 0.000 claims description 8
- 230000001360 synchronised effect Effects 0.000 claims description 7
- 238000012545 processing Methods 0.000 description 16
- 230000008569 process Effects 0.000 description 6
- 238000012423 maintenance Methods 0.000 description 5
- 238000001914 filtration Methods 0.000 description 4
- 238000012986 modification Methods 0.000 description 4
- 230000004048 modification Effects 0.000 description 4
- 238000011161 development Methods 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 238000007781 pre-processing Methods 0.000 description 2
- 230000009467 reduction Effects 0.000 description 2
- 230000007547 defect Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 230000002035 prolonged effect Effects 0.000 description 1
- 238000012827 research and development Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a big data storage system and a big data storage method, and belongs to the field of data storage. The big data storage system includes: the intelligent control system comprises a data transmission interface, a switch control module, an intelligent control module, a data verification module and a data structure extension module; the data transmission interface is used for being in butt joint with an external system to acquire data transmitted by the external system; the switch control module is used for setting a switch control rule; the intelligent control module is used for controlling the data verification module to verify the data; and the data structure extension module is used for increasing the data items of the current data storage structure under the condition that the switch control rule does not need to strictly check the data according to the data storage structure and automatically extend the data storage structure and the data check module judges that the data items of the data exceed the range of the data items of the current data storage structure. The invention can intelligently realize the expansion of the data storage structure, has high efficiency, is not easy to make mistakes and has little influence on the storage service.
Description
Technical Field
The invention relates to the field of data storage, in particular to a big data storage system and a big data storage method.
Background
With the development of services, more and more data are stored, and it is often necessary to add new data items to an existing data storage structure, and in such a case, the data storage structure needs to be modified and the stored data needs to be refreshed, and the storage service often needs to be suspended and manually processed by corresponding research and development personnel. However, manual modification inevitably causes errors, and the errors usually prolong the modification time and affect the development of normal services.
Disclosure of Invention
Therefore, the technical problem to be solved by the embodiments of the present invention is to overcome the defects that manual processing is required to suspend a storage service when a storage data item of information needs to be added to an existing data storage structure, and the storage service is affected by a processing time that is prolonged due to a failure of the manual processing, so as to provide a large data storage system and method.
To this end, an embodiment of the present invention provides a big data storage system, which includes a data storage node, where the data storage node includes: the intelligent control system comprises a data transmission interface, a switch control module, an intelligent control module, a data verification module and a data structure extension module;
the data transmission interface is used for being in butt joint with an external system to acquire data transmitted by the external system;
the switch control module is used for setting a switch control rule, wherein the switch control rule comprises whether the data needs to be checked strictly according to a data storage structure and whether the data storage structure is automatically expanded;
the intelligent control module is used for controlling the data verification module to verify the data;
the data structure extension module is used for increasing the data items of the current data storage structure under the condition that the switch control rule does not need to check the data strictly according to the data storage structure and automatically extend the data storage structure and the data check module judges that the data items of the data exceed the data item range of the current data storage structure.
Optionally, the system further includes: a data storage module for storing data;
the intelligent control module is also used for controlling the data storage module and adjusting the storage structure of the historical storage data according to the data storage structure after the data items are added.
Optionally, the intelligent control module is further configured to write the data into the data storage module after the historical storage data in the data storage module is adjusted according to the data storage structure after the data items are added; and/or the presence of a gas in the gas,
the intelligent control module is further configured to output feedback information when the switch control rule includes that the data needs to be checked strictly according to a data storage structure and the data checking module determines that the data structure of the data is inconsistent with the current data storage structure, where the feedback information is used to indicate difference information between the data structure of the data and the current data storage structure; and/or the presence of a gas in the gas,
the intelligent control module is further configured to write the data into the data storage module when the switch control rule includes that the data needs to be checked strictly according to a data storage structure and the data checking module determines that the data structure of the data is consistent with the current data storage structure.
Optionally, the intelligent control module is further configured to, when the switch control rule includes that the data does not need to be strictly checked according to a data storage structure and the data checking module determines that the data lacks a data item of the current data storage structure, supplement the missing data item and data information corresponding to the missing data item for the data, and write the supplemented data into the data storage module for storing the data.
Optionally, the number of the data storage nodes is at least two, and the intelligent control module of a first data storage node in the at least two data storage nodes is further configured to send notification information to a second data storage node when the data structure extension module of the first data storage node increases a current data item of the data storage structure, where the notification information is used to indicate that the second data storage node needs to add the data item, and the second data storage node is another data storage node in the at least two data storage nodes except the first data storage node;
at least two data storage nodes are used as block chain link points to form a block chain network, and data stored on the data storage nodes can be mutually synchronized.
In addition, an embodiment of the present invention further provides a big data storage method, which is applied to any one of the above big data storage systems, and the method includes the following steps:
the data transmission interface acquires data transmitted by an external system;
the intelligent control module controls the data checking module to check the data according to a switch control rule set by the switch control module, wherein the switch control rule comprises whether the data needs to be checked strictly according to a data storage structure and whether the data storage structure is automatically expanded or not;
and the data structure extension module increases the data items of the current data storage structure under the condition that the switch control rule does not need to check the data strictly according to the data storage structure and automatically extend the data storage structure and the data check module judges that the data items of the data exceed the data item range of the current data storage structure.
Optionally, the big data storage system includes a data storage module for storing data; the method further comprises the following steps:
the intelligent control module controls the data storage module and adjusts the storage structure of the historical storage data according to the data storage structure after the data items are added.
Optionally, the method further includes:
after the historical storage data in the data storage module are adjusted according to the data storage structure after data items are added, the intelligent control module writes the data into the data storage module; and/or the presence of a gas in the gas,
the intelligent control module outputs feedback information under the condition that the switch control rule includes that the data needs to be checked strictly according to a data storage structure and the data checking module judges that the data structure of the data is inconsistent with the current data storage structure, wherein the feedback information is used for indicating difference information between the data structure of the data and the current data storage structure; and/or the presence of a gas in the gas,
and the intelligent control module writes the data into the data storage module under the condition that the switch control rule includes that the data needs to be checked strictly according to a data storage structure and the data checking module judges that the data structure of the data is consistent with the current data storage structure.
Optionally, the method further includes:
the intelligent control module is further configured to, when the switch control rule includes that the data does not need to be strictly checked according to a data storage structure and the data checking module determines that the data lacks a data item of the current data storage structure, supplement the missing data item and data information corresponding to the missing data item for the data, and write the supplemented data into the data storage module for storing the data.
Optionally, at least two data storage nodes in the big data storage system are used as block link points to form a block link network, and data stored on each data storage node can be synchronized with each other;
the method further comprises the following steps: the intelligent control module of a first data storage node in the at least two data storage nodes sends notification information to a second data storage node when the data structure extension module of the first data storage node increases the current data item of the data storage structure, wherein the notification information is used for indicating the data item which needs to be added by the second data storage node, and the second data storage node is the other data storage node except the first data storage node in the at least two data storage nodes.
The technical scheme of the embodiment of the invention has the following advantages:
1. in the embodiment of the invention, when the data storage structure needs to be expanded, manual processing is not needed any more, but the data storage structure is intelligently realized by the system, so that the labor cost is reduced, the influence caused by manual error is reduced, the unavailable time of the system is shortened, and the maintenance cost of the data storage system is reduced.
2. In the embodiment of the invention, the missing data items and the corresponding data information thereof can be automatically supplemented for the data transmitted by the external system, thereby improving the fault tolerance of the system and reducing the workload of manual maintenance.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.
Fig. 1 is a schematic block diagram of a specific example of a large data storage system in embodiment 1 of the present invention;
fig. 2 is a flowchart of a specific example of a big data storage method according to embodiment 2 of the present invention;
fig. 3 is a flowchart of another specific example of a big data storage method according to embodiment 2 of the present invention;
fig. 4 is a flowchart of another specific example of a big data storage method in embodiment 2 of the present invention.
Detailed Description
The technical solutions of the present invention will be described clearly and completely with reference to the accompanying drawings, and it should be understood that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In describing the present invention, it is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. The terms "comprises" and/or "comprising," when used in this specification, are intended to specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. The term "and/or" includes any and all combinations of one or more of the associated listed items. The terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," "outer," and the like are used in the orientation or positional relationship indicated in the drawings for convenience in describing the invention and for simplicity in description, and do not indicate or imply that the referenced device or element must have a particular orientation, be constructed and operated in a particular orientation, and are therefore not to be construed as limiting the invention. The terms "first," "second," and "third" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance. The terms "mounted," "connected," and "coupled" are to be construed broadly and may, for example, be fixedly coupled, detachably coupled, or integrally coupled; can be mechanically or electrically connected; the two elements can be directly connected, indirectly connected through an intermediate medium, or communicated with each other inside; either a wireless or a wired connection. The specific meanings of the above terms in the present invention can be understood in specific cases to those skilled in the art.
In addition, the technical features involved in the different embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.
Example 1
Referring to fig. 1, the present embodiment provides a big data storage system, which includes a data storage node 10, where the data storage node 10 includes: the system comprises a data transmission interface 101, a switch control module 102, an intelligent control module 103, a data checking module 104 and a data structure extension module 105;
the data transmission interface 101 is configured to interface with an external system to obtain data transmitted by the external system;
the data transmission interface 101 may also be used to provide data services to an external system; the data may be data in a JS Object Notation (JavaScript Object Notation, JSON for short, which is a lightweight data exchange format) format, or may be structured data or other semi-structured data such as extended markup language (XML for short, data in the language is data in a semi-structured format). The data may be various types of data, such as service data, and is not limited in this embodiment;
the switch control module 102 is configured to set a switch control rule, where the switch control rule includes whether to check the data strictly according to a data storage structure and whether to automatically expand the data storage structure;
the intelligent control module 103 is configured to control the data verification module 104 to verify the data;
the data structure extension module 105 is configured to, when the switch control rule includes that it is not necessary to check the data strictly according to a data storage structure and automatically extend the data storage structure, and the data check module 104 determines that the data item of the data exceeds the data item range of the current data storage structure, add the data item of the current data storage structure. Specifically, the added data items are the parts of the data items of the data beyond the current data storage structure.
In this embodiment, the data checking module 104 may be used for checking a data structure, checking integrity of a data item, and the like.
When the data storage structure needs to be expanded, manual processing is not needed any more, but system intelligence is used, so that the labor cost is reduced, the influence caused by manual errors is reduced, the time spent on system unavailability is reduced, and the maintenance cost of the data storage system is reduced.
Optionally, the system further includes: a data storage module for storing data;
the intelligent control module 103 is further configured to control the data storage module, and adjust a storage structure of the historical storage data according to the data storage structure after the data items are added. I.e. to refresh the historical stored data.
In this embodiment, the data storage module is used for storing data. The data may specifically be stored in a storage data table of the data storage module, and the data structure extension module 105 may not only add a storage data item to an existing storage data table, but also create a new storage data table.
In an alternative embodiment, when a data item needs to be added to the current data storage structure, the data structure extension module 105 may create a new storage data table, where the new storage data table is used to store the added data item, and the system automatically fills a portion of the new storage data table corresponding to the historical storage data, for example, the portion is filled to 0. By creating a new storage data table to store the added data items, the data storage table where the historical storage data are located does not need to be changed, and the workload of updating the historical data storage structure is reduced. The data storage module of this embodiment includes a plurality of hash slots for mapping different data items (the data includes data item information and data information), and the hash slots map the data item information to corresponding data information through a hash function, so as to ensure a corresponding relationship between the data item information and the data information.
Further, in this embodiment, instead of creating a new storage data table each time a data item needs to be added to the current data storage structure, the data structure extension module 105 first determines the load amount in all the existing storage data tables, and creates a new storage data table if the load amount in the existing storage data tables is greater than or equal to a preset threshold.
In addition, the number of the data storage modules in this embodiment is at least two, and the data stored in each data storage module is the same, in other words, there are at least two copies of the stored data, and each copy of the data is stored in a different data storage module. One of the at least two data storage modules is a master data storage module, the other data storage modules are slave data storage modules, when the system provides data service outwards, data are preferentially read from the master data storage module, and when the master data storage module breaks down, the data are read from the slave data storage modules.
Or the data storage module comprises at least two storage blocks, and the data stored in the data storage module has at least two copies which are respectively stored in different storage blocks. According to the embodiment, the reliability of data storage can be improved by storing data in multiple copies, and even if the data of one copy is lost or damaged, the data of other copies is not affected. One of the at least two storage blocks is a main storage block, the other storage blocks are auxiliary storage blocks, when the system provides data service outwards, the data are read from the main storage block preferentially, and when the main storage block fails, the data are read from the auxiliary storage block.
Optionally, the intelligent control module 103 is further configured to write the data into the data storage module after the historical storage data in the data storage module is adjusted according to the data storage structure after the data items are added; specifically, a temporary storage data table or a storage file may be added to the intelligent control module 103, and is used to store the data acquired by the data transmission interface 101, and after the corresponding data storage structure expansion and historical data refresh are completed, the temporarily stored data is imported into the data storage module;
and/or the presence of a gas in the gas,
the intelligent control module 103 is further configured to output feedback information when the switch control rule includes that the data needs to be checked strictly according to a data storage structure and the data checking module 104 determines that the data structure of the data is inconsistent with the current data storage structure, where the feedback information is used to indicate difference information between the data structure of the data and the current data storage structure; wherein the case that the data structure of the data is inconsistent with the current data storage structure comprises at least one of: 1. more data items of the data structure of the data than the data items of the current data storage structure (except for the data items which are more than the data items, the other data items are the same), 2, less data items of the data structure of the data than the data items of the current data storage structure (except for the data items which are less than the data items, the other data items are the same), and different data items of the data structure of the data than the data items of the current data storage structure;
and/or the presence of a gas in the gas,
the intelligent control module 103 is further configured to write the data into the data storage module when the switch control rule includes that the data needs to be checked strictly according to a data storage structure, and the data checking module 104 determines that the data structure of the data is consistent with the current data storage structure.
Optionally, the intelligent control module 103 is further configured to, when the switch control rule includes that the data does not need to be checked strictly according to a data storage structure and the data checking module 104 determines that the data lacks a data item of the current data storage structure, supplement the missing data item and data information corresponding to the missing data item for the data, and write the supplemented data into a data storage module for storing data.
In this embodiment, the system may also automatically supplement missing data items and corresponding data information for data transmitted from an external system, thereby improving the fault tolerance of the system and reducing the workload of manual maintenance.
Optionally, the number of the data storage nodes 10 is at least two, and the intelligent control module 103 of a first data storage node of the at least two data storage nodes 10 is further configured to send notification information to a second data storage node when the data structure extension module 105 of the first data storage node increases a current data item of the data storage structure, where the notification information is used to indicate that the second data storage node needs to increase the data item, and the second data storage node is another data storage node of the at least two data storage nodes 10 except the first data storage node;
at least two data storage nodes 10 are used as block chain link points to form a block chain network, and data stored on each data storage node 10 can be mutually synchronized.
Example 2
Referring to fig. 2, the present embodiment provides a big data storage method, which is applied to any one of the big data storage systems described in embodiment 1, and the method includes the following steps:
s1: the data transmission interface acquires data transmitted by an external system;
s2: the intelligent control module controls the data checking module to check the data according to a switch control rule set by the switch control module, wherein the switch control rule comprises whether the data needs to be checked strictly according to a data storage structure and whether the data storage structure is automatically expanded or not;
s3: and the data structure extension module increases the data items of the current data storage structure under the condition that the switch control rule does not need to check the data strictly according to the data storage structure and automatically extend the data storage structure and the data check module judges that the data items of the data exceed the data item range of the current data storage structure.
When the data storage structure needs to be expanded, manual processing is not needed any more, but system intelligence is used, so that the labor cost is reduced, the influence caused by manual errors is reduced, the time spent on system unavailability is reduced, and the maintenance cost of the data storage system is reduced.
Optionally, in this embodiment, before the intelligent control module controls the data verification module to verify the data according to the switch control rule set by the switch control module, the method further includes:
the intelligent control module is used for preprocessing data acquired by the data transmission interface from an external system, and the preprocessing comprises the following steps:
and performing data hiding shell extraction processing on the data, performing filtering processing on the data subjected to the data hiding shell extraction processing, specifically performing Kalman filtering processing, and performing data space dimensionality reduction processing on the data subjected to the filtering processing.
At present, in order to protect data from being illegally read or illegally modified, many data include a hidden shell, and in this embodiment, for reasons such as reducing storage cost and improving storage efficiency, it is necessary to perform hidden shell removal processing on data including the hidden shell to obtain the data itself. In addition, in order to further reduce the storage cost and improve the storage efficiency, the embodiment further performs filtering processing on the data to remove error information in the data. Moreover, at present, data types are increasing under many scenes (for example, internet scenes), structures are also increasing and complex, and in order to facilitate data storage and subsequent search and utilization of data, the embodiment also performs dimension reduction processing on the data to be stored.
Optionally, the big data storage system includes a data storage module for storing data; the method further comprises the following steps:
the intelligent control module controls the data storage module and adjusts the storage structure of the historical storage data according to the data storage structure after the data items are added.
Optionally, the method further includes:
after the historical storage data in the data storage module are adjusted according to the data storage structure after data items are added, the intelligent control module writes the data into the data storage module; and/or the presence of a gas in the gas,
the intelligent control module outputs feedback information under the condition that the switch control rule includes that the data needs to be checked strictly according to a data storage structure and the data checking module judges that the data structure of the data is inconsistent with the current data storage structure, wherein the feedback information is used for indicating difference information between the data structure of the data and the current data storage structure; and/or the presence of a gas in the gas,
and the intelligent control module writes the data into the data storage module under the condition that the switch control rule includes that the data needs to be checked strictly according to a data storage structure and the data checking module judges that the data structure of the data is consistent with the current data storage structure.
Optionally, the method further includes:
the intelligent control module is further configured to, when the switch control rule includes that the data does not need to be strictly checked according to a data storage structure and the data checking module determines that the data lacks a data item of the current data storage structure, supplement the missing data item and data information (or called a data value) corresponding to the missing data item for the data, and write the supplemented data into the data storage module for storing the data.
Optionally, at least two data storage nodes in the big data storage system are used as block link points to form a block link network, and data stored on each data storage node can be synchronized with each other;
referring to fig. 3, the method further includes:
s4: the intelligent control module of a first data storage node in the at least two data storage nodes sends notification information to a second data storage node when the data structure extension module of the first data storage node increases the current data item of the data storage structure, wherein the notification information is used for indicating the data item which needs to be added by the second data storage node, and the second data storage node is the other data storage node except the first data storage node in the at least two data storage nodes.
Optionally, the method further includes: and before the intelligent control module writes the data into the data storage module, generating a check value corresponding to the data according to the data. The specific process of generating the check value may be: respectively calculating the maximum common divisor after rounding up of the data information corresponding to each data item in the data, then calculating the average value of the maximum common divisor corresponding to each data item, judging whether the digit of the average value is greater than or equal to a preset value, if not, performing zero filling processing in front of the average value, and selecting the value of a preset position from the average value or the average value after zero filling as the check value, wherein the preset position comprises the position with the digit greater than the preset value and/or the position with the digit equal to the preset value.
If the intelligent control module of the first data storage node generates a check value corresponding to the data according to the data and then writes the data into the data storage module, the second data storage node generates the check value in the same way after acquiring the data synchronized by the first data storage node, and if the check value generated by the second data storage node is the same as the check value generated by the first data storage node, the second data storage node considers that the data is not tampered in the synchronization process and stores the data into the data storage module.
In addition, in order to reduce the time taken by the data verification module when comparing the data item of the data with the data item of the current data storage structure, the present embodiment may count in advance a first word count of the name of each data item of the current data storage structure and a second word count of the name of each data item of the data, determine whether a word count that is not the first word count exists in the second word count, and if so, consider that the data item of the data and the data item of the current data storage structure are not identical. If it is only necessary to determine that the data item of the data and the data item of the current data storage structure are not consistent, the flow proceeds so far. If the second word count does not have the word count not found in the first word count, or if the second word count does exist but does not match the first word count, the data items of the data are grouped according to the word count of the data item name to obtain a plurality of first groups, and the data items of the current data storage structure are grouped according to the word count of the data item name to obtain a plurality of second groups. And comparing the names of the data items in the first grouping and the second grouping with the same word number to obtain a comparison result. When the names of the data items in the first group and the second group with the same number of words are compared, the first word of the name of the data item can be compared first, if the names of the data items in the first group and the second group are different, the comparison is not required to be continued, if the names of the data items in the first group and the second group are the same, the comparison is continued, if the names of the data items in the second group and the second group are the same, the comparison is continued, and if the names of the data items in the second group and the second group are still the same, the comparison is not required to be continued … …. In other words, when the names of the data items in the first grouping and the second grouping with the same number of words are compared, the data items are compared one by one according to the sequence of the words in the names of the data items, if the data items are the same, the next word is continuously compared until the last word in the names, and if not, the comparison is finished.
Referring to fig. 4, a method for storing big data provided in this embodiment is illustrated below.
S401: after the big data storage task is started, the system (for example, the data verification module 104) parses the data structure of the JSON data transmitted from the external system.
S402: the system (for example, the intelligent control module 103 may be specifically configured) acquires the switch control rule set by the switch control module 102 and the current data storage structure;
s403: the system (which may be specifically the data verification module 104, for example) automatically compares the difference between the data structure of the JSON data and the current data storage structure.
S404: whether the switch control rule set by the switch control module 102 is to strictly check the data according to the data storage structure, if so, S405 is entered, otherwise, S407 is entered.
S405: the system (which may be the data checking module 104, for example) determines whether the data structure of the JSON data is consistent with the current data storage structure, and if so, proceeds to S411, otherwise, proceeds to S406.
S406: the system (for example, the data verification module 104) feeds back the specific situation that the data structure of the JSON data is inconsistent with the current data storage structure, and the process ends.
S407: the system (which may be the data checking module 104, for example) determines whether the data item in the data structure of the JSON data exceeds the data item range of the current data storage structure, if so, then S408 is entered, otherwise, S409 is entered.
S408: the system (for example, the data structure extension module 105 may specifically) automatically modifies the current data storage structure and adds the corresponding data item, and the system (the intelligent control module 103 may specifically control the data storage module) automatically refreshes the historical storage data.
S409: the system (for example, the data structure extension module 105) determines whether the data items in the data structure of the JSON data are fewer than the data items in the current data storage structure, if so, the process proceeds to S410, otherwise, the process proceeds to S411.
S410: the system (for example, the intelligent control module 103 may be specifically configured) automatically supplements the missing data items and the corresponding data information for the transmission data, and then proceeds to S411.
S411: the system (for example, the intelligent control module 103 may be specific) writes the JSON data into the data storage module, and the process ends.
It should be understood that the above examples are only for clarity of illustration and are not intended to limit the embodiments. Other variations and modifications will be apparent to persons skilled in the art in light of the above description. And are neither required nor exhaustive of all embodiments. And obvious variations or modifications therefrom are within the scope of the invention.
Claims (10)
1. A big data storage system, comprising a data storage node, the data storage node comprising: the intelligent control system comprises a data transmission interface, a switch control module, an intelligent control module, a data verification module and a data structure extension module;
the data transmission interface is used for being in butt joint with an external system to acquire data transmitted by the external system;
the switch control module is used for setting a switch control rule, wherein the switch control rule comprises whether the data needs to be checked strictly according to a data storage structure and whether the data storage structure is automatically expanded;
the intelligent control module is used for controlling the data verification module to verify the data;
the data structure extension module is used for increasing the data items of the current data storage structure under the condition that the switch control rule does not need to check the data strictly according to the data storage structure and automatically extend the data storage structure and the data check module judges that the data items of the data exceed the data item range of the current data storage structure.
2. The system of claim 1, further comprising: a data storage module for storing data;
the intelligent control module is also used for controlling the data storage module and adjusting the storage structure of the historical storage data according to the data storage structure after the data items are added.
3. The system of claim 2, wherein the intelligent control module is further configured to write the data into the data storage module after the adjustment of the history storage data in the data storage module according to the data storage structure after the data items are added is completed; and/or the presence of a gas in the gas,
the intelligent control module is further configured to output feedback information when the switch control rule includes that the data needs to be checked strictly according to a data storage structure and the data checking module determines that the data structure of the data is inconsistent with the current data storage structure, where the feedback information is used to indicate difference information between the data structure of the data and the current data storage structure; and/or the presence of a gas in the gas,
the intelligent control module is further configured to write the data into the data storage module when the switch control rule includes that the data needs to be checked strictly according to a data storage structure and the data checking module determines that the data structure of the data is consistent with the current data storage structure.
4. The system of claim 1, wherein the intelligent control module is further configured to, in a case that the switch control rule includes that the data does not need to be checked strictly according to a data storage structure and the data checking module determines that the data lacks a data item of a current data storage structure, supplement the data with the missing data item and corresponding data information, and write the supplemented data into a data storage module for storing data.
5. The system according to claim 1, wherein the number of the data storage nodes is at least two, and the intelligent control module of a first data storage node of the at least two data storage nodes is further configured to send notification information to a second data storage node when the data structure extension module of the first data storage node increases a current data item of the data storage structure, where the notification information is used to indicate that the second data storage node needs to increase the data item, and the second data storage node is another data storage node of the at least two data storage nodes except the first data storage node;
at least two data storage nodes are used as block chain link points to form a block chain network, and data stored on the data storage nodes can be mutually synchronized.
6. A big data storage method applied to the big data storage system of any one of claims 1-5, the method comprising the steps of:
the data transmission interface acquires data transmitted by an external system;
the intelligent control module controls the data checking module to check the data according to a switch control rule set by the switch control module, wherein the switch control rule comprises whether the data needs to be checked strictly according to a data storage structure and whether the data storage structure is automatically expanded or not;
and the data structure extension module increases the data items of the current data storage structure under the condition that the switch control rule does not need to check the data strictly according to the data storage structure and automatically extend the data storage structure and the data check module judges that the data items of the data exceed the data item range of the current data storage structure.
7. The method of claim 6, wherein the big data storage system comprises a data storage module for storing data; the method further comprises the following steps:
the intelligent control module controls the data storage module and adjusts the storage structure of the historical storage data according to the data storage structure after the data items are added.
8. The method of claim 7, further comprising:
after the historical storage data in the data storage module are adjusted according to the data storage structure after data items are added, the intelligent control module writes the data into the data storage module; and/or the presence of a gas in the gas,
the intelligent control module outputs feedback information under the condition that the switch control rule includes that the data needs to be checked strictly according to a data storage structure and the data checking module judges that the data structure of the data is inconsistent with the current data storage structure, wherein the feedback information is used for indicating difference information between the data structure of the data and the current data storage structure; and/or the presence of a gas in the gas,
and the intelligent control module writes the data into the data storage module under the condition that the switch control rule includes that the data needs to be checked strictly according to a data storage structure and the data checking module judges that the data structure of the data is consistent with the current data storage structure.
9. The method of claim 6, further comprising:
the intelligent control module is further configured to, when the switch control rule includes that the data does not need to be strictly checked according to a data storage structure and the data checking module determines that the data lacks a data item of the current data storage structure, supplement the missing data item and data information corresponding to the missing data item for the data, and write the supplemented data into the data storage module for storing the data.
10. The method of claim 5, wherein there are at least two data storage nodes in the big data storage system, at least two data storage nodes form a block chain network as block chain link points, and data stored on each data storage node can be synchronized with each other;
the method further comprises the following steps: the intelligent control module of a first data storage node in the at least two data storage nodes sends notification information to a second data storage node when the data structure extension module of the first data storage node increases the current data item of the data storage structure, wherein the notification information is used for indicating the data item which needs to be added by the second data storage node, and the second data storage node is the other data storage node except the first data storage node in the at least two data storage nodes.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110998811.5A CN113806364B (en) | 2021-08-28 | 2021-08-28 | Big data storage system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110998811.5A CN113806364B (en) | 2021-08-28 | 2021-08-28 | Big data storage system and method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113806364A true CN113806364A (en) | 2021-12-17 |
| CN113806364B CN113806364B (en) | 2023-12-22 |
Family
ID=78894285
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110998811.5A Active CN113806364B (en) | 2021-08-28 | 2021-08-28 | Big data storage system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113806364B (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105144162A (en) * | 2013-04-30 | 2015-12-09 | 西斯特米特公司 | Integration database framework |
| CN107203478A (en) * | 2017-06-21 | 2017-09-26 | 郑州云海信息技术有限公司 | A kind of method and device of data storage |
| CN108595664A (en) * | 2018-04-28 | 2018-09-28 | 尚谷科技（天津）有限公司 | A kind of agricultural data monitoring method under hadoop environment |
| CN109885581A (en) * | 2019-03-14 | 2019-06-14 | 苏州达家迎信息技术有限公司 | Synchronous method, device, equipment and the storage medium of database |
| CN111159153A (en) * | 2019-12-30 | 2020-05-15 | 北京三快在线科技有限公司 | Service data checking method and device, computer equipment and storage medium |
| CN112491989A (en) * | 2020-11-17 | 2021-03-12 | 平安普惠企业管理有限公司 | Data transmission method, device, equipment and storage medium |
| CN113132459A (en) * | 2021-03-09 | 2021-07-16 | 西安电子科技大学 | Distributed storage method, system, storage medium, information data processing terminal |
-
2021
- 2021-08-28 CN CN202110998811.5A patent/CN113806364B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105144162A (en) * | 2013-04-30 | 2015-12-09 | 西斯特米特公司 | Integration database framework |
| CN107203478A (en) * | 2017-06-21 | 2017-09-26 | 郑州云海信息技术有限公司 | A kind of method and device of data storage |
| CN108595664A (en) * | 2018-04-28 | 2018-09-28 | 尚谷科技（天津）有限公司 | A kind of agricultural data monitoring method under hadoop environment |
| CN109885581A (en) * | 2019-03-14 | 2019-06-14 | 苏州达家迎信息技术有限公司 | Synchronous method, device, equipment and the storage medium of database |
| CN111159153A (en) * | 2019-12-30 | 2020-05-15 | 北京三快在线科技有限公司 | Service data checking method and device, computer equipment and storage medium |
| CN112491989A (en) * | 2020-11-17 | 2021-03-12 | 平安普惠企业管理有限公司 | Data transmission method, device, equipment and storage medium |
| CN113132459A (en) * | 2021-03-09 | 2021-07-16 | 西安电子科技大学 | Distributed storage method, system, storage medium, information data processing terminal |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113806364B (en) | 2023-12-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP2919130A1 (en) | Method and system for synchronizing distributed database | |
| CN107729366A (en) | A kind of pervasive multi-source heterogeneous large-scale data synchronization system | |
| CN106254094A (en) | A kind of method of data synchronization and system | |
| CN104935481B (en) | Data reconstruction method based on redundancy scheme under a kind of distributed storage | |
| EP2493083B1 (en) | Method, device and service control point (scp) for data disaster tolerance preprocessing | |
| CN109918360A (en) | Database platform system, creation method, management method, equipment and storage medium | |
| CN108306955A (en) | A kind of car-mounted terminal interconnects cluster method on a large scale | |
| CN110033095A (en) | A kind of fault-tolerance approach and system of high-available distributed machine learning Computational frame | |
| CN110244936A (en) | Method and apparatus for supporting VISIO flow chart to be automatically imported DCS algorithm configuration software | |
| CN110351139B (en) | Multi-machine main-standby implementation method of power quality management system | |
| JP6511739B2 (en) | Redundant system and redundant method | |
| CN117061512A (en) | Distributed information management method and system based on big data | |
| CN111049886A (en) | Multi-region SDN controller data synchronization method, server and system | |
| CN113806364A (en) | Big data storage system and method | |
| CN107147708A (en) | A kind of peer-to-peer network redundant data balance method and system | |
| CN114328033A (en) | Method and device for keeping service configuration consistency of high-availability equipment group | |
| CN106502842B (en) | Data reconstruction method and system | |
| CN113553373A (en) | Data synchronization method and device, storage medium and electronic equipment | |
| CN110716828B (en) | Database real-time backup method | |
| EP1213651B1 (en) | Data mirroring restoration in a distributed system | |
| CN111506668B (en) | Robot cluster intelligent data synchronization method and system | |
| CN112231142B (en) | System backup recovery method, device, computer equipment and storage medium | |
| CN115587141A (en) | Database synchronization method and device | |
| CN114546978A (en) | Bitmap management method, system, equipment and medium for storage cluster | |
| CN113472891A (en) | SDN controller cluster data processing method, device and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20231130 Address after: 518066 Qianhai Free Trade Building 2601, No. 3048 Xinghai Avenue, Nanshan Street, Qianhai Shenzhen Hong Kong Cooperation Zone, Shenzhen, Guangdong Province Applicant after: Shenzhen Telian Intelligent Technology Co.,Ltd. Address before: 100027 floor 11, Qihao Beijing West Tower, No. 8, Xinyuan South Road, Chaoyang District, Beijing Applicant before: Teslan Technology Group Co.,Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |