CN111985194A - Data storage method and device, electronic equipment and storage medium - Google Patents
Data storage method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN111985194A CN111985194A CN202010916524.0A CN202010916524A CN111985194A CN 111985194 A CN111985194 A CN 111985194A CN 202010916524 A CN202010916524 A CN 202010916524A CN 111985194 A CN111985194 A CN 111985194A
- Authority
- CN
- China
- Prior art keywords
- data
- slave
- data table
- tables
- table set
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/10—Text processing
- G06F40/166—Editing, e.g. inserting or deleting
- G06F40/174—Form filling; Merging
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
Abstract
The invention relates to the technical field of data processing, and discloses a data storage method, which comprises the following steps: acquiring a data set to be stored and a corresponding data table set, and performing master-slave table classification on the data table set to obtain a master data table set and a slave data table set; calculating the data saturation of each main data table in the main data table set, and selecting the main data table with the data saturation greater than a preset threshold value to obtain a target main data table set; merging the target main data tables in the target main data table set to generate one or more main data wide tables; merging the slave data tables in the slave data table set to generate one or more slave data wide tables; and executing data storage of the data set to be stored according to the master data width table and the slave data width table. The invention also provides a data storage device, an electronic device and a storage medium. In addition, the invention also relates to a block chain technology, and the master data wide table and the slave data wide table can be stored in the block chain. The invention can improve the storage efficiency of data storage.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a data storage method and apparatus, an electronic device, and a computer-readable storage medium.
Background
With the increasing development of big data, how to perform efficient data storage is gradually developedImportantly, at present, in data storage, T is usually configured logically according to data1，T2，T3，T4，T5，T6.......TnThe data sheet stores the corresponding data transmitted by the system according to the configured data sheet, however, the method has the following disadvantages when the data storage is executed: at least one data storage operation needs to be performed on each data table, and because the number of the data tables related to the operation is large, a large amount of time is often spent on completing the storage of a complete piece of data, so that the storage efficiency of data storage is low.
Disclosure of Invention
The invention provides a data storage method, a data storage device, electronic equipment and a computer readable storage medium, and mainly aims to improve the storage efficiency of data storage.
In order to achieve the above object, the present invention provides a data storage method, including:
acquiring a data set to be stored and a corresponding data table set, and performing master-slave table classification on the data table set to obtain a master data table set and a slave data table set;
calculating the data saturation of each main data table in the main data table set, and selecting the main data table with the data saturation larger than a preset threshold value to obtain a target main data table set;
merging the target main data tables in the target main data table set to generate one or more main data wide tables;
merging the slave data tables in the slave data table set to generate one or more slave data wide tables;
and executing data storage of the data set to be stored according to the master data width table and the slave data width table.
Optionally, the performing a master-slave table classification on the data table set to obtain a master data table set and a slave data table set includes:
acquiring the data form of each data table in the data table set;
based on the data form, identifying the incidence relation among the data tables in the data table set;
and executing master-slave table division of the data table set according to the incidence relation to generate the master data table set and the slave data table set.
Optionally, the calculating the data saturation of each main data table in the main data table set includes:
acquiring all fields in the main data table;
acquiring a storage data set contained in each field;
if the storage data set is identified to have illegal storage data, screening the illegal storage data from the storage data set;
and calculating the saturation of the storage data set by using a preset saturation calculation formula to obtain the corresponding field saturation, and calculating the data saturation corresponding to the main data table according to the field saturation.
Optionally, the preset saturation calculation formula includes:
P＝n/m*100％
wherein P represents the saturation of the stored data set, n represents the number of stored data in the stored data set, and m represents the total number of stored data contained in the field in the main data table.
Optionally, the merging the target master data tables in the target master data table set includes:
and acquiring the field quantity contained in each target main data table in the target main data table set, selecting the target main data tables of which the field quantities are added and are less than the preset field quantities, and merging the selected target main data tables.
Optionally, the merging the slave data tables in the slave data table set to obtain a slave data wide table set includes:
selecting slave data tables with the same fields from the slave data table set, wherein the number of the fields exceeds the preset number of the fields;
and configuring a slave data width table according to the selected field type in the slave data table to obtain the slave data width table set.
Optionally, the configuring a slave data width table according to the selected field type in the slave data table includes:
and according to the field type, performing union processing on the fields in the selected slave data table to obtain the number of fields required in the slave data table to be configured, and executing the configuration of the slave data table according to the number of the required fields.
In order to solve the above problems, the present invention also provides a data storage device, comprising:
the classification module is used for acquiring a data set to be stored and a corresponding data table set, and performing master-slave table classification on the data table set to generate a master data table set and a slave data table set;
the selecting module is used for calculating the data saturation of each main data table in the main data table set, selecting the main data table with the data saturation larger than a preset threshold value and obtaining a target main data table set;
the merging module is used for merging the target main data tables in the target main data table set to generate one or more main data wide tables;
the merging module is further configured to merge the slave data tables in the slave data table set to generate one or more slave data wide tables;
and the storage module is used for executing data storage of the data set to be stored according to the master data width table and the slave data width table.
In order to solve the above problem, the present invention also provides an electronic device, including:
at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform a data storage method as claimed in any one of claims 1 to 7.
In order to solve the above problem, the present invention further provides a computer-readable storage medium, in which a computer program is stored, and the computer program realizes the above data storage method when being executed by a processor.
The embodiment of the invention firstly obtains the data set to be stored and the corresponding data table set, and carries out master-slave table classification on the data table set to generate the master data table set and the slave data table set, thereby being beneficial to reducing the storage pressure brought to a background database in the subsequent data storage process; secondly, in the embodiment of the present invention, the data saturation of all fields in the master data table set is calculated, the master data table with the data saturation greater than the preset threshold is selected, the selected master data table sets are merged to generate one or more master data wide tables, the slave data tables in the slave data table sets are merged to obtain one or more slave data wide tables, and the number of the master data tables in the master data table set and the number of the slave data tables in the slave data table sets can be compressed, so that the storage time of the master data and the slave data in the data to be stored can be improved. Therefore, the data storage method, the data storage device, the electronic equipment and the storage medium can improve the storage efficiency of data storage.
Drawings
Fig. 1 is a schematic flow chart of a data storage method according to an embodiment of the present invention;
FIG. 2 is a detailed flowchart of the step S1 of the data storage method provided in FIG. 1 according to an embodiment of the present invention;
FIG. 3 is a detailed flowchart of the step S2 of the data storage method provided in FIG. 1 according to an embodiment of the present invention;
FIG. 4 is a detailed flowchart of the step S4 of the data storage method provided in FIG. 1 according to an embodiment of the present invention;
FIG. 5 is a block diagram of a data storage device according to an embodiment of the present invention;
fig. 6 is a schematic internal structural diagram of an electronic device implementing a data storage method according to an embodiment of the present invention;
the implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.
Detailed Description
It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
The execution subject of the data storage method provided by the embodiment of the present application includes, but is not limited to, at least one of electronic devices, such as a server and a terminal, which can be configured to execute the method provided by the embodiment of the present application. In other words, the data storage method may be performed by software or hardware installed in the terminal device or the server device, and the software may be a block chain platform. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, and the like.
Referring to fig. 1, a schematic flow chart of a data storage method according to an embodiment of the present invention is shown. In an embodiment of the present invention, the data storage method includes:
s1, acquiring a data set to be stored and a corresponding data table set, and performing master-slave table classification on the data table set to obtain a master data table set and a slave data table set.
In a preferred embodiment of the present invention, the data set to be stored includes data generated by a user interacting with a front-end system, such as: the data table set refers to data tables for summarizing the data to be stored, such as a student basic information data table, a student achievement table, a student behavior data table and the like.
In a preferred embodiment, the data set to be stored and the corresponding data table set are obtained by using a preset data obtaining script. Preferably, the preset data acquisition script can be compiled through a JavaScript scripting language.
Further, since the data to be stored has an inclusion relationship, that is, one data exists in a manner of being attached to another data, for example, the data to be stored is student performance, and it may include data including: the comprehensive achievement and the single department achievement, wherein the comprehensive achievement comprises: the simple achievement comprises the following components: the system comprises a Chinese score, a mathematic score, an English score and the like, so that the student score can be used as main data, and the comprehensive score and the single-subject score can be used as auxiliary data of the student score; or the comprehensive achievement is used as the master data, the diathesis expansion achievement, the skill achievement and the spoken achievement are used as the slave data of the comprehensive achievement, the monadic achievement is used as the master data, the Chinese achievement, the mathematics achievement and the English achievement are used as the slave data of the monadic achievement, and the achievement is divided into the master data and the slave data, so that the storage of a large amount of repeated data in an achievement table can be avoided, and the storage efficiency of data storage is ensured.
In detail, referring to fig. 2, the performing the master-slave table classification on the data table set to obtain a master data table set and a slave data table set includes:
s10, acquiring the data form of each data table in the data table set;
s11, identifying the association relation among the data tables in the data table set based on the data form;
and S12, according to the incidence relation, executing master-slave table division of the data table set to obtain the master data table set and the slave data table set.
The data form is used for describing the data type of each data table, for example, the data type described by the student achievement data table is related to student achievement, and the data type described by the student basic information table is related to student basic information.
Further, in the embodiment of the present invention, based on the data form, an ER (Entity-Relationship Diagram) Diagram is used to identify an association Relationship between the data tables in the data table set, for example, an integrated score table and a spoken language score table, and the ER Diagram is used to identify an association Relationship between the integrated score table and the spoken language score table as an inclusion Relationship, that is, the integrated score table is a master table and the spoken language score table is a slave table.
Based on the generation of the master data table and the slave data table, the storage pressure brought to a background database in the subsequent data storage process is favorably reduced.
S2, calculating the data saturation of each main data table in the main data table set, and selecting the main data table with the data saturation greater than a preset threshold value to obtain a target main data table set.
Since there are many fields in the main data table set, and there may be illegal values for the data records corresponding to each field, for example, for the age field, there may be data records whose age is not input by the user or whose age input by the user is 200, so the embodiment of the present invention identifies the saturation of the corresponding data table by calculating the data saturation of all the fields in the main data table set.
Specifically, referring to fig. 3, the calculating the data saturation of each main data table in the main data table set includes:
s20, acquiring all fields in the main data table;
s21, acquiring a storage data set contained in each field;
for example, for the age field, 500 stored data are included.
S22, if the storage data set is identified to have illegal storage data, screening the illegal storage data from the storage data set;
for example, if there is stored data not filled with age in the 500 age stored data or there is illegally stored data filled with an age sample and not a number but other characters, the illegally stored data is filtered out from the 500 age stored data.
And S23, calculating the saturation of the stored data set by using a preset saturation calculation formula to obtain the corresponding field saturation, and calculating the data saturation of the corresponding main data table according to the field saturation.
In a preferred embodiment, the preset saturation calculation formula includes:
P＝n/m*100％
wherein P represents the saturation of the stored data set, n represents the number of stored data in the stored data set, and m represents the total number of stored data contained in the field in the main data table.
In a preferred embodiment, the calculating the data saturation of the corresponding main data table includes: and averaging the calculated saturation of the field to obtain the data saturation of the corresponding main data table.
In a preferred embodiment, the preset threshold is 10%, so that the present invention selects the main data table with a data saturation greater than 10%, to obtain the target main data table set.
And S3, merging the target main data tables in the target main data table set to generate one or more main data wide tables.
In a preferred embodiment of the present invention, merging of the target main data tables in the target main data table set is performed based on a preset field merging principle, so as to obtain one or more main data wide tables. And setting the preset field merging principle based on the number of fields contained in each data table in the target main data table set.
Specifically, the executing the merging of the target main data tables in the target main data table set based on a preset field merging principle includes:
acquiring the field quantity contained in each target main data table in the target main data table set, selecting the target main data tables of which the field quantities are added and are less than the preset field quantity, and merging the selected target main data tables, wherein the preset field quantity is set according to the user requirements.
Illustratively, in the target master data table set, there is T1-4、T2-3、T3-5、T4-6、T5-10And T6-7These 5 tables, where Tm-n is that table m has n fields, the preset field merging rule merges data tables with the number of fields added less than Q fields, e.g. 20 fields, and thus, the T is the sum of the T1-4、T2-3、T3-5、T4-6、T5-10、T6-7And (3) merging to obtain:
ta Table with 18 fields (T)1-4、T2-3、T3-5、T4-6The sum of the number of fields, i.e., 4+3+5+ 6);
tb Table, number of fields 17 (T)5-10、T6-7Sum of the number of fields, i.e., 10+ 7).
And the Ta table and the Tb table are generated main data width tables.
Based on the generation of the main data wide table set, the number of target main data tables in the target main data table set can be compressed, and therefore the storage time of main data in the data to be stored can be improved.
It should be emphasized that, in order to further ensure the privacy and security of the master data wide table, the master data wide table set may also be stored in a node of a block chain.
And S4, merging the slave data tables in the slave data table set to obtain one or more slave data wide tables.
Referring to fig. 4, the S4 includes:
s40, selecting the slave data tables with the same fields and the number of the fields exceeding the preset number from the slave data table set;
s41, configuring the slave data width table according to the selected field type in the slave data table to obtain one or more master data width tables.
Preferably, in the present invention, the preset number is configured based on user requirements and according to a field set included in each data table in the data table set, for example, there are 10 data tables in the data table set, and the number range of fields included in each data table is: if 15 to 20 slave data tables are configured in a preset number according to the user requirement, for example, if the user a requires to merge not less than 8 slave data tables with the same field, the preset number may be 7, and if the user B requires to merge not less than 12 slave data tables with the same field, the preset number may be 11.
Further, in a preferred embodiment of the present invention, the configuring the slave data width table set according to the selected field type in the data table includes: and according to the field type, performing union processing on the fields in the selected data table to obtain the number of the fields of the slave data table to be configured, and executing the configuration of the slave data table according to the number of the fields. For example, if the field type of the slave data table a includes 14 fields, and the field type of the slave data table B includes 16 fields, where the same field type of the slave data table a and the same field type of the slave data table B are 12 fields, the union processing is performed on the fields in the slave data table a and the slave data table B, and the number of the fields of the slave data table to be configured is 18, so that the number of the fields of the slave data width table C to be configured is 18.
Illustratively, if the slave data table existing in the slave data table set includes: the method comprises the following steps of obtaining a field set contained in a customer order data table, a customer address data table, a customer contact information data table and the like, wherein the field set comprises the following steps: id. name, date, age, and order; the field set contained in the client address data table comprises id, name, date, age and address; and the field sets contained in the customer contact data table comprise data, age, number and time, if the same number is preset to be 3 according to the number of fields contained in each slave data table, two data tables of a customer order data table and a customer address data table are selected, the number of fields of the customer order data table and the customer address data table is summarized to be 6, and the number of fields of the data width table of the customer order data table and the customer address data table is obtained to be 6.
Based on the implementation means, the number of the slave data tables in the slave data table set can be compressed, so that the storage time of the slave data in the data to be stored can be prolonged.
It should be emphasized that, in order to further ensure the privacy and security of the master data wide table, the master data wide table set may also be stored in a node of a block chain.
And S5, executing data storage of the data set to be stored according to the master data width table and the slave data width table.
In a preferred embodiment of the present invention, the data storage of the data set to be stored is executed according to the master data width table and the slave data width table, that is, the data in the data set to be stored is stored into the corresponding field records of the master data width table and the slave data width table.
In a preferred embodiment, the data storage of the data set to be stored may be implemented by using currently known index storage methods.
In summary, in the embodiments of the present invention, a data set to be stored and a corresponding data table set are first obtained, and the data table set is subjected to master-slave table classification to generate a master data table set and a slave data table set, which is beneficial to reducing storage pressure on a background database in subsequent data storage; secondly, in the embodiment of the present invention, the data saturation of all fields in the master data table set is calculated, the master data table with the data saturation greater than the preset threshold is selected, the selected master data table sets are merged to generate one or more master data wide tables, the slave data tables in the slave data table sets are merged to obtain one or more slave data wide tables, and the number of the master data tables in the master data table set and the number of the slave data tables in the slave data table sets can be compressed, so that the storage time of the master data and the slave data in the data to be stored can be improved. Therefore, the data storage method provided by the invention can improve the storage efficiency of data storage.
FIG. 5 is a functional block diagram of a data storage device according to the present invention.
The data storage device 100 of the present invention may be installed in an electronic device. Depending on the implemented functionality, the data storage device 100 may include a classification module 101, a selection module 102, a merging module 103, and a storage module 104. A module according to the present invention, which may also be referred to as a unit, refers to a series of computer program segments that can be executed by a processor of an electronic device and that can perform a fixed function, and that are stored in a memory of the electronic device.
In the present embodiment, the functions regarding the respective modules/units are as follows:
the classification module 101 is configured to obtain a data set to be stored and a corresponding data table set, perform master-slave table classification on the data table set, and generate a master data table set and a slave data table set.
In a preferred embodiment of the present invention, the data set to be stored includes data generated by a user interacting with a front-end system, such as: the data table set refers to data tables for summarizing the data to be stored, such as a student basic information data table, a student achievement table, a student behavior data table and the like.
In a preferred embodiment, the data set to be stored and the corresponding data table set are obtained by using a preset data obtaining script. Preferably, the preset data acquisition script can be compiled through a JavaScript scripting language.
Further, since the data to be stored has an inclusion relationship, that is, one data exists in a manner of being attached to another data, for example, the data to be stored is student performance, and it may include data including: the comprehensive achievement and the single department achievement, wherein the comprehensive achievement comprises: the simple achievement comprises the following components: the system comprises a Chinese score, a mathematic score, an English score and the like, so that the student score can be used as main data, and the comprehensive score and the single-subject score can be used as auxiliary data of the student score; or the comprehensive achievement is used as the master data, the diathesis expansion achievement, the skill achievement and the spoken achievement are used as the slave data of the comprehensive achievement, the monadic achievement is used as the master data, the Chinese achievement, the mathematics achievement and the English achievement are used as the slave data of the monadic achievement, and the achievement is divided into the master data and the slave data, so that the storage of a large amount of repeated data in an achievement table can be avoided, and the storage efficiency of data storage is ensured.
In detail, the classification module 101 performs a master-slave table classification on the data table set to generate a master data table set and a slave data table set, including:
step A, acquiring the data form of each data table in the data table set;
b, identifying the incidence relation among the data tables in the data table set based on the data form;
and step C, executing master-slave table division of the data table set according to the incidence relation, and generating a master data table set and a slave data table set.
The data form is used for describing the data type of each data table, for example, the data type described by the student achievement data table is related to student achievement, and the data type described by the student basic information table is related to student basic information.
Further, in the embodiment of the present invention, based on the data form, the ER map is used to identify the association relationship between the data tables in the data table set, for example, the comprehensive result table and the spoken language result table, and the ER map is used to identify the association relationship between the comprehensive result table and the spoken language result table as the inclusion relationship, that is, the comprehensive result table is the master table and the spoken language result table is the slave table.
Based on the generation of the master data table and the slave data table, the storage pressure brought to a background database in the subsequent data storage process is favorably reduced.
The selecting module 102 is configured to calculate a data saturation of each main data table in the main data table set, and select a main data table with a data saturation greater than a preset threshold, so as to obtain a target main data table set.
Since there are many fields in the main data table set, and there may be illegal values for the data records corresponding to each field, for example, for the age field, there may be data records whose age is not input by the user or whose age input by the user is 200, so the embodiment of the present invention identifies the saturation of the corresponding data table by calculating the data saturation of all the fields in the main data table set.
Specifically, the calculating, by the selecting module 102, the data saturation of each main data table in the main data table set includes:
step a, acquiring all fields in the main data table;
b, acquiring a storage data set contained in each field;
for example, for the age field, 500 stored data are included.
Step c, if the storage data set is identified to have illegal storage data, screening the illegal storage data from the storage data set;
for example, if there is stored data that is not filled with age in the 500 age stored data or there is illegally stored data that is filled with an age sample and is not a number but other characters, the illegally stored data is filtered out from the 500 age stored data.
Step d: and calculating the saturation of the storage data set by using a preset saturation calculation formula to obtain the corresponding field saturation, and calculating the data saturation corresponding to the main data table according to the field saturation.
In a preferred embodiment, the preset saturation calculation formula includes:
P＝n/m*100％
wherein P represents the saturation of the stored data set, n represents the number of stored data in the stored data set, and m represents the total number of stored data contained in the field in the main data table.
In a preferred embodiment, the calculating the data saturation of the corresponding main data table includes: and averaging the calculated saturation of the field to obtain the data saturation of the corresponding main data table.
In a preferred embodiment, the preset threshold is 10%, so that the present invention selects the main data table with a data saturation greater than 10%, to obtain the target main data table set.
The merging module 103 is configured to merge the target main data tables in the target main data table set to generate one or more main data wide tables.
In a preferred embodiment of the present invention, merging of the target main data tables in the target main data table set is performed based on a preset field merging principle, so as to obtain one or more main data wide tables. And setting the preset field merging principle based on the number of fields contained in each data table in the target main data table set.
Specifically, the merging module 103 executes merging of the target main data tables in the target main data table set based on a preset field merging principle, including:
acquiring the field quantity contained in each target main data table in the target main data table set, selecting the target main data tables of which the field quantities are added and are less than the preset field quantity, and merging the selected target main data tables, wherein the preset field quantity is set according to the user requirements.
Illustratively, in the target master data table set, there is T1-4、T2-3、T3-5、T4-6、T5-10And T6-7These 5 tables, where Tm-n is that table m has n fields, the preset field merging rule merges data tables with the number of fields added less than Q fields, e.g. 20 fields, and thus, the T is the sum of the T1-4、T2-3、T3-5、T4-6、T5-10、T6-7And (3) merging to obtain:
ta Table with 18 fields (T)1-4、T2-3、T3-5、T4-6The sum of the number of fields, i.e., 4+3+5+ 6);
tb Table, number of fields 17 (T)5-10、T6-7Sum of the number of fields, i.e., 10+ 7).
And the Ta table and the Tb table are generated main data width tables.
Based on the generation of the main data wide table set, the number of target main data tables in the target main data table set can be compressed, and therefore the storage time of main data in the data to be stored can be improved.
It should be emphasized that, in order to further ensure the privacy and security of the master data wide table, the master data wide table set may also be stored in a node of a block chain.
The merging module 103 is further configured to merge the slave data tables in the slave data table set to obtain one or more slave data width tables.
In a preferred embodiment of the present invention, the merging module 103 merges the slave data tables in the slave data table set to obtain one or more slave data wide tables, including:
step I, selecting slave data tables with the same fields from the slave data table set, wherein the number of the fields exceeds the preset number;
and step II, configuring a slave data width table according to the selected field type in the slave data table to obtain one or more master data width tables.
Preferably, in the present invention, the preset number is configured based on user requirements and according to a field set included in each data table in the data table set, for example, there are 10 data tables in the data table set, and the number range of fields included in each data table is: if 15 to 20 slave data tables are configured in a preset number according to the user requirement, for example, if the user a requires to merge not less than 8 slave data tables with the same field, the preset number may be 7, and if the user B requires to merge not less than 12 slave data tables with the same field, the preset number may be 11.
Further, in a preferred embodiment of the present invention, the configuring the slave data width table set according to the selected field type in the data table includes: and according to the field type, performing union processing on the fields in the selected data table to obtain the number of the fields of the slave data table to be configured, and executing the configuration of the slave data table according to the number of the fields. For example, if the field type of the slave data table a includes 14 fields, and the field type of the slave data table B includes 16 fields, where the same field type of the slave data table a and the same field type of the slave data table B are 12 fields, the union processing is performed on the fields in the slave data table a and the slave data table B, and the number of the fields of the slave data table to be configured is 18, so that the number of the fields of the slave data width table C to be configured is 18.
Illustratively, if the slave data table existing in the slave data table set includes: the method comprises the following steps of obtaining a field set contained in a customer order data table, a customer address data table, a customer contact information data table and the like, wherein the field set comprises the following steps: id. name, date, age, and order; the field set contained in the client address data table comprises id, name, date, age and address; and the field sets contained in the customer contact data table comprise data, age, number and time, if the same number is preset to be 3 according to the number of fields contained in each slave data table, two data tables of a customer order data table and a customer address data table are selected, the number of fields of the customer order data table and the customer address data table is summarized to be 6, and the number of fields of the data width table of the customer order data table and the customer address data table is obtained to be 6.
Based on the implementation means, the number of the slave data tables in the slave data table set can be compressed, so that the storage time of the slave data in the data to be stored can be prolonged.
It should be emphasized that, in order to further ensure the privacy and security of the master data wide table, the master data wide table set may also be stored in a node of a block chain.
The storage module 104 is configured to execute data storage of the data set to be stored according to the master data width table and the slave data width table.
In a preferred embodiment of the present invention, the data storage of the data set to be stored is executed according to the master data width table and the slave data width table, that is, the data in the data set to be stored is stored into the corresponding field records of the master data width table and the slave data width table.
In a preferred embodiment, the data storage of the data set to be stored may be implemented by using currently known index storage methods.
In summary, in the embodiments of the present invention, a data set to be stored and a corresponding data table set are first obtained, and the data table set is subjected to master-slave table classification to generate a master data table set and a slave data table set, which is beneficial to reducing storage pressure on a background database in subsequent data storage; secondly, in the embodiment of the present invention, the data saturation of all fields in the master data table set is calculated, the master data table with the data saturation greater than the preset threshold is selected, the selected master data table sets are merged to generate one or more master data wide tables, the slave data tables in the slave data table sets are merged to obtain one or more slave data wide tables, and the number of the master data tables in the master data table set and the number of the slave data tables in the slave data table sets can be compressed, so that the storage time of the master data and the slave data in the data to be stored can be improved. Therefore, the data storage device provided by the invention can improve the storage efficiency of data storage.
Fig. 6 is a schematic structural diagram of an electronic device implementing the data storage method according to the present invention.
The electronic device 1 may comprise a processor 10, a memory 11 and a bus, and may further comprise a computer program, such as a data storage program 12, stored in the memory 11 and executable on the processor 10.
The memory 11 includes at least one type of readable storage medium, which includes flash memory, removable hard disk, multimedia card, card-type memory (e.g., SD or DX memory, etc.), magnetic memory, magnetic disk, optical disk, etc. The memory 11 may in some embodiments be an internal storage unit of the electronic device 1, such as a removable hard disk of the electronic device 1. The memory 11 may also be an external storage device of the electronic device 1 in other embodiments, such as a plug-in mobile hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like, which are provided on the electronic device 1. Further, the memory 11 may also include both an internal storage unit and an external storage device of the electronic device 1. The memory 11 may be used not only to store application software installed in the electronic device 1 and various types of data, such as codes of data storage, but also to temporarily store data that has been output or is to be output.
The processor 10 may be composed of an integrated circuit in some embodiments, for example, a single packaged integrated circuit, or may be composed of a plurality of integrated circuits packaged with the same or different functions, including one or more Central Processing Units (CPUs), microprocessors, digital Processing chips, graphics processors, and combinations of various control chips. The processor 10 is a Control Unit (Control Unit) of the electronic device, connects various components of the electronic device by using various interfaces and lines, and executes various functions and processes data of the electronic device 1 by running or executing programs or modules (for example, executing data storage, etc.) stored in the memory 11 and calling data stored in the memory 11.
The bus may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is arranged to enable connection communication between the memory 11 and at least one processor 10 or the like.
Fig. 6 only shows an electronic device with components, and it will be understood by a person skilled in the art that the structure shown in fig. 6 does not constitute a limitation of the electronic device 1, and may comprise fewer or more components than shown, or a combination of certain components, or a different arrangement of components.
For example, although not shown, the electronic device 1 may further include a power supply (such as a battery) for supplying power to each component, and preferably, the power supply may be logically connected to the at least one processor 10 through a power management device, so as to implement functions of charge management, discharge management, power consumption management, and the like through the power management device. The power supply may also include any component of one or more dc or ac power sources, recharging devices, power failure detection circuitry, power converters or inverters, power status indicators, and the like. The electronic device 1 may further include various sensors, a bluetooth module, a Wi-Fi module, and the like, which are not described herein again.
Further, the electronic device 1 may further include a network interface, and optionally, the network interface may include a wired interface and/or a wireless interface (such as a WI-FI interface, a bluetooth interface, etc.), which are generally used for establishing a communication connection between the electronic device 1 and other electronic devices.
Optionally, the electronic device 1 may further comprise a user interface, which may be a Display (Display), an input unit (such as a Keyboard), and optionally a standard wired interface, a wireless interface. Alternatively, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, or the like. The display, which may also be referred to as a display screen or display unit, is suitable for displaying information processed in the electronic device 1 and for displaying a visualized user interface, among other things.
It is to be understood that the described embodiments are for purposes of illustration only and that the scope of the appended claims is not limited to such structures.
The data store 12 stored by the memory 11 in the electronic device 1 is a combination of instructions that, when executed in the processor 10, enable:
acquiring a data set to be stored and a corresponding data table set, and performing master-slave table classification on the data table set to generate a master data table set and a slave data table set;
calculating the data saturation of each main data table in the main data table set, and selecting the main data table with the data saturation larger than a preset threshold value to obtain a target main data table set;
merging the target main data tables in the target main data table set to generate one or more main data wide tables;
merging the slave data tables in the slave data table set to generate one or more slave data wide tables;
and executing data storage of the data set to be stored according to the master data width table and the slave data width table.
Specifically, the specific implementation method of the processor 10 for the instruction may refer to the description of the relevant steps in the embodiment corresponding to fig. 1, which is not described herein again.
Further, the integrated modules/units of the electronic device 1, if implemented in the form of software functional units and sold or used as separate products, may be stored in a non-volatile computer-readable storage medium. The computer-readable medium may include: any entity or device capable of carrying said computer program code, recording medium, U-disk, removable hard disk, magnetic disk, optical disk, computer Memory, Read-Only Memory (ROM).
In the embodiments provided in the present invention, it should be understood that the disclosed apparatus, device and method can be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the modules is only one logical functional division, and other divisions may be realized in practice.
The modules described as separate parts may or may not be physically separate, and parts displayed as modules may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment.
In addition, functional modules in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, or in a form of hardware plus a software functional module.
It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential attributes thereof.
The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference signs in the claims shall not be construed as limiting the claim concerned.
The block chain is a novel application mode of computer technologies such as distributed data storage, point-to-point transmission, a consensus mechanism, an encryption algorithm and the like. A block chain (Blockchain), which is essentially a decentralized database, is a series of data blocks associated by using a cryptographic method, and each data block contains information of a batch of network transactions, so as to verify the validity (anti-counterfeiting) of the information and generate a next block. The blockchain may include a blockchain underlying platform, a platform product service layer, an application service layer, and the like.
Furthermore, it is obvious that the word "comprising" does not exclude other elements or steps, and the singular does not exclude the plural. A plurality of units or means recited in the system claims may also be implemented by one unit or means in software or hardware. The terms second, etc. are used to denote names, but not any particular order.
Finally, it should be noted that the above embodiments are only for illustrating the technical solutions of the present invention and not for limiting, and although the present invention is described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that modifications or equivalent substitutions may be made on the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.
Claims (10)
1. A method of data storage, the method comprising:
acquiring a data set to be stored and a corresponding data table set, and performing master-slave table classification on the data table set to obtain a master data table set and a slave data table set;
calculating the data saturation of each main data table in the main data table set, and selecting the main data table with the data saturation larger than a preset threshold value to obtain a target main data table set;
merging the target main data tables in the target main data table set to generate one or more main data wide tables;
merging the slave data tables in the slave data table set to generate one or more slave data wide tables;
and executing data storage of the data set to be stored according to the master data width table and the slave data width table.
2. The data storage method of claim 1, wherein said performing a master-slave table classification on said set of data tables to obtain a master set of data tables and a slave set of data tables comprises:
acquiring the data form of each data table in the data table set;
based on the data form, identifying the incidence relation among the data tables in the data table set;
and executing master-slave table division of the data table set according to the incidence relation to generate the master data table set and the slave data table set.
3. The data storage method of claim 1, wherein said calculating a data saturation for each of the set of master data tables comprises:
acquiring all fields in the main data table;
acquiring a storage data set contained in each field;
if the storage data set is identified to have illegal storage data, screening the illegal storage data from the storage data set;
and calculating the saturation of the storage data set by using a preset saturation calculation formula to obtain the corresponding field saturation, and calculating the data saturation corresponding to the main data table according to the field saturation.
4. The data storage method of claim 3, wherein the preset saturation calculation formula comprises:
P＝n/m*100％
wherein P represents the saturation of the stored data set, n represents the number of stored data in the stored data set, and m represents the total number of stored data contained in the field in the main data table.
5. The data storage method of any one of claims 1 to 4, wherein said merging the target master data tables in the target master data table set comprises:
and acquiring the field quantity contained in each target main data table in the target main data table set, selecting the target main data tables of which the field quantities are added and are less than the preset field quantities, and merging the selected target main data tables.
6. The data storage method of claim 1, wherein said merging the slave data tables in the slave data table set to obtain a slave data wide table set comprises:
selecting slave data tables with the same fields from the slave data table set, wherein the number of the fields exceeds the preset number of the fields;
and configuring a slave data width table according to the selected field type in the slave data table to obtain the slave data width table set.
7. The data storage method of claim 6, wherein configuring a slave data wide table based on the selected field type in the slave data table comprises:
and according to the field type, performing union processing on the fields in the selected slave data table to obtain the number of fields required in the slave data table to be configured, and executing the configuration of the slave data table according to the number of the required fields.
8. A data storage device, characterized in that the device comprises:
the classification module is used for acquiring a data set to be stored and a corresponding data table set, and performing master-slave table classification on the data table set to obtain a master data table set and a slave data table set;
the selecting module is used for calculating the data saturation of each main data table in the main data table set, selecting the main data table with the data saturation larger than a preset threshold value and obtaining a target main data table set;
the merging module is used for merging the target main data tables in the target main data table set to generate one or more main data wide tables;
the merging module is further configured to merge the slave data tables in the slave data table set to generate one or more slave data wide tables;
and the storage module is used for executing data storage of the data set to be stored according to the master data width table and the slave data width table.
9. An electronic device, characterized in that the electronic device comprises:
at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform a data storage method as claimed in any one of claims 1 to 7.
10. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out a data storage method according to any one of claims 1 to 7.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010916524.0A CN111985194A (en) | 2020-09-03 | 2020-09-03 | Data storage method and device, electronic equipment and storage medium |
| PCT/CN2021/109483 WO2022048362A1 (en) | 2020-09-03 | 2021-07-30 | Data storage method and apparatus, electronic device, and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010916524.0A CN111985194A (en) | 2020-09-03 | 2020-09-03 | Data storage method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111985194A true CN111985194A (en) | 2020-11-24 |
Family
ID=73448448
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010916524.0A Pending CN111985194A (en) | 2020-09-03 | 2020-09-03 | Data storage method and device, electronic equipment and storage medium |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN111985194A (en) |
| WO (1) | WO2022048362A1 (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2022048362A1 (en) * | 2020-09-03 | 2022-03-10 | 深圳壹账通智能科技有限公司 | Data storage method and apparatus, electronic device, and storage medium |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114547314B (en) * | 2022-04-25 | 2022-07-05 | 北京安华金和科技有限公司 | Data classification and classification method and system based on master-slave table |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9870401B2 (en) * | 2014-04-17 | 2018-01-16 | Wisoncsin Alumni Research Foundation | Database system with highly denormalized database structure |
| CN110119413A (en) * | 2019-04-30 | 2019-08-13 | 京东城市(南京)科技有限公司 | The method and apparatus of data fusion |
| CN111459944B (en) * | 2020-04-07 | 2023-09-01 | 北京红山信息科技研究院有限公司 | MR data storage method, device, server and storage medium |
| CN111553133B (en) * | 2020-04-21 | 2023-04-07 | 招商局金融科技有限公司 | Report generation method and device, electronic equipment and storage medium |
| CN111985194A (en) * | 2020-09-03 | 2020-11-24 | 深圳壹账通智能科技有限公司 | Data storage method and device, electronic equipment and storage medium |
-
2020
- 2020-09-03 CN CN202010916524.0A patent/CN111985194A/en active Pending
-
2021
- 2021-07-30 WO PCT/CN2021/109483 patent/WO2022048362A1/en active Application Filing
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2022048362A1 (en) * | 2020-09-03 | 2022-03-10 | 深圳壹账通智能科技有限公司 | Data storage method and apparatus, electronic device, and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2022048362A1 (en) | 2022-03-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112052370A (en) | Message generation method and device, electronic equipment and computer readable storage medium | |
| CN111428458A (en) | Universal report generation method and device and computer readable storage medium | |
| CN112115143A (en) | Automatic data updating and synchronizing method and device, electronic equipment and storage medium | |
| CN112115145A (en) | Data acquisition method and device, electronic equipment and storage medium | |
| CN112883042A (en) | Data updating and displaying method and device, electronic equipment and storage medium | |
| CN111694844A (en) | Enterprise operation data analysis method and device based on configuration algorithm and electronic equipment | |
| CN112579621A (en) | Data display method and device, electronic equipment and computer storage medium | |
| CN114881616A (en) | Business process execution method and device, electronic equipment and storage medium | |
| CN111985194A (en) | Data storage method and device, electronic equipment and storage medium | |
| CN112949278A (en) | Data checking method and device, electronic equipment and readable storage medium | |
| CN114979120A (en) | Data uploading method, device, equipment and storage medium | |
| CN112256783A (en) | Data export method and device, electronic equipment and storage medium | |
| CN114880368A (en) | Data query method and device, electronic equipment and readable storage medium | |
| CN113468175A (en) | Data compression method and device, electronic equipment and storage medium | |
| CN113868528A (en) | Information recommendation method and device, electronic equipment and readable storage medium | |
| CN113434542A (en) | Data relation identification method and device, electronic equipment and storage medium | |
| CN112948380A (en) | Data storage method and device based on big data, electronic equipment and storage medium | |
| CN112541688A (en) | Service data checking method and device, electronic equipment and computer storage medium | |
| CN114816371B (en) | Message processing method, device, equipment and medium | |
| CN114911479A (en) | Interface generation method, device, equipment and storage medium based on configuration | |
| CN115496166A (en) | Multitasking method and device, electronic equipment and storage medium | |
| CN114896164A (en) | Interface optimization method and device, electronic equipment and storage medium | |
| CN111553133B (en) | Report generation method and device, electronic equipment and storage medium | |
| CN114547011A (en) | Data extraction method and device, electronic equipment and storage medium | |
| CN115277469A (en) | Weak network visualization control method and device, electronic equipment and readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |