CN112579605B - Data storage method, device, storage medium and server - Google Patents
Data storage method, device, storage medium and server Download PDFInfo
- Publication number
- CN112579605B CN112579605B CN202011545550.3A CN202011545550A CN112579605B CN 112579605 B CN112579605 B CN 112579605B CN 202011545550 A CN202011545550 A CN 202011545550A CN 112579605 B CN112579605 B CN 112579605B
- Authority
- CN
- China
- Prior art keywords
- target data
- data
- field
- target
- key
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000013500 data storage Methods 0.000 title claims abstract description 59
- 238000000034 method Methods 0.000 title claims abstract description 46
- 230000008859 change Effects 0.000 claims abstract description 21
- 238000004590 computer program Methods 0.000 claims description 3
- 239000002609 medium Substances 0.000 description 13
- 230000008569 process Effects 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 239000012120 mounting media Substances 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000008707 rearrangement Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention discloses a data storage method, a data storage device, a storage medium and a server. The method comprises the following steps: acquiring target data; filling target data with null field information based on non-null field information corresponding to the same key field, and adding identification information of whether the target data is change data or not according to filling operation; the field information is the same as the field information corresponding to other key fields of the target data to which the non-null field information belongs; combining target data with the same field information corresponding to all key fields in the target data after the filling operation is performed, and determining the type information of the combined target data according to the identification information and the source of each target data in the combined target data; and storing the combined target data according to the type information. By the scheme, the resource consumption of data storage can be reduced, and the availability of the locally stored data during data storage is ensured.
Description
Technical Field
The embodiment of the invention relates to the technical field of data processing, in particular to a data storage method, a data storage device, a storage medium and a server.
Background
With the development of computer technology, the amount of data to be processed by a system is continuously increasing, and the association relationship between data to be extracted is also increasing. Along with the increase of the data volume, the time required by the mode of storing the data in a whole amount is longer and longer after the whole amount of data is batched and the association relation is extracted; the way in which the existing data is purged and then full stored will result in the data being unusable for a long period of time.
Disclosure of Invention
The embodiment of the invention provides a data storage method, a data storage device, a storage medium and a server, which can reduce the resource consumption of data storage and ensure the availability of local stored data during data storage.
In a first aspect, an embodiment of the present invention provides a data storage method, including:
Acquiring target data; wherein the target data comprises at least one piece of external input data and at least one piece of local storage data;
filling target data with null field information based on non-null field information corresponding to the same key field, and adding identification information of whether the target data is change data or not according to filling operation; the field information is the same as the field information corresponding to other key fields of the target data to which the non-null field information belongs;
Combining target data with the same field information corresponding to all key fields in the target data after the filling operation is performed, and determining the type information of the combined target data according to the identification information of each target data in the combined target data and the source of each target data; wherein the source of the target data comprises external input data and local storage data;
and storing the combined target data according to the type information.
In a second aspect, an embodiment of the present invention further provides a data storage device, including:
The data acquisition module is used for acquiring target data; wherein the target data comprises at least one piece of external input data and at least one piece of local storage data;
The field filling module is used for filling target data with empty field information based on non-empty field information corresponding to the same key field, and adding identification information of whether the target data is change data or not according to filling operation; the field information is the same as the field information corresponding to other key fields of the target data to which the non-null field information belongs;
The data merging module is used for merging target data with the same field information corresponding to all key fields in the target data after the filling operation is performed, and determining the type information of the merged target data according to the identification information of each target data in the merged target data and the source of each target data; wherein the source of the target data comprises external input data and local storage data;
and the data storage module is used for storing the combined target data according to the type information.
In a third aspect, an embodiment of the present invention provides a computer readable storage medium, on which a computer program is stored, which when executed by a processor implements a data storage method according to the first aspect of the embodiment of the present invention.
In a fourth aspect, an embodiment of the present invention provides a server, including:
One or more processors;
a memory for storing one or more programs;
The one or more programs, when executed by the one or more processors, cause the one or more processors to implement the data storage method according to the first aspect of the embodiments of the present invention.
The data storage scheme provided by the embodiment of the invention acquires target data; wherein the target data comprises at least one piece of external input data and at least one piece of local storage data; filling target data with null field information based on non-null field information corresponding to the same key field, and adding identification information of whether the target data is change data or not according to filling operation; the field information is the same as the field information corresponding to other key fields of the target data to which the non-null field information belongs; combining target data with the same field information corresponding to all key fields in the target data after the filling operation is performed, and determining the type information of the combined target data according to the identification information of each target data in the combined target data and the source of each target data; wherein the source of the target data comprises external input data and local storage data; and storing the combined target data according to the type information. By adopting the technical scheme, the technical problems that the data is unavailable for a long time due to the large resource consumption during the full-volume data storage and the mode of clearing the existing data and then performing the full-volume data storage can be solved, the resource consumption of the data storage can be reduced, the availability of the locally stored data during the data storage is ensured, and the user experience is greatly improved.
Drawings
FIG. 1 is a flow chart of a data storage method according to a first embodiment of the present invention;
Fig. 2 is a flow chart of a data storage method according to a second embodiment of the present invention;
FIG. 3 is a schematic diagram of a data storage device according to a third embodiment of the present invention;
fig. 4 is a schematic structural diagram of a server according to a fifth embodiment of the present invention.
Detailed Description
The invention is described in further detail below with reference to the drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting thereof. It should be further noted that, for convenience of description, only some, but not all of the structures related to the present invention are shown in the drawings.
Before discussing exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although a flowchart depicts steps as a sequential process, many of the steps may be implemented in parallel, concurrently, or with other steps. Furthermore, the order of the steps may be rearranged. The process may be terminated when its operations are completed, but may have additional steps not included in the figures. The processes may correspond to methods, functions, procedures, subroutines, and the like.
Example 1
Fig. 1 is a schematic flow chart of a data storage method according to an embodiment of the present invention, where the method is applicable to a case where a server stores data, and may be executed by a data storage device, where the device may be implemented by software and/or hardware, and may be generally integrated in the server. As shown in fig. 1, the method includes:
s101, acquiring target data; wherein the target data includes at least one piece of external input data and at least one piece of locally stored data.
The external input data is data to be stored, and the external input data may be data transmitted from other servers or terminal devices, and the local storage data is data already stored in a local database. Optionally, acquiring the target data includes: at least one piece of external input data and at least one piece of locally stored data are read from the distributed file system based on different path information, respectively. In particular, the local storage data and the external input data are stored under different path files in a general distributed file system, so that at least one external input data and at least one local storage data can be read from the distributed file system based on different path information, and the obtained external input data and local storage data are used as target data. Optionally, the target data may also be identified based on the path information.
S102, filling target data with field information being empty based on non-empty field information corresponding to the same key field, and adding identification information of whether the target data is change data or not according to filling operation; and the field information of the null target data is the same as the field information corresponding to other key fields of the target data to which the non-null field information belongs.
In the embodiment of the invention, each item of target data may include field information corresponding to at least one key field, where the field information corresponding to the key field in the target data may be null field information or non-null field information. It will be appreciated that when the target data contains null field information, it may be indicated that the target data is incomplete. Specifically, each piece of target data obtained from the distributed file system may contain a plurality of key fields, but there may be some data in the target data that has some of these key fields and some data that has some other of these key fields. According to the specific conditions of the service and the data, the key fields can be considered to be the same piece of data as long as one field exists in the service, so that the target data with the field information being null can be backfilled based on the non-null field information corresponding to the same key field.
Specifically, the target data with the null field information is filled in based on the non-null field information corresponding to the same key field, wherein the target data with the null field information is the same as the field information corresponding to other key fields of the target data to which the non-null field information belongs. For example, the target data 1 and the target data 2 each include field information corresponding to four key fields of field a, field B, field C and field D, wherein the field information corresponding to the key field B in the target data 1 is null, the field information corresponding to the other key fields (field a, field C and field D) is non-null, the field information corresponding to the four key fields in the target data 2 is non-null, and the field information corresponding to the field a, field C and field D in the target data 1 and the target data 2 are the same, that is, the field information corresponding to the same key field except for the null field is the same in the target data 1 and the target data 2, and the null field information corresponding to the key field B in the target data 1 can be filled out based on the field information corresponding to the key field B in the target data 2.
In the embodiment of the invention, the identification information of whether the target data is changed data is added according to the filling operation, wherein the identification information is used for identifying whether the target data is changed data. Optionally, adding, according to a filling operation, identification information of whether the target data is change data, including: identifying the target data filled in by the non-empty field information as changed data; target data not filled in by the non-empty field information is identified as unchanged data. Specifically, when null field information in certain item target data is filled in by non-null field information corresponding to the same key field in other target data, marking the filled item target data as changed data; when the empty field information is not contained in a certain item of target data or the empty field information contained in a certain item of target data is not contained in other target data which is completely identical to other non-empty field information in the target data, the empty field information in the target data is not filled by the non-empty field information corresponding to the same key field in other target data, the target data is identified as unchanged data.
S103, merging target data with the same field information corresponding to all key fields in the target data after the filling operation is performed, and determining the type information of the merged target data according to the identification information of each target data in the merged target data and the source of each target data; wherein the source of the target data includes external input data and locally stored data.
The method includes the steps of determining whether at least two pieces of target data with identical field information corresponding to all key fields exist in all target data after filling operation is completed, and if yes, merging the target data with identical field information corresponding to all key fields, namely merging a plurality of pieces of target data into one piece of target data. And then, determining the type information of the combined target data according to the identification information of each target data in the combined target data and the source of each target data, wherein the source of the target data comprises whether the target data is external input data or local storage data. Optionally, the type information includes newly added data, unmodified data and modified data; determining type information of the combined target data according to identification information of each target data in the combined target data and a source of the target data, wherein the type information comprises the following steps: when the combined target data comprises local storage data and the identification information of the local storage data is changed data, determining the combined target data as modified data; when the combined target data comprises local storage data and the identification information of the local storage data is unchanged data, determining the combined target data as unmodified data; and when all the combined target data are the external input data, determining the combined target data as the newly added data.
Specifically, when one piece of target data is local storage data and the identification information is changed data in the multiple pieces of target data for merging, the merged target data is determined to be modified data; when one item of target data exists as local storage data and the identification information is unchanged data in the multiple item of target data used for merging, determining the merged target data as unmodified data; when the local storage data does not exist in the multi-item target data for merging, namely the multi-item target data for merging are all external input data, the merged target data are determined to be newly added data.
S104, storing the combined target data according to the type information.
In the embodiment of the invention, the storage mode of the combined target data is determined according to the type information of the combined target data. Optionally, storing the combined target data according to the type information includes: when the combined target data is modified data, replacing corresponding local storage data based on the combined target data; when the combined target data is newly added data, the combined target data is stored locally; and deleting the merged target data when the merged target data is unmodified data. Specifically, when the combined target data is modified data, it is indicated that the data corresponding to the combined target data is already stored locally, but the locally stored corresponding data needs to be updated based on the combined target data now, so that the corresponding locally stored data can be replaced based on the combined target data, that is, the locally stored original data corresponding to the combined target data is deleted, and the combined target data is stored. When the combined target data is newly added data, the fact that the data corresponding to the combined target data is not stored locally is indicated, so that the combined target data is directly stored, and a local database is enriched. When the combined target data is unmodified data, it indicates that the data completely identical to the combined target data is stored locally, and in order to avoid the repeated storage of the identical data, the combined target data can be deleted directly.
The data storage method provided by the embodiment of the invention acquires target data; wherein the target data comprises at least one piece of external input data and at least one piece of local storage data; filling target data with null field information based on non-null field information corresponding to the same key field, and adding identification information of whether the target data is change data or not according to filling operation; the field information is the same as the field information corresponding to other key fields of the target data to which the non-null field information belongs; combining target data with the same field information corresponding to all key fields in the target data after the filling operation is performed, and determining the type information of the combined target data according to the identification information of each target data in the combined target data and the source of each target data; wherein the source of the target data comprises external input data and local storage data; and storing the combined target data according to the type information. By adopting the technical scheme, the technical problems that the data is unavailable for a long time due to the large resource consumption during the full-volume data storage and the mode of clearing the existing data and then performing the full-volume data storage can be solved, the resource consumption of the data storage can be reduced, the availability of the locally stored data during the data storage is ensured, and the user experience is greatly improved.
In some embodiments, before filling out the target data whose field information is null based on the non-null field information corresponding to the same key field, the method further includes: determining at least one key field related to the target data, and taking any one key field in the at least one key field as a target key field; filling out target data with null field information based on non-null field information corresponding to the same key field, and adding identification information of whether the target data is change data according to filling operation, wherein the method comprises the following steps: grouping the target data based on the target key field; dividing target data with the same field information corresponding to the target key field into a group; for each group of target data, filling the target data with empty field information based on non-empty field information corresponding to the same key field, and adding identification information of whether the target data is change data or not according to filling operation; after merging target data with the same field information corresponding to all key fields in the target data after the filling operation is executed, determining type information of the merged target data according to identification information of each target data in the merged target data and sources of each target data, wherein the method comprises the following steps: after merging target data with the same field information corresponding to all key fields in the target data after the filling operation is executed, determining the type information of the merged target data according to the identification information of each target data in the merged target data and the source of each target data, and taking the merged target data as new target data; and taking the next key field in at least one key field related to the target data as a target key field, and returning to the step of grouping the target data based on the target key field until the key fields are respectively taken as target key fields. The advantage of this arrangement is that the merging of target data that actually belong to the same data can be performed quickly and accurately.
For example, the key fields related to different target data may be different, at least one key field related to all target data is determined, any one of the at least one key field related to the target data is taken as a target key field, and it is understood that one key field may be randomly selected as a target key field from a plurality of key fields related to the target data. Then, the target data is grouped based on the target key field, and the target data with the same field information corresponding to the target key field is divided into a group. And filling out the target data with the null field information based on the non-null field information corresponding to the same key field for each group of target data, wherein the field information is identical to the field information corresponding to other key fields before the null field is removed in the target data with the null field information. Then adding identification information of whether the target data is changed data or not according to filling operation, wherein the target data filled by the non-empty field information is identified as changed data; target data not filled in by the non-empty field information is identified as unchanged data. And then, merging target data with the same field information corresponding to all key fields in the target data after the filling operation is executed, and taking the merged target data as new target data. Then, the next key field of the plurality of key fields related to the target data is taken as a target key field, and the step of grouping the target data based on the target key field is returned until all the key fields are respectively taken as target key fields. For example, the total of the target data is 100, the total of the key fields related to the target data is 4, the target data can be grouped based on the first key field, then, for each group of target data, the target data with null field information is filled based on the non-null field information corresponding to the same key field, finally, after the target data with the same field information corresponding to all the key fields in the target data after the filling operation is performed is combined, the target data is changed from 100 to 80, then, the 80 item target data is used as new target data, the grouping, filling and combining processing is performed on the 80 item target data based on the second key field, and so on until the grouping, filling and combining processing is performed based on the last key field.
The target data is device information data, key fields related to the target data comprise data id, device card number and device card id, and the target data comprises two pieces of external input data and one piece of local storage data. Specifically, the three items of label data are respectively: target data 1: data id1, device card number 1, null value; target data 2: data id2, null value, device card number 1, device card id1; target data: data id3, device id2, device card number 2, device card id2. The target data 1 and the target data 2 are external input data, and the target data 3 is local storage data. If the data id is used as the target key field, because the field information corresponding to the target key field (data id) in the three item of target data is different, the three item of target data can be divided into three groups, and the filling operation of the empty field information can not be performed; if the device card number is used as the target key field, three pieces of target data may be divided into two groups based on the device card number, and since field information corresponding to the target key field (device card number) in the target data 1 and the target data 2 is the same, the target data 1 and the target data 2 may be used as 1 group and the target data 3 may be used as one group. The field information corresponding to the key field of the device id in the target data 2 is null, so that the field information corresponding to the device id in the target data 2 can be filled in based on the non-null field information (device id 1) corresponding to the key field of the device id in the target data 1; the field information corresponding to the key field of the device card id in the target data 1 is null, and thus, the field information corresponding to the device card id in the target data 1 can be filled in based on the non-null field information (device card id 1) corresponding to the key field of the device card id in the target data 2. The three filled-in item label data and the identification information thereof are as follows: target data 1: data id1, device card number 1, device card id1, identification information: externally inputting data and changing the data; target data: data id2, device id1, device card number 1, device card id1, identification information: storing data locally and changing the data; target data 3: data id3, device id2, device card number 2, device card id2, identification information: data is externally input, and unchanged data. Then, the target data 1 and the target data 2 after the completion of the filling operation may be subjected to the merging process.
Example two
Fig. 2 is a flowchart of a data storage method according to a second embodiment of the present invention, as shown in fig. 2, the method includes the following steps:
S201, at least one piece of external input data and at least one piece of local storage data are respectively read from the distributed file system based on different path information.
S202, determining at least one key field related to target data, and taking any one key field in the at least one key field as a target key field.
S203, grouping target data based on the target key field; and dividing the target data with the same field information corresponding to the target key field into a group.
S204, filling out target data with empty field information based on non-empty field information corresponding to the same key field for each group of target data, and adding identification information of whether the target data is change data or not according to filling operation; the target data with the null field information is the same as the field information corresponding to other key fields of the target data to which the non-null field information belongs.
Optionally, adding, for the target data, identification information of whether the target data is change data according to the filling operation includes: identifying the target data filled in by the non-empty field information as changed data; target data not filled in by the non-empty field information is identified as unchanged data.
S205, merging target data with the same field information corresponding to all key fields in the target data after the filling operation is performed, determining the type information of the merged target data according to the identification information of each target data in the merged target data and the source of each target data, and taking the merged target data as new target data.
The source of the target data comprises external input data and local storage data.
Optionally, the type information includes newly added data, unmodified data and modified data; determining type information of the combined target data according to identification information of each target data in the combined target data and a source of the target data, wherein the type information comprises the following steps: when the combined target data comprises local storage data and the identification information of the local storage data is changed data, determining the combined target data as modified data; when the combined target data comprises local storage data and the identification information of the local storage data is unchanged data, determining the combined target data as unmodified data; and when all the combined target data are the external input data, determining the combined target data as the newly added data.
S206, taking the next key field in at least one key field related to the target data as a target key field, and returning to S203.
S207, judging whether at least one key field related to the target data is respectively used as a target key field, if so, executing S208, otherwise, executing S206.
S208, storing the combined target data according to the type information.
Optionally, storing the combined target data according to the type information includes: when the combined target data is modified data, replacing corresponding local storage data based on the combined target data; when the combined target data is newly added data, the combined target data is locally stored; and deleting the merged target data when the merged target data is unmodified data.
The data storage method provided by the embodiment of the invention not only can solve the technical problem that the data is unavailable for a long time due to the large resource consumption when the data is stored in a full amount and the mode of firstly clearing the existing data and then carrying out the full amount storage, but also can reduce the resource consumption of the data storage, ensure the availability of the locally stored data during the data storage and greatly improve the user experience.
Example III
Fig. 3 is a schematic structural diagram of a data storage device according to a third embodiment of the present invention. As shown in fig. 3, the apparatus includes: a data acquisition module 301, a field filling module 302, a data merging module 303 and a data storage module 304, wherein:
a data acquisition module 301, configured to acquire target data; wherein the target data comprises at least one piece of external input data and at least one piece of local storage data;
The field filling module 302 is configured to fill in target data with field information being empty based on non-empty field information corresponding to the same key field, and add, to the target data, identification information about whether the target data is change data according to filling operation; the field information is the same as the field information corresponding to other key fields of the target data to which the non-null field information belongs;
The data merging module 303 is configured to merge target data with the same field information corresponding to all key fields in the target data after the filling operation is performed, and determine type information of the merged target data according to identification information of each target data in the merged target data and a source of each target data; wherein the source of the target data comprises external input data and local storage data;
And the data storage module 304 is configured to store the combined target data according to the type information.
The data storage device provided by the embodiment of the invention acquires target data; wherein the target data comprises at least one piece of external input data and at least one piece of local storage data; filling target data with null field information based on non-null field information corresponding to the same key field, and adding identification information of whether the target data is change data or not according to filling operation; the field information is the same as the field information corresponding to other key fields of the target data to which the non-null field information belongs; combining target data with the same field information corresponding to all key fields in the target data after the filling operation is performed, and determining the type information of the combined target data according to the identification information of each target data in the combined target data and the source of each target data; wherein the source of the target data comprises external input data and local storage data; and storing the combined target data according to the type information. By adopting the technical scheme, the technical problems that the data is unavailable for a long time due to the large resource consumption during the full-volume data storage and the mode of clearing the existing data and then performing the full-volume data storage can be solved, the resource consumption of the data storage can be reduced, the availability of the locally stored data during the data storage is ensured, and the user experience is greatly improved.
Optionally, the key field determining module is configured to determine at least one key field related to the target data before filling the target data with the field information being empty based on non-empty field information corresponding to the same key field, and take any one key field in the at least one key field as a target key field;
The field filling module is used for:
grouping the target data based on the target key field; dividing target data with the same field information corresponding to the target key field into a group;
for each group of target data, filling the target data with empty field information based on non-empty field information corresponding to the same key field, and adding identification information of whether the target data is change data or not according to filling operation;
the data merging module is used for:
After merging target data with the same field information corresponding to all key fields in the target data after the filling operation is executed, determining the type information of the merged target data according to the identification information of each target data in the merged target data and the source of each target data, and taking the merged target data as new target data;
and taking the next key field in at least one key field related to the target data as a target key field, and returning to the step of grouping the target data based on the target key field until the key fields are respectively taken as target key fields.
Optionally, adding, according to a filling operation, identification information of whether the target data is change data, including:
Identifying the target data filled in by the non-empty field information as changed data;
Target data not filled in by the non-empty field information is identified as unchanged data.
Optionally, the type information includes newly added data, unmodified data and modified data;
Determining type information of the combined target data according to identification information of each target data in the combined target data and a source of the target data, wherein the type information comprises the following steps:
when the combined target data comprises local storage data and the identification information of the local storage data is changed data, determining the combined target data as modified data;
when the combined target data comprises local storage data and the identification information of the local storage data is unchanged data, determining the combined target data as unmodified data;
and when all the combined target data are the external input data, determining the combined target data as the newly added data.
Optionally, the data storage module is configured to:
When the combined target data is modified data, replacing corresponding local storage data based on the combined target data;
when the combined target data is newly added data, the combined target data is stored locally;
and deleting the merged target data when the merged target data is unmodified data.
Optionally, the data acquisition module is configured to:
At least one piece of external input data and at least one piece of locally stored data are read from the distributed file system based on different path information, respectively.
Example IV
A fourth embodiment of the invention provides a storage medium containing computer-executable instructions for a data storage method when executed by a computer processor.
Storage media-any of various types of memory devices or storage devices. The term "storage medium" is intended to include: mounting media such as CD-ROM, floppy disk or tape devices; computer system memory or random access memory, such as DRAM, DDRRAM, SRAM, EDORAM, rambus (Rambus) RAM, or the like; nonvolatile memory such as flash memory, magnetic media (e.g., hard disk or optical storage); registers or other similar types of memory elements, etc. The storage medium may also include other types of memory or combinations thereof. In addition, the storage medium may be located in a first computer system in which the program is executed, or may be located in a second, different computer system connected to the first computer system through a network such as the internet. The second computer system may provide program instructions to the first computer for execution. The term "storage medium" may include two or more storage media that may reside in different locations (e.g., in different computer systems connected by a network). The storage medium may store program instructions (e.g., embodied as a computer program) executable by one or more processors.
Of course, the storage medium containing the computer executable instructions provided in the embodiments of the present invention is not limited to the data storage operation or the file retrieval operation described above, and may also perform the related operations in the data storage method provided in any embodiment of the present invention.
Example five
Fig. 4 is a schematic structural diagram of a server according to a fifth embodiment of the present invention. The server shown in fig. 4 is only an example, and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention. As shown in fig. 4, the server provided in the fifth embodiment of the present invention includes a processor 41, a memory 42, an input device 43 and an output device 44; the number of processors 41 in the server may be one or more, one processor 41 being taken as an example in fig. 4; the processor 41, memory 42, input device 43 and output device 44 in the server may be connected by a bus or other means, in fig. 4 by way of example.
The memory 42 is a computer readable storage medium, and may be used to store software programs, computer executable programs, and modules, such as program instructions/modules (e.g., the data acquisition module 301, the field filling module 302, the data merging module 303, and the data storage module 304 in the data storage device) corresponding to the bluetooth-based operating mode configuration method in the embodiment of the invention. The processor 41 executes various functional applications and file processing by running software programs, instructions and modules stored in the memory 42, for example, implementing the data storage method applied to a server provided by the embodiment of the present invention.
The memory 42 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, at least one application program required for functions; the storage data area may store data created according to the use of the server, etc. In addition, memory 42 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid-state storage device. In some examples, memory 42 may further comprise memory located remotely from processor 41, which may be connected to a server via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The input device 43 may be used to receive numeric or character information entered by a user to generate key signal inputs related to user settings and function control of the server. The output device 44 may include a display device such as a display screen.
The data storage device, the storage medium and the server provided in the above embodiments can execute the corresponding data storage method provided in the embodiments of the present invention, and have the corresponding functional modules and beneficial effects of executing the method. Technical details not described in detail in the above embodiments may be referred to the data storage and retrieval method provided in any embodiment of the present invention.
Note that the above is only a preferred embodiment of the present invention and the technical principle applied. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, while the invention has been described in connection with the above embodiments, the invention is not limited to the embodiments, but may be embodied in many other equivalent forms without departing from the spirit or scope of the invention, which is set forth in the following claims.
Claims (8)
1. A method of data storage, comprising:
Acquiring target data; wherein the target data comprises at least one piece of external input data and at least one piece of local storage data;
filling target data with null field information based on non-null field information corresponding to the same key field, and adding identification information of whether the target data is change data or not according to filling operation; the field information is the same as the field information corresponding to other key fields of the target data to which the non-null field information belongs;
Combining target data with the same field information corresponding to all key fields in the target data after the filling operation is performed, and determining the type information of the combined target data according to the identification information of each target data in the combined target data and the source of each target data; wherein the source of the target data comprises external input data and local storage data;
Storing the combined target data according to the type information;
before filling out the target data with the field information being empty based on the non-empty field information corresponding to the same key field, the method further comprises the following steps:
determining at least one key field related to the target data, and taking any one key field in the at least one key field as a target key field;
The filling of the target data with the field information being empty based on the non-empty field information corresponding to the same key field, and adding the identification information of whether the target data is the change data according to the filling operation, comprises the following steps:
grouping the target data based on the target key field; dividing target data with the same field information corresponding to the target key field into a group;
for each group of target data, filling the target data with empty field information based on non-empty field information corresponding to the same key field, and adding identification information of whether the target data is change data or not according to filling operation;
after merging the target data with the same field information corresponding to all key fields in the target data after the filling operation is performed, determining the type information of the merged target data according to the identification information of each target data in the merged target data and the source of each target data, wherein the method comprises the following steps:
After merging target data with the same field information corresponding to all key fields in the target data after the filling operation is executed, determining the type information of the merged target data according to the identification information of each target data in the merged target data and the source of each target data, and taking the merged target data as new target data;
and taking the next key field in at least one key field related to the target data as a target key field, and returning to the step of grouping the target data based on the target key field until the key fields are respectively taken as target key fields.
2. The method according to claim 1, wherein adding identification information of whether or not it is change data to the target data according to a filling operation includes:
Identifying the target data filled in by the non-empty field information as changed data;
Target data not filled in by the non-empty field information is identified as unchanged data.
3. The method of claim 2, wherein the type information includes newly added data, unmodified data, and modified data;
Determining type information of the combined target data according to identification information of each target data in the combined target data and a source of the target data, wherein the type information comprises the following steps:
when the combined target data comprises local storage data and the identification information of the local storage data is changed data, determining the combined target data as modified data;
when the combined target data comprises local storage data and the identification information of the local storage data is unchanged data, determining the combined target data as unmodified data;
and when all the combined target data are the external input data, determining the combined target data as the newly added data.
4. A method according to claim 3, wherein storing the merged target data in accordance with the type information comprises:
When the combined target data is modified data, replacing corresponding local storage data based on the combined target data;
when the combined target data is newly added data, the combined target data is stored locally;
and deleting the merged target data when the merged target data is unmodified data.
5. The method of claim 1, wherein obtaining the target data comprises:
At least one piece of external input data and at least one piece of locally stored data are read from the distributed file system based on different path information, respectively.
6. A data storage device, comprising:
The data acquisition module is used for acquiring target data; wherein the target data comprises at least one piece of external input data and at least one piece of local storage data;
The field filling module is used for filling target data with empty field information based on non-empty field information corresponding to the same key field, and adding identification information of whether the target data is change data or not according to filling operation; the field information is the same as the field information corresponding to other key fields of the target data to which the non-null field information belongs;
The data merging module is used for merging target data with the same field information corresponding to all key fields in the target data after the filling operation is performed, and determining the type information of the merged target data according to the identification information of each target data in the merged target data and the source of each target data; wherein the source of the target data comprises external input data and local storage data;
the data storage module is used for storing the combined target data according to the type information;
The key field determining module is used for determining at least one key field related to target data before filling target data with empty field information based on non-empty field information corresponding to the same key field, and taking any one key field in the at least one key field as a target key field;
The field filling module is used for:
grouping the target data based on the target key field; dividing target data with the same field information corresponding to the target key field into a group;
for each group of target data, filling the target data with empty field information based on non-empty field information corresponding to the same key field, and adding identification information of whether the target data is change data or not according to filling operation;
the data merging module is used for:
After merging target data with the same field information corresponding to all key fields in the target data after the filling operation is executed, determining the type information of the merged target data according to the identification information of each target data in the merged target data and the source of each target data, and taking the merged target data as new target data;
and taking the next key field in at least one key field related to the target data as a target key field, and returning to the step of grouping the target data based on the target key field until the key fields are respectively taken as target key fields.
7. A computer readable storage medium, on which a computer program is stored, characterized in that the program, when being executed by a processor, implements a data storage method according to any of claims 1-5.
8. A server, comprising:
One or more processors;
a memory for storing one or more programs;
When executed by the one or more processors, causes the one or more processors to implement the data storage method of any of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011545550.3A CN112579605B (en) | 2020-12-23 | 2020-12-23 | Data storage method, device, storage medium and server |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011545550.3A CN112579605B (en) | 2020-12-23 | 2020-12-23 | Data storage method, device, storage medium and server |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112579605A CN112579605A (en) | 2021-03-30 |
| CN112579605B true CN112579605B (en) | 2024-05-10 |
Family
ID=75139323
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011545550.3A Active CN112579605B (en) | 2020-12-23 | 2020-12-23 | Data storage method, device, storage medium and server |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112579605B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106599104A (en) * | 2016-11-29 | 2017-04-26 | 北京锐安科技有限公司 | Mass data association method based on redis cluster |
| CN109558400A (en) * | 2018-11-28 | 2019-04-02 | 北京锐安科技有限公司 | Data processing method, device, equipment and storage medium |
-
2020
- 2020-12-23 CN CN202011545550.3A patent/CN112579605B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106599104A (en) * | 2016-11-29 | 2017-04-26 | 北京锐安科技有限公司 | Mass data association method based on redis cluster |
| CN109558400A (en) * | 2018-11-28 | 2019-04-02 | 北京锐安科技有限公司 | Data processing method, device, equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112579605A (en) | 2021-03-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108985742B (en) | Transaction processing method and device and block chain system | |
| CN108647357B (en) | Data query method and device | |
| CN106649828B (en) | Data query method and system | |
| CN107301214B (en) | Data migration method and device in HIVE and terminal equipment | |
| CN102129425B (en) | The access method of big object set table and device in data warehouse | |
| CN103544153A (en) | Data updating method and system based on database | |
| US11934371B2 (en) | Data processing method and apparatus, and computer-readable storage medium | |
| CN107992492B (en) | Data block storage method, data block reading method, data block storage device, data block reading device and block chain | |
| CN109033365B (en) | Data processing method and related equipment | |
| CN104423982A (en) | Request processing method and device | |
| CN106897338A (en) | A kind of data modification request processing method and processing device for database | |
| CN113485781A (en) | Report generation method and device, electronic equipment and computer readable medium | |
| CN112860412B (en) | Service data processing method and device, electronic equipment and storage medium | |
| CN112597151A (en) | Data processing method, device, equipment and storage medium | |
| CN112579605B (en) | Data storage method, device, storage medium and server | |
| CN110928941A (en) | Data fragment extraction method and device | |
| CN113254399A (en) | Log query method and device, electronic equipment and medium | |
| US8296336B2 (en) | Techniques for efficient dataloads into partitioned tables using swap tables | |
| CN111158873A (en) | System state machine implementation method and device, storage medium and processor | |
| CN113127436B (en) | Data synchronization method, device, server and storage medium | |
| CN115455035B (en) | Data query model construction method and computer-readable storage medium | |
| CN117435589A (en) | Data transfer method, device, computer equipment and storage medium | |
| CN106815289B (en) | Data storage method and device in transaction processing system | |
| CN110569223A (en) | database log processing method and device | |
| CN113742246A (en) | Information processing method and device for test case, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |