CN112579605A - Data storage method and device, storage medium and server - Google Patents
Data storage method and device, storage medium and server Download PDFInfo
- Publication number
- CN112579605A CN112579605A CN202011545550.3A CN202011545550A CN112579605A CN 112579605 A CN112579605 A CN 112579605A CN 202011545550 A CN202011545550 A CN 202011545550A CN 112579605 A CN112579605 A CN 112579605A
- Authority
- CN
- China
- Prior art keywords
- target data
- data
- field
- target
- merged
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000013500 data storage Methods 0.000 title claims abstract description 48
- 238000000034 method Methods 0.000 title claims abstract description 42
- 238000004590 computer program Methods 0.000 claims description 4
- 239000002609 medium Substances 0.000 description 13
- 230000008569 process Effects 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 239000012120 mounting media Substances 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000008707 rearrangement Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
Abstract
The embodiment of the invention discloses a data storage method, a data storage device, a storage medium and a server. The method comprises the following steps: acquiring target data; filling target data with empty field information based on non-empty field information corresponding to the same key field, and adding identification information of whether the target data is changed data or not according to filling operation; the field information corresponding to other key fields of the target data of which the field information is null is the same as the field information corresponding to other key fields of the target data of which the non-null field information belongs; merging target data with the same field information corresponding to all key fields in the target data after the completion of filling operation, and determining the type information of the merged target data according to the identification information and the source of each target data in the merged target data; and storing the merged target data according to the type information. By the scheme, the resource consumption of data storage can be reduced, and the availability of locally stored data during data storage is ensured.
Description
Technical Field
The embodiment of the invention relates to the technical field of data processing, in particular to a data storage method, a data storage device, a data storage medium and a server.
Background
With the development of computer technology, the amount of data to be processed by a system is continuously increased, and the incidence relation among data to be extracted is increased. With the increase of the data volume, the time required by the way of storing the data in full volume is longer and longer after the full volume of data is processed in batch and the incidence relation is extracted; the way of clearing the existing data and then storing the full amount of data leads to the data being unusable for a long time.
Disclosure of Invention
Embodiments of the present invention provide a data storage method, an apparatus, a storage medium, and a server, which can reduce resource consumption of data storage and ensure availability of locally stored data during data storage.
In a first aspect, an embodiment of the present invention provides a data storage method, including:
acquiring target data; wherein the target data comprises at least one piece of external input data and at least one piece of local storage data;
filling target data with empty field information based on non-empty field information corresponding to the same key field, and adding identification information of whether the target data is changed data or not according to filling operation; the field information corresponding to other key fields of the target data of which the field information is empty is the same as the field information corresponding to other key fields of the target data to which the non-empty field information belongs;
merging target data with the same field information corresponding to all key fields in the target data after the completion of filling operation, and determining the type information of the merged target data according to the identification information of each target data in the merged target data and the source of each target data; wherein the source of the target data comprises external input data and local storage data;
and storing the merged target data according to the type information.
In a second aspect, an embodiment of the present invention further provides a data storage device, including:
the data acquisition module is used for acquiring target data; wherein the target data comprises at least one piece of external input data and at least one piece of local storage data;
the field filling module is used for filling target data with empty field information based on non-empty field information corresponding to the same key field and adding identification information of whether the target data is changed data or not according to filling operation; the field information corresponding to other key fields of the target data of which the field information is empty is the same as the field information corresponding to other key fields of the target data to which the non-empty field information belongs;
the data merging module is used for merging the target data which are identical in field information and correspond to all key fields in the target data after the filling operation is executed, and determining the type information of the merged target data according to the identification information of each target data in the merged target data and the source of each target data; wherein the source of the target data comprises external input data and local storage data;
and the data storage module is used for storing the merged target data according to the type information.
In a third aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the data storage method according to the first aspect of the embodiment of the present invention.
In a fourth aspect, an embodiment of the present invention provides a server, including:
one or more processors;
a memory for storing one or more programs;
when the one or more programs are executed by the one or more processors, the one or more processors implement the data storage method according to the first aspect of the embodiment of the present invention.
According to the data storage scheme provided by the embodiment of the invention, target data is obtained; wherein the target data comprises at least one piece of external input data and at least one piece of local storage data; filling target data with empty field information based on non-empty field information corresponding to the same key field, and adding identification information of whether the target data is changed data or not according to filling operation; the field information corresponding to other key fields of the target data of which the field information is empty is the same as the field information corresponding to other key fields of the target data to which the non-empty field information belongs; merging target data with the same field information corresponding to all key fields in the target data after the completion of filling operation, and determining the type information of the merged target data according to the identification information of each target data in the merged target data and the source of each target data; wherein the source of the target data comprises external input data and local storage data; and storing the merged target data according to the type information. By adopting the technical scheme, the technical problems that the resource consumption is large when the data is stored in full quantity and the data is unavailable for a long time due to the mode of clearing the existing data and then storing the data in full quantity can be solved, the resource consumption of data storage can be reduced, the usability of the locally stored data is ensured when the data is stored, and the user experience is greatly improved.
Drawings
Fig. 1 is a schematic flow chart of a data storage method according to an embodiment of the present invention;
fig. 2 is a schematic flowchart of a data storage method according to a second embodiment of the present invention;
fig. 3 is a schematic structural diagram of a data storage device according to a third embodiment of the present invention;
fig. 4 is a schematic structural diagram of a server according to a fifth embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.
Before discussing exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although a flowchart may describe the steps as a sequential process, many of the steps can be performed in parallel, concurrently or simultaneously. In addition, the order of the steps may be rearranged. The process may be terminated when its operations are completed, but may have additional steps not included in the figure. The processes may correspond to methods, functions, procedures, subroutines, and the like.
Example one
Fig. 1 is a schematic flowchart of a data storage method according to an embodiment of the present invention, where the method is applicable to a server for storing data, and can be executed by a data storage device, where the device can be implemented by software and/or hardware, and can be generally integrated in the server. As shown in fig. 1, the method includes:
s101, acquiring target data; wherein the target data comprises at least one piece of external input data and at least one piece of local storage data.
The external input data is data to be stored, the external input data may be data transmitted from other servers or terminal devices, and the local storage data is data stored in a local database. Optionally, acquiring the target data includes: at least one piece of external input data and at least one piece of local storage data are read from the distributed file system based on different path information, respectively. Specifically, the distributed file system generally stores the local storage data and the external input data in different path files, and therefore, at least one piece of external input data and at least one piece of local storage data can be read from the distributed file system based on different path information, and the obtained external input data and the obtained local storage data are used as target data. Optionally, the target data may be identified based on the path information.
S102, filling target data with empty field information based on non-empty field information corresponding to the same key field, and adding identification information of whether the target data is changed data or not according to filling operation; and the field information corresponding to other key fields of the target data of which the field information is empty is the same as the field information corresponding to other key fields of the target data of which the non-empty field information belongs.
In the embodiment of the present invention, each piece of target data may include field information corresponding to at least one key field, where the field information corresponding to the key field in the target data may be null field information or non-null field information. It is understood that when the target data includes the null field information, it may indicate that the target data is the incomplete data. Specifically, each piece of target data obtained from the distributed file system may include a plurality of key fields, but some pieces of data may have some of the key fields and some other pieces of data may have other parts of the key fields in the target data. According to the specific conditions of the service and the data, the target data can be considered to be the same data as long as one field is the same in the service among the key fields, so that the target data with the field information being empty can be backfilled based on the non-empty field information corresponding to the same key field.
Specifically, target data with empty field information is filled based on non-empty field information corresponding to the same key field, wherein the target data with empty field information is the same as field information corresponding to other key fields of the target data to which the non-empty field information belongs. Illustratively, the target data 1 and the target data 2 each include field information corresponding to four key fields, namely, a field a, a field B, a field C, and a field D, where the field information corresponding to the key field B in the target data 1 is null, the field information corresponding to other key fields (the field a, the field C, and the field D) is non-null, the field information corresponding to the four key fields in the target data 2 is all non-null, and the field information corresponding to the field a, the field C, and the field D in the target data 1 is the same as the field information corresponding to the field a, the field C, and the field D in the target data 2, that is, the field information corresponding to the same key field in the target data 1 is the same as the field information corresponding to the key field B in the target data 2 except for the null field, and the null field information corresponding to the key field B in the target data 1 can be filled based on the field information.
In the embodiment of the invention, the identification information of whether the target data is changed data or not is added to the target data according to the filling operation, wherein the identification information is used for identifying whether the target data is changed data or not. Optionally, adding, according to the filling operation, identification information of whether the target data is changed data, including: identifying the target data filled by the non-empty field information as changed data; and identifying the target data which is not filled by the non-empty field information as unchanged data. Specifically, when the empty field information in a certain target data is filled in by the non-empty field information corresponding to the same key field in other target data, the filled entry mark data is identified as changed data; when a certain piece of target data does not contain null field information, or the null field information contained in a certain piece of target data does not exist other target data which is completely the same as other non-null field information in the target data, the null field information in the target data is not filled by the non-null field information corresponding to the same key field in other target data, and the target data is identified as unchanged data.
S103, merging the target data with the same field information corresponding to all key fields in the target data after the filling operation is executed, and determining the type information of the merged target data according to the identification information of each target data in the merged target data and the source of each target data; wherein the source of the target data comprises external input data and local storage data.
For example, it is determined whether at least two pieces of target data having completely the same field information corresponding to all the key fields exist in all the target data after the completion of the filling operation, and if yes, the target data having completely the same field information corresponding to all the key fields are merged, that is, multiple pieces of target data are merged into one piece of target data. And then, determining the type information of the merged target data according to the identification information of each target data in the merged target data and the source of each target data, wherein the source of the target data comprises whether the target data is external input data or local storage data. Optionally, the type information includes new data, unmodified data, and modified data; determining the type information of the merged target data according to the identification information of each target data in the merged target data and the source of the target data, wherein the type information comprises the following steps: when the merged target data comprise local storage data and the identification information of the local storage data is changed data, determining the merged target data as modified data; when the merged target data comprise local storage data and the identification information of the local storage data is unchanged data, determining the merged target data as unmodified data; and when all the merged target data are external input data, determining the merged target data as new data.
Specifically, when an entry index exists in a plurality of pieces of target data for merging, the entry index is locally stored data, and the identification information is changed data, the merged target data is determined as modified data; when one piece of target data exists in the plurality of pieces of target data for merging, the target data is locally stored data, and the identification information is unchanged data, determining the merged target data as unmodified data; and when the plurality of pieces of target data for merging do not have local storage data, that is, all the plurality of pieces of target data for merging are external input data, determining the merged target data as newly added data.
And S104, storing the merged target data according to the type information.
In the embodiment of the invention, the storage mode of the merged target data is determined according to the type information of the merged target data. Optionally, storing the merged target data according to the type information includes: when the merged target data are modified data, replacing corresponding local storage data based on the merged target data; when the merged target data is newly added data, locally storing the merged target data; and deleting the merged target data when the merged target data is unmodified data. Specifically, when the merged target data is modified data, it is described that data corresponding to the merged target data is locally stored, but the locally stored corresponding data needs to be updated based on the merged target data, so that the locally stored data can be replaced based on the merged target data, that is, the locally stored original data corresponding to the merged target data is deleted, and the merged target data is stored. When the merged target data is newly added data, it is indicated that data corresponding to the merged target data is not stored locally, so that the merged target data is directly stored, and a local database is enriched. When the merged target data is unmodified data, it is indicated that data completely identical to the merged target data is stored locally, so as to avoid repeated storage of the identical data and waste of system resources, and the merged target data can be directly deleted.
The data storage method provided by the embodiment of the invention obtains target data; wherein the target data comprises at least one piece of external input data and at least one piece of local storage data; filling target data with empty field information based on non-empty field information corresponding to the same key field, and adding identification information of whether the target data is changed data or not according to filling operation; the field information corresponding to other key fields of the target data of which the field information is empty is the same as the field information corresponding to other key fields of the target data to which the non-empty field information belongs; merging target data with the same field information corresponding to all key fields in the target data after the completion of filling operation, and determining the type information of the merged target data according to the identification information of each target data in the merged target data and the source of each target data; wherein the source of the target data comprises external input data and local storage data; and storing the merged target data according to the type information. By adopting the technical scheme, the technical problems that the resource consumption is large when the data is stored in full quantity and the data is unavailable for a long time due to the mode of clearing the existing data and then storing the data in full quantity can be solved, the resource consumption of data storage can be reduced, the usability of the locally stored data is ensured when the data is stored, and the user experience is greatly improved.
In some embodiments, before filling the target data with field information empty based on non-empty field information corresponding to the same key field, the method further includes: determining at least one key field related to the target data, and taking any one key field in the at least one key field as a target key field; filling target data with empty field information based on non-empty field information corresponding to the same key field, and adding identification information of whether the target data is changed data or not according to filling operation, wherein the identification information comprises the following steps: grouping the target data based on the target key field; dividing the target data with the same field information corresponding to the target key field into a group; for each group of target data, filling target data with empty field information based on non-empty field information corresponding to the same key field, and adding identification information of whether the target data is changed data or not according to filling operation; merging the target data with the same field information corresponding to all key fields in the target data after the completion of the filling operation, and determining the type information of the merged target data according to the identification information of each target data in the merged target data and the source of each target data, wherein the type information comprises the following steps: merging target data with the same field information corresponding to all key fields in the target data after the filling operation is executed, determining the type information of the merged target data according to the identification information of each target data in the merged target data and the source of each target data, and taking the merged target data as new target data; and taking the next key field in at least one key field related to the target data as a target key field, and returning to the step of grouping the target data based on the target key field until the key fields are respectively taken as target key fields. The advantage of this arrangement is that target data actually belonging to the same data can be quickly and accurately merged.
For example, the key fields related to different target data may be different, at least one key field related to all the target data is determined, and any one of the at least one key field related to the target data is used as the target key field. And then, grouping the target data based on the target key fields, and grouping the target data with the same field information corresponding to the target key fields into a group. And for each group of target data, filling target data with null field information based on non-null field information corresponding to the same key field, wherein the field information corresponding to other key fields is completely the same in the target data with null field information and the target data to which the non-null field information belongs except for the null fields. Then adding identification information of whether the filled target data is changed data or not according to the filling operation, wherein the target data filled by the non-empty field information is identified as changed data; and identifying the target data which is not filled by the non-empty field information as unchanged data. And then merging the target data with the same field information corresponding to all the key fields in the target data after the filling operation is executed, and taking the merged target data as new target data. Then, the next key field in the plurality of key fields related by the target data is used as the target key field, and the step of grouping the target data based on the target key field is returned to be executed until all the key fields are respectively used as the target key fields. For example, the target data includes 100 pieces of target data, and 4 key fields related to the target data, the target data may be grouped based on a first key field, then for each group of target data, the target data whose field information is null is filled based on non-null field information corresponding to the same key field, and finally, after the target data whose field information is the same as that corresponding to all key fields in the target data after the completion of the filling operation is merged, the target data is changed from 100 pieces to 80 pieces, then the 80-piece target data is used as new target data, the 80-piece target data is subjected to the grouping, filling and merging processing based on a second key field, and so on until the grouping, filling and merging processing is completed based on a last key field.
Illustratively, the target data is device information data, and the key fields related to the target data include a data id, a device card number, and a device card id, and the target data contains two pieces of external input data and one piece of local storage data. Specifically, the three pieces of target data are respectively: target data 1: data id1, device id1, device card number 1, null value; target data 2: data id2, null value, device card number 1, device card id 1; target data: data id3, device id2, device card number 2, device card id 2. The target data 1 and the target data 2 are external input data, and the target data 3 is local storage data. If the data id is used as the target key field, the three pieces of target data can be divided into three groups because the field information corresponding to the target key field (data id) in the three pieces of target data is different, and the filling operation of the empty field information cannot be performed; if the device card number is used as the target key field, the three pieces of target data can be divided into two groups based on the device card number, and since the field information corresponding to the target key field (device card number) in the target data 1 and the target data 2 is the same, the target data 1 and the target data 2 can be used as 1 group, and the target data 3 can be used as one group. The field information corresponding to the key field of the device id in the target data 2 is null, and therefore, the field information corresponding to the device id in the target data 2 can be filled based on the non-null field information (device id1) corresponding to the key field of the device id in the target data 1; the field information corresponding to the key field of the device card id in the target data 1 is null, and therefore, the field information corresponding to the device card id in the target data 1 can be filled based on the non-null field information (device card id1) corresponding to the key field of the device card id in the target data 2. The three filled target data and the identification information thereof are as follows: target data 1: data id1, device id1, device card number 1, device card id1, identification information: external input data, change data; target data: data id2, device id1, device card number 1, device card id1, identification information: storing data locally, changing data; target data 3: data id3, device id2, device card number 2, device card id2, identification information: external input data, unchanged data. Then, the target data 1 and the target data 2 after the completion of the filling operation may be subjected to a merging process.
Example two
Fig. 2 is a flowchart of a data storage method according to a second embodiment of the present invention, and as shown in fig. 2, the method includes the following steps:
s201, reading at least one piece of external input data and at least one piece of local storage data from the distributed file system respectively based on different path information.
S202, determining at least one key field related to the target data, and taking any one of the at least one key field as a target key field.
S203, grouping the target data based on the target key field; and dividing the target data with the same field information corresponding to the target key field into a group.
S204, for each group of target data, filling target data with null field information based on non-null field information corresponding to the same key field, and adding identification information of whether the target data is changed data or not according to filling operation; the target data with the field information being null is the same as the field information corresponding to other key fields of the target data to which the non-null field information belongs.
Optionally, adding, according to the filling operation, identification information of whether the target data is changed data, including: identifying the target data filled by the non-empty field information as changed data; and identifying the target data which is not filled by the non-empty field information as unchanged data.
And S205, merging the target data with the same field information corresponding to all the key fields in the target data after the completion of the filling operation, determining the type information of the merged target data according to the identification information of each target data in the merged target data and the source of each target data, and taking the merged target data as new target data.
Wherein the source of the target data comprises external input data and local storage data.
Optionally, the type information includes new data, unmodified data, and modified data; determining the type information of the merged target data according to the identification information of each target data in the merged target data and the source of the target data, wherein the type information comprises the following steps: when the merged target data comprise local storage data and the identification information of the local storage data is changed data, determining the merged target data as modified data; when the merged target data comprise local storage data and the identification information of the local storage data is unchanged data, determining the merged target data as unmodified data; and when all the merged target data are external input data, determining the merged target data as new data.
S206, taking the next key field in the at least one key field related by the target data as the target key field, and returning to execute S203.
S207, judging whether at least one key field related to the target data is respectively used as a target key field, if so, executing S208, and otherwise, executing S206.
And S208, storing the merged target data according to the type information.
Optionally, storing the merged target data according to the type information includes: when the merged target data are modified data, replacing corresponding local storage data based on the merged target data; when the merged target data is newly added data, the merged target data is locally stored; and when the merged target data is unmodified data, deleting the merged target data.
The data storage method provided by the embodiment of the invention can solve the technical problems that the resource consumption is large when the data is stored in full quantity and the data is unavailable for a long time due to the mode of removing the existing data and then storing the data in full quantity, reduce the resource consumption of data storage, ensure the availability of the locally stored data when the data is stored and greatly improve the user experience.
EXAMPLE III
Fig. 3 is a schematic structural diagram of a data storage device according to a third embodiment of the present invention. As shown in fig. 3, the apparatus includes: a data acquisition module 301, a field filling module 302, a data merging module 303, and a data storage module 304, wherein:
a data acquisition module 301, configured to acquire target data; wherein the target data comprises at least one piece of external input data and at least one piece of local storage data;
a field filling module 302, configured to fill, based on non-empty field information corresponding to the same key field, target data whose field information is empty, and add, according to a filling operation, identification information of whether the target data is changed data; the field information corresponding to other key fields of the target data of which the field information is empty is the same as the field information corresponding to other key fields of the target data to which the non-empty field information belongs;
the data merging module 303 is configured to merge target data, which have the same field information and correspond to all key fields, in the target data after the completion of the filling operation, and determine type information of the merged target data according to identification information of each target data in the merged target data and a source of each target data; wherein the source of the target data comprises external input data and local storage data;
and a data storage module 304, configured to store the merged target data according to the type information.
The data storage device provided by the embodiment of the invention acquires target data; wherein the target data comprises at least one piece of external input data and at least one piece of local storage data; filling target data with empty field information based on non-empty field information corresponding to the same key field, and adding identification information of whether the target data is changed data or not according to filling operation; the field information corresponding to other key fields of the target data of which the field information is empty is the same as the field information corresponding to other key fields of the target data to which the non-empty field information belongs; merging target data with the same field information corresponding to all key fields in the target data after the completion of filling operation, and determining the type information of the merged target data according to the identification information of each target data in the merged target data and the source of each target data; wherein the source of the target data comprises external input data and local storage data; and storing the merged target data according to the type information. By adopting the technical scheme, the technical problems that the resource consumption is large when the data is stored in full quantity and the data is unavailable for a long time due to the mode of clearing the existing data and then storing the data in full quantity can be solved, the resource consumption of data storage can be reduced, the usability of the locally stored data is ensured when the data is stored, and the user experience is greatly improved.
Optionally, the key field determining module is configured to determine at least one key field related to the target data before filling target data whose field information is null based on non-null field information corresponding to the same key field, and use any one key field in the at least one key field as a target key field;
the field filling module is used for:
grouping the target data based on the target key field; dividing the target data with the same field information corresponding to the target key field into a group;
for each group of target data, filling target data with empty field information based on non-empty field information corresponding to the same key field, and adding identification information of whether the target data is changed data or not according to filling operation;
the data merging module is configured to:
merging target data with the same field information corresponding to all key fields in the target data after the filling operation is executed, determining the type information of the merged target data according to the identification information of each target data in the merged target data and the source of each target data, and taking the merged target data as new target data;
and taking the next key field in at least one key field related to the target data as a target key field, and returning to the step of grouping the target data based on the target key field until the key fields are respectively taken as target key fields.
Optionally, adding, according to the filling operation, identification information of whether the target data is changed data, including:
identifying the target data filled by the non-empty field information as changed data;
and identifying the target data which is not filled by the non-empty field information as unchanged data.
Optionally, the type information includes new data, unmodified data, and modified data;
determining the type information of the merged target data according to the identification information of each target data in the merged target data and the source of the target data, wherein the type information comprises the following steps:
when the merged target data comprise local storage data and the identification information of the local storage data is changed data, determining the merged target data as modified data;
when the merged target data comprise local storage data and the identification information of the local storage data is unchanged data, determining the merged target data as unmodified data;
and when all the merged target data are external input data, determining the merged target data as new data.
Optionally, the data storage module is configured to:
when the merged target data are modified data, replacing corresponding local storage data based on the merged target data;
when the merged target data is newly added data, locally storing the merged target data;
and deleting the merged target data when the merged target data is unmodified data.
Optionally, the data obtaining module is configured to:
at least one piece of external input data and at least one piece of local storage data are read from the distributed file system based on different path information, respectively.
Example four
A fourth embodiment of the present invention provides a storage medium containing computer-executable instructions for a data storage method when executed by a computer processor.
Storage medium-any of various types of memory devices or storage devices. The term "storage medium" is intended to include: mounting media such as CD-ROM, floppy disk, or tape devices; computer system memory or random access memory such as DRAM, DDRRAM, SRAM, EDORAM, Lanbas (Rambus) RAM, etc.; non-volatile memory such as flash memory, magnetic media (e.g., hard disk or optical storage); registers or other similar types of memory elements, etc. The storage medium may also include other types of memory or combinations thereof. In addition, the storage medium may be located in a first computer system in which the program is executed, or may be located in a different second computer system connected to the first computer system through a network (such as the internet). The second computer system may provide program instructions to the first computer for execution. The term "storage medium" may include two or more storage media that may reside in different locations, such as in different computer systems that are connected by a network. The storage medium may store program instructions (e.g., embodied as a computer program) that are executable by one or more processors.
Of course, the storage medium provided by the embodiment of the present invention contains computer-executable instructions, and the computer-executable instructions are not limited to the data storage operation or the file retrieval operation described above, and may also perform related operations in the data storage method provided by any embodiment of the present invention.
EXAMPLE five
Fig. 4 is a schematic structural diagram of a server according to a fifth embodiment of the present invention. The server shown in fig. 4 is only an example, and should not bring any limitation to the function and the scope of use of the embodiments of the present invention. As shown in fig. 4, a server according to a fifth embodiment of the present invention includes a processor 41, a memory 42, an input device 43, and an output device 44; the number of the processors 41 in the server may be one or more, and one processor 41 is taken as an example in fig. 4; the processor 41, the memory 42, the input device 43 and the output device 44 in the server may be connected by a bus or other means, and the bus connection is exemplified in fig. 4.
The memory 42 is a computer-readable storage medium, and can be used for storing software programs, computer-executable programs, and modules, such as program instructions/modules corresponding to the bluetooth-based operation mode configuration method in the embodiment of the present invention (for example, the data acquisition module 301, the field filling module 302, the data merging module 303, and the data storage module 304 in the data storage device). The processor 41 executes various functional applications and file processing by executing software programs, instructions, and modules stored in the memory 42, for example, to implement a data storage method applied to a server provided by an embodiment of the present invention.
The memory 42 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to the use of the server, and the like. Further, the memory 42 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some examples, memory 42 may further include memory located remotely from processor 41, which may be connected to a server over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The input device 43 may be used to receive numeric or character information input by a user to generate key signal inputs related to user settings and function control of the server. The output device 44 may include a display device such as a display screen.
The data storage device, the storage medium and the server provided in the above embodiments may execute the corresponding data storage method provided in the embodiments of the present invention, and have corresponding functional modules and beneficial effects for executing the method. For technical details that are not described in detail in the above embodiments, reference may be made to a data storage and retrieval method provided in any embodiment of the present invention.
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims (10)
1. A method of storing data, comprising:
acquiring target data; wherein the target data comprises at least one piece of external input data and at least one piece of local storage data;
filling target data with empty field information based on non-empty field information corresponding to the same key field, and adding identification information of whether the target data is changed data or not according to filling operation; the field information corresponding to other key fields of the target data of which the field information is empty is the same as the field information corresponding to other key fields of the target data to which the non-empty field information belongs;
merging target data with the same field information corresponding to all key fields in the target data after the completion of filling operation, and determining the type information of the merged target data according to the identification information of each target data in the merged target data and the source of each target data; wherein the source of the target data comprises external input data and local storage data;
and storing the merged target data according to the type information.
2. The method of claim 1, further comprising, before filling the target data with null field information based on non-null field information corresponding to the same key field:
determining at least one key field related to the target data, and taking any one key field in the at least one key field as a target key field;
filling target data with empty field information based on non-empty field information corresponding to the same key field, and adding identification information of whether the target data is changed data or not according to filling operation, wherein the identification information comprises the following steps:
grouping the target data based on the target key field; dividing the target data with the same field information corresponding to the target key field into a group;
for each group of target data, filling target data with empty field information based on non-empty field information corresponding to the same key field, and adding identification information of whether the target data is changed data or not according to filling operation;
merging the target data with the same field information corresponding to all key fields in the target data after the completion of the filling operation, and determining the type information of the merged target data according to the identification information of each target data in the merged target data and the source of each target data, wherein the type information comprises the following steps:
merging target data with the same field information corresponding to all key fields in the target data after the filling operation is executed, determining the type information of the merged target data according to the identification information of each target data in the merged target data and the source of each target data, and taking the merged target data as new target data;
and taking the next key field in at least one key field related to the target data as a target key field, and returning to the step of grouping the target data based on the target key field until the key fields are respectively taken as target key fields.
3. The method according to claim 1 or 2, wherein adding identification information of whether the target data is changed data or not according to the filling operation comprises:
identifying the target data filled by the non-empty field information as changed data;
and identifying the target data which is not filled by the non-empty field information as unchanged data.
4. The method of claim 3, wherein the type information includes new data, unmodified data, and modified data;
determining the type information of the merged target data according to the identification information of each target data in the merged target data and the source of the target data, wherein the type information comprises the following steps:
when the merged target data comprise local storage data and the identification information of the local storage data is changed data, determining the merged target data as modified data;
when the merged target data comprise local storage data and the identification information of the local storage data is unchanged data, determining the merged target data as unmodified data;
and when all the merged target data are external input data, determining the merged target data as new data.
5. The method of claim 4, wherein storing the merged target data according to the type information comprises:
when the merged target data are modified data, replacing corresponding local storage data based on the merged target data;
when the merged target data is newly added data, locally storing the merged target data;
and deleting the merged target data when the merged target data is unmodified data.
6. The method of claim 1, wherein obtaining target data comprises:
at least one piece of external input data and at least one piece of local storage data are read from the distributed file system based on different path information, respectively.
7. A data storage device, comprising:
the data acquisition module is used for acquiring target data; wherein the target data comprises at least one piece of external input data and at least one piece of local storage data;
the field filling module is used for filling target data with empty field information based on non-empty field information corresponding to the same key field and adding identification information of whether the target data is changed data or not according to filling operation; the field information corresponding to other key fields of the target data of which the field information is empty is the same as the field information corresponding to other key fields of the target data to which the non-empty field information belongs;
the data merging module is used for merging the target data which are identical in field information and correspond to all key fields in the target data after the filling operation is executed, and determining the type information of the merged target data according to the identification information of each target data in the merged target data and the source of each target data; wherein the source of the target data comprises external input data and local storage data;
and the data storage module is used for storing the merged target data according to the type information.
8. The apparatus of claim 7, further comprising:
the key field determining module is used for determining at least one key field related to target data before filling the target data with field information being null based on non-null field information corresponding to the same key field, and taking any one key field in the at least one key field as a target key field;
the field filling module is used for:
grouping the target data based on the target key field; dividing the target data with the same field information corresponding to the target key field into a group;
for each group of target data, filling target data with empty field information based on non-empty field information corresponding to the same key field, and adding identification information of whether the target data is changed data or not according to filling operation;
the data merging module is configured to:
merging target data with the same field information corresponding to all key fields in the target data after the filling operation is executed, determining the type information of the merged target data according to the identification information of each target data in the merged target data and the source of each target data, and taking the merged target data as new target data;
and taking the next key field in at least one key field related to the target data as a target key field, and returning to the step of grouping the target data based on the target key field until the key fields are respectively taken as target key fields.
9. A computer-readable storage medium, on which a computer program is stored, which program, when being executed by a processor, carries out the data storage method of any one of claims 1 to 6.
10. A server, comprising:
one or more processors;
a memory for storing one or more programs;
when executed by the one or more processors, cause the one or more processors to implement a data storage method as claimed in any one of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011545550.3A CN112579605A (en) | 2020-12-23 | 2020-12-23 | Data storage method and device, storage medium and server |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011545550.3A CN112579605A (en) | 2020-12-23 | 2020-12-23 | Data storage method and device, storage medium and server |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112579605A true CN112579605A (en) | 2021-03-30 |
Family
ID=75139323
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011545550.3A Pending CN112579605A (en) | 2020-12-23 | 2020-12-23 | Data storage method and device, storage medium and server |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112579605A (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106599104A (en) * | 2016-11-29 | 2017-04-26 | 北京锐安科技有限公司 | Mass data association method based on redis cluster |
| CN109558400A (en) * | 2018-11-28 | 2019-04-02 | 北京锐安科技有限公司 | Data processing method, device, equipment and storage medium |
-
2020
- 2020-12-23 CN CN202011545550.3A patent/CN112579605A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106599104A (en) * | 2016-11-29 | 2017-04-26 | 北京锐安科技有限公司 | Mass data association method based on redis cluster |
| CN109558400A (en) * | 2018-11-28 | 2019-04-02 | 北京锐安科技有限公司 | Data processing method, device, equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107301214B (en) | Data migration method and device in HIVE and terminal equipment | |
| CN111352902A (en) | Log processing method and device, terminal equipment and storage medium | |
| CN112597153B (en) | Block chain-based data storage method, device and storage medium | |
| CN103544153A (en) | Data updating method and system based on database | |
| CN108647357B (en) | Data query method and device | |
| US20210232561A1 (en) | Data processing method and apparatus, and computer-readable storage medium | |
| CN112445729B (en) | Operation address determination method, PCIe system, electronic device and storage medium | |
| CN104423982A (en) | Request processing method and device | |
| CN110515795A (en) | A kind of monitoring method of big data component, device, electronic equipment | |
| CN111737564A (en) | Information query method, device, equipment and medium | |
| CN115114232A (en) | Method, device and medium for enumerating historical version objects | |
| US10719497B2 (en) | Utilization of optimized ordered metadata structure for container-based large-scale distributed storage | |
| CN112860412B (en) | Service data processing method and device, electronic equipment and storage medium | |
| CN114138558A (en) | Object storage method and device, electronic equipment and storage medium | |
| CN109697234B (en) | Multi-attribute information query method, device, server and medium for entity | |
| CN112579605A (en) | Data storage method and device, storage medium and server | |
| CN111694992A (en) | Data processing method and device | |
| CN114547206A (en) | Data synchronization method and data synchronization system | |
| CN114116908A (en) | Data management method and device and electronic equipment | |
| CN111782588A (en) | File reading method, device, equipment and medium | |
| CN105279103A (en) | Data management method and apparatus | |
| CN113127436B (en) | Data synchronization method, device, server and storage medium | |
| CN114513558B (en) | User request processing method and device | |
| CN111459411B (en) | Data migration method, device, equipment and storage medium | |
| CN109033271B (en) | Data insertion method and device based on column storage, server and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |