CN114356909A - Data optimization method, system, electronic device and storage medium - Google Patents
Data optimization method, system, electronic device and storage medium Download PDFInfo
- Publication number
- CN114356909A CN114356909A CN202210030676.XA CN202210030676A CN114356909A CN 114356909 A CN114356909 A CN 114356909A CN 202210030676 A CN202210030676 A CN 202210030676A CN 114356909 A CN114356909 A CN 114356909A
- Authority
- CN
- China
- Prior art keywords
- data
- preset identification
- service data
- service
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/217—Database tuning
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data optimization method, a system, an electronic device and a storage medium, wherein the data optimization method comprises the following steps: acquiring input service data and preset identification data corresponding to the service data; verifying the service data according to the preset identification data to obtain a verification result, wherein the verification result comprises that the preset identification data is the same as or different from the service data; if the service data is different from the preset identification data, newly-input service data is obtained again and verification is continued; and if the service data is the same as the preset identification data, storing the service data into a data set corresponding to the preset identification data. The data optimization method realizes the optimization of the data verification process, saves the data storage space, improves the data utilization rate and improves the operation speed of a data system.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a data optimization method, a system electronic device, and a storage medium.
Background
Currently, in the process of informatization development, a large amount of information data is rapidly generated. The system is always flooded with a large amount of redundant data. The redundant data contains a lot of repeated data, error data and invalid data, so that the data utilization rate is low, the occupied storage space is large, the system operation is slow, and the personnel operation efficiency is low. How to further process the redundant data, how to increase the data operation speed, save the data storage space and improve the data utilization rate is a major focus of attention.
Disclosure of Invention
The invention aims to overcome the defects that a system in the prior art is always filled with a large amount of redundant data, and the redundant data comprises a large amount of repeated data, error data and invalid data, so that the data utilization rate is low, the occupied storage space is large, the system runs slowly and the personnel operation efficiency is low, and provides a data optimization method, a system, electronic equipment and a storage medium.
The invention solves the technical problems through the following technical scheme:
in a first aspect, a data optimization method is provided, where the data optimization method includes:
acquiring input service data and preset identification data corresponding to the service data;
verifying the service data according to the preset identification data to obtain a verification result, wherein the verification result comprises that the preset identification data is the same as or different from the service data;
if the service data is different from the preset identification data, newly-input service data is obtained again and verification is continued;
and if the service data is the same as the preset identification data, storing the service data into a data set corresponding to the preset identification data.
The data optimization method comprises the steps of acquiring input service data and preset identification data corresponding to the service data; checking the service data according to the preset identification data to obtain a checking result, and if the service data is different from the preset identification data, re-acquiring newly input service data to continue checking; if the service data is the same as the preset identification data, the service data is stored in a data set corresponding to the preset identification data, so that the optimization of a data verification process is realized, the data storage space is saved, the data utilization rate is improved, and the operation speed of a data system is improved.
Preferably, the data optimization method further includes:
comparing the preset identification data and the service data in the same data set one by one to obtain repeated data; the repeated data comprises the same service data and the preset identification data;
and deleting the service data in the repeated data.
According to the data optimization method, the preset identification data and the business data in the same data set are compared one by one to obtain the repeated data, the repeated data comprise the same business data and the preset identification data, and the business data in the repeated data are deleted, so that the screening and deletion of the repeated data in the data system are realized, the data storage space is further saved, and the data utilization rate and the operation speed of the data system are further improved.
Preferably, the service data includes current input data, and the preset identification data includes historical reserved data corresponding to the current input data;
and checking the current input data according to the historical reserved data to obtain a checking result.
Preferably, the data optimization method includes:
outputting the checking result to a terminal display interface to display the checking result; and/or
And performing dynamic key storage on the service data and the preset identification data.
According to the data optimization method, the verification result is output to the terminal display interface to display the verification result, so that a user can conveniently know the data verification result in time, the data verification process is optimized, and/or dynamic key storage is performed on the service data and the preset identification data, the safety of data storage is improved, the risk of data restoration is reduced, and the data storage process is optimized.
In a second aspect, there is provided a data optimization system, comprising: the device comprises a data acquisition module, a verification module and a storage module;
the data acquisition module is used for acquiring input service data and preset identification data corresponding to the service data;
the checking module is used for checking the service data according to the preset identification data to obtain a checking result, wherein the checking result comprises that the preset identification data is the same as or different from the service data;
the data acquisition module is further used for re-acquiring newly input service data and continuously verifying through the verification module if the service data is different from the preset identification data;
the storage module is used for storing the service data into a data set corresponding to the preset identification data if the service data is the same as the preset identification data.
According to the data optimization system, the data acquisition module, the verification module and the storage module are matched with each other, so that the optimization of a data verification process is realized, the data storage space is saved, the data utilization rate is improved, and the operation speed of the data system is improved.
Preferably, the data optimization system further comprises a comparison module and a deletion module;
the comparison module is used for comparing the preset identification data and the service data in the same data set one by one to obtain repeated data; the repeated data comprises the same service data and the preset identification data;
the deleting module is used for deleting the service data in the repeated data.
Preferably, the service data includes current input data, and the preset identification data includes historical reserved data corresponding to the current input data;
the verification module is specifically configured to verify the current input data according to the historical reserved data to obtain a verification result.
Preferably, the data optimization system further comprises a display module;
the display module is used for outputting the verification result to a terminal display interface so as to display the verification result; and/or
The storage module comprises a dynamic storage unit, and the dynamic storage unit is used for storing the service data and the preset identification data by a dynamic key.
In a third aspect, an electronic device is provided, which includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and when the processor executes the computer program, the data optimization method described in any of the above is implemented.
In a fourth aspect, there is provided a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the data optimization method of any of the above.
The positive progress effects of the invention are as follows:
the data optimization method comprises the steps of acquiring input service data and preset identification data corresponding to the service data; checking the service data according to the preset identification data to obtain a checking result, and if the service data is different from the preset identification data, re-acquiring newly input service data to continue checking; if the service data is the same as the preset identification data, the service data is stored in a data set corresponding to the preset identification data, so that the optimization of a data verification process is realized, the data storage space is saved, the data utilization rate is improved, and the operation speed of a data system is improved.
Drawings
Fig. 1 is a schematic flow chart of a data optimization method according to embodiment 1 of the present invention;
fig. 2 is a schematic flow chart of another data optimization method provided in embodiment 1 of the present invention;
fig. 3 is a schematic structural diagram of a data optimization system according to embodiment 2 of the present invention;
fig. 4 is a schematic structural diagram of an electronic device according to embodiment 3 of the present invention.
Detailed Description
The invention is further illustrated by the following examples, which are not intended to limit the scope of the invention.
Example 1
The present embodiment provides a data optimization method, and fig. 1 is a schematic flow chart of the data optimization method provided in the present embodiment; as shown in fig. 1, the data optimization method includes:
The input service data and the preset identification data have corresponding relation.
And 102, verifying the service data according to the preset identification data to obtain a verification result.
And the checking result comprises that the preset identification data is the same as or different from the service data.
And if the service data is different from the preset identification data, returning to the step 101 to obtain newly input service data again to continue checking.
If the service data is the same as the preset identification data, step 103 is executed.
And 103, storing the service data into a data set corresponding to the preset identification data.
When the verification result represents that the service data is the same as the preset identification data, storing the service data into a data set corresponding to the preset identification data; the data system is internally stored with a plurality of data sets, and each data set is internally stored with service data and preset identification data which correspond to each other.
In the data optimization method of the embodiment, input service data and preset identification data corresponding to the service data are acquired; checking the service data according to the preset identification data to obtain a checking result, and if the service data is different from the preset identification data, re-acquiring newly input service data to continue checking; if the service data is the same as the preset identification data, the service data is stored in a data set corresponding to the preset identification data, so that the optimization of a data verification process is realized, the data storage space is saved, the data utilization rate is improved, and the operation speed of a data system is improved.
In an optional implementation manner, fig. 2 is a schematic flow diagram of a data optimization method provided in this implementation manner, and on the basis of example 1, the data optimization method in this implementation manner further includes:
In a certain data set, the service data and the preset identification data corresponding to the service data are stored, and the preset identification data and the service data in the data set are compared one by one to obtain the repeated data, wherein the repeated data comprises the same service data and the same preset identification data, that is, in the data set, the currently stored service data is the same as the preset identification data.
And 105, deleting the service data in the repeated data.
The currently stored service data is the same as the preset identification data, so that the service data stored in the data set is deleted, only the preset identification data is reserved to provide a basis for next data verification, and meanwhile, the storage space is saved.
In the prior art, the mainstream databases include a relational database, a column storage database, and a document-oriented database, wherein the relational database model is to reduce a complex data structure into a simple binary relationship. In a relational database, operations on data are almost entirely established on one or more relational tables, and the management of the database is realized by operations such as classification, combination, connection or selection of the related tables. The column storage database stores data in column families, one column family stores related data that is often queried together, occupies a large memory, and is relatively clumsy compared to other databases in view of low read performance due to establishment on a HDFS (Hadoop Distributed File System) optimized for batch analysis. And the document database oriented storage in a document form is not friendly to the support of transactions and has larger storage space. In summary, the existing data storage methods mostly adopt relational database horizontal and vertical tables, have poor retrieval capability, and have a bottleneck for storage of large data volume.
Based on data content identification, the data deduplication method mainly identifies the data format of the record. It adopts the metadata of the file system embedded in the backup data to identify the file; it is then compared byte by byte with other versions in its data store, finds the differences between the version and the first stored version and creates a delta file for these different data. This approach may avoid concurrent conflicts, but extracts metadata when needed for use.
The deduplication method is used, and a content recognition method is used, so that the new file is compared with the previous file version at the same position and the file versions backed up from other positions, and therefore the repeated data at all positions can be eliminated.
According to the data optimization method, the preset identification data and the business data in the same data set are compared one by one to obtain the repeated data, the repeated data comprise the same business data and the preset identification data, and the business data in the repeated data are deleted, so that the screening and deletion of the repeated data in the data system are realized, the data storage space is further saved, and the data utilization rate and the operation speed of the data system are further improved.
In an optional embodiment, the service data includes current input data, and the preset identification data includes historical reserved data corresponding to the current input data; and checking the current input data according to the historical reserved data to obtain a checking result.
In this embodiment, the service data mainly includes current input data, the preset identification data mainly includes historical reserved data, the current input data and the historical reserved data have a corresponding relationship, the current input data is verified according to the historical reserved data to obtain a verification result, and the verification result includes that the historical reserved data is the same as or different from the current input data.
For example, in the predetermined service, the currently input data is the mobile phone number currently input by the user, the historically reserved data is the mobile phone number historically reserved by the user, the mobile phone number currently input by the user is checked according to the mobile phone number historically reserved by the user to obtain a check result, and the check result comprises that the historically reserved mobile phone number is the same as or different from the currently input mobile phone number.
In an optional implementation manner, the data optimization method includes outputting the verification result to a terminal display interface to display the verification result.
In the embodiment, the verification result is subjected to visualization processing, and the verification result after visualization processing is output to a terminal display interface so that a user can view the data verification result.
According to the data optimization method, the verification result is output to the terminal display interface to display the verification result, so that a user can conveniently know the data verification result in time, and the data verification process is optimized.
In an optional embodiment, the service data and the preset identification data are subjected to dynamic key storage.
The method comprises the steps of evaluating the data volume needing to be checked in a database, updating a key matrix, and storing data by adopting a dynamic key, so that the data storage is safer, the updating of the key matrix is carried out according to the actual data volume of the database, the association is established between data simplification and remote backup, if more screened data exist, the data volume stored in the database is less, the useful data of the database are less, at the moment, a slightly simple key can be used for encryption, if less screened data exist in the database, more data stored in the database are represented, at the moment, the key of the database needs to be upgraded, and the effective protection of the data is realized by adopting a more complex key.
According to the data optimization method, the service data and the preset identification data are subjected to dynamic key storage, so that the safety of data storage is improved, the risk of data restoration is reduced, and the data storage process is optimized.
Example 2
Fig. 3 is a schematic structural diagram of a data optimization system provided in this embodiment, and as shown in fig. 3, the data optimization system includes: the device comprises a data acquisition module 1, a verification module 2 and a storage module 3; the data acquisition module 1 is used for acquiring input service data and preset identification data corresponding to the service data; the checking module 2 is used for checking the service data according to the preset identification data to obtain a checking result, wherein the checking result comprises that the preset identification data is the same as or different from the service data; the data acquisition module 1 is further used for re-acquiring newly input service data and continuously verifying through the verification module 2 if the service data is different from the preset identification data; the storage module 3 is configured to store the service data into a data set corresponding to the preset identification data if the service data is the same as the preset identification data.
The data optimization system of the embodiment realizes optimization of a data verification process through mutual cooperation of the data acquisition module, the verification module and the storage module, saves data storage space, improves data utilization rate and improves operation speed of the data system.
In an optional embodiment, the data optimization system further includes an alignment module 4 and a deletion module 5; the comparison module 4 is used for comparing preset identification data and service data in the same data set one by one to obtain repeated data; the repeated data comprises the same service data and preset identification data; the deleting module 5 is used for deleting the service data in the repeated data.
In an optional embodiment, the service data includes current input data, and the preset identification data includes historical reserved data corresponding to the current input data; the checking module 2 is specifically configured to check the current input data according to the historical reserved data to obtain a checking result.
In an optional embodiment, the data optimization system further comprises a display module 6; the display module 6 is used for outputting the checking result to a terminal display interface so as to display the checking result.
In an optional embodiment, the storage module 3 includes a dynamic storage unit 7, and the dynamic storage unit 7 is configured to perform dynamic key storage on the service data and the preset identification data.
Example 3
Fig. 4 is a schematic structural diagram of an electronic device provided in this embodiment. The electronic device comprises a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the data optimization method as in embodiment 1 above when executing the program. The electronic device 40 shown in fig. 4 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiment of the present invention.
As shown in FIG. 4, electronic device 40 may take the form of a general purpose computing device, which may be a server device, for example. The components of electronic device 40 may include, but are not limited to: the at least one processor 41, the at least one memory 42, and a bus 43 connecting the various system components (including the memory 42 and the processor 41).
The bus 43 includes a data bus, an address bus, and a control bus.
The memory 42 may include volatile memory, such as Random Access Memory (RAM)421 and/or cache memory 422, and may further include Read Only Memory (ROM) 423.
The processor 41 executes a computer program stored in the memory 42, thereby executing various functional applications and data processing, such as the data optimization method of the present invention as in embodiment 1 above.
The electronic device 40 may also communicate with one or more external devices 44 (e.g., keyboard, pointing device, etc.). Such communication may be through an input/output (I/O) interface 45. Also, model-generating device 40 may also communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network, such as the Internet) via network adapter 46. As shown in FIG. 4, the network adapter 46 communicates with the other modules of the model-generated device 40 via the bus 43. It should be understood that although not shown in the figures, other hardware and/or software modules may be used in conjunction with the model-generating device 40, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (disk array) systems, tape drives, and data backup storage systems, etc.
It should be noted that although in the above detailed description several units/modules or sub-units/modules of the electronic device are mentioned, such a division is merely exemplary and not mandatory. Indeed, the features and functionality of two or more of the units/modules described above may be embodied in one unit/module according to embodiments of the invention. Conversely, the features and functions of one unit/module described above may be further divided into embodiments by a plurality of units/modules.
Example 4
The present embodiment provides a computer-readable storage medium on which a computer program is stored, which when executed by a processor implements the steps in the data optimization method as in embodiment 1 above.
More specific examples, among others, that the readable storage medium may employ may include, but are not limited to: a portable disk, a hard disk, random access memory, read only memory, erasable programmable read only memory, optical storage device, magnetic storage device, or any suitable combination of the foregoing.
In a possible embodiment, the invention may also be implemented in the form of a program product comprising program code for causing a terminal device to perform the steps of implementing the data optimization method as in embodiment 1 above, when the program product is executed on the terminal device.
Where program code for carrying out the invention is written in any combination of one or more programming languages, the program code may execute entirely on the user device, partly on the user device, as a stand-alone software package, partly on the user device and partly on a remote device or entirely on the remote device.
While specific embodiments of the invention have been described above, it will be appreciated by those skilled in the art that this is by way of example only, and that the scope of the invention is defined by the appended claims. Various changes and modifications to these embodiments may be made by those skilled in the art without departing from the spirit and scope of the invention, and these changes and modifications are within the scope of the invention.
Claims (10)
1. A data optimization method, characterized in that the data optimization method comprises:
acquiring input service data and preset identification data corresponding to the service data;
verifying the service data according to the preset identification data to obtain a verification result, wherein the verification result comprises that the preset identification data is the same as or different from the service data;
if the service data is different from the preset identification data, newly-input service data is obtained again and verification is continued;
and if the service data is the same as the preset identification data, storing the service data into a data set corresponding to the preset identification data.
2. The data optimization method of claim 1, further comprising:
comparing the preset identification data and the service data in the same data set one by one to obtain repeated data; the repeated data comprises the same service data and the preset identification data;
and deleting the service data in the repeated data.
3. The data optimization method of claim 1, wherein the service data comprises current input data, and the preset identification data comprises historical reservation data corresponding to the current input data;
and checking the current input data according to the historical reserved data to obtain a checking result.
4. The data optimization method of claim 1, wherein the data optimization method comprises:
outputting the checking result to a terminal display interface to display the checking result; and/or
And performing dynamic key storage on the service data and the preset identification data.
5. A data optimization system, characterized in that the data optimization system comprises: the device comprises a data acquisition module, a verification module and a storage module;
the data acquisition module is used for acquiring input service data and preset identification data corresponding to the service data;
the checking module is used for checking the service data according to the preset identification data to obtain a checking result, wherein the checking result comprises that the preset identification data is the same as or different from the service data;
the data acquisition module is further used for re-acquiring newly input service data and continuously verifying through the verification module if the service data is different from the preset identification data;
the storage module is used for storing the service data into a data set corresponding to the preset identification data if the service data is the same as the preset identification data.
6. The data optimization system of claim 5, further comprising an alignment module and a deletion module;
the comparison module is used for comparing the preset identification data and the service data in the same data set one by one to obtain repeated data; the repeated data comprises the same service data and the preset identification data;
the deleting module is used for deleting the service data in the repeated data.
7. The data optimization system of claim 5, wherein the service data comprises current input data, and the preset identification data comprises historical reservation data corresponding to the current input data;
the verification module is specifically configured to verify the current input data according to the historical reserved data to obtain a verification result.
8. The data optimization system of claim 5, further comprising a display module;
the display module is used for outputting the verification result to a terminal display interface so as to display the verification result; and/or
The storage module comprises a dynamic storage unit, and the dynamic storage unit is used for storing the service data and the preset identification data by a dynamic key.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the data optimization method of any one of claims 1-4 when executing the computer program.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the data optimization method of any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210030676.XA CN114356909A (en) | 2022-01-12 | 2022-01-12 | Data optimization method, system, electronic device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210030676.XA CN114356909A (en) | 2022-01-12 | 2022-01-12 | Data optimization method, system, electronic device and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114356909A true CN114356909A (en) | 2022-04-15 |
Family
ID=81109214
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210030676.XA Pending CN114356909A (en) | 2022-01-12 | 2022-01-12 | Data optimization method, system, electronic device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114356909A (en) |
-
2022
- 2022-01-12 CN CN202210030676.XA patent/CN114356909A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107506451B (en) | Abnormal information monitoring method and device for data interaction | |
| US9367574B2 (en) | Efficient query processing in columnar databases using bloom filters | |
| WO2022126974A1 (en) | Kafka-based incremental data synchronization method and apparatus, device, and medium | |
| CN111356996B (en) | System and computer-implemented method for version verification | |
| US11294973B2 (en) | Codeless information service for abstract retrieval of disparate data | |
| CN104813276A (en) | Streaming restore of a database from a backup system | |
| US8972677B1 (en) | Systems and methods for implementing a storage interface specific to an archiving platform | |
| CN109471851B (en) | Data processing method, device, server and storage medium | |
| CN108664359A (en) | A kind of database restoring method, device, equipment and storage medium | |
| CN111651519B (en) | Data synchronization method, data synchronization device, electronic equipment and storage medium | |
| CN110737594A (en) | Database standard conformance testing method and device for automatically generating test cases | |
| US10788997B2 (en) | Method and device for storage management with metadata stored in disk extents in an extent pool | |
| US11853284B2 (en) | In-place updates with concurrent reads in a decomposed state | |
| CN110704431A (en) | Hierarchical storage management method for mass data | |
| CN103970844A (en) | Big data write-in method and device, big data read method and device and big data processing system | |
| US10747738B2 (en) | Efficient database management system and method for prioritizing analytical calculations on datasets | |
| CN114356909A (en) | Data optimization method, system, electronic device and storage medium | |
| US9053100B1 (en) | Systems and methods for compressing database objects | |
| CN113010399A (en) | Log data processing method, system, device and medium | |
| CN113722296A (en) | Agricultural information processing method and device, electronic equipment and storage medium | |
| CN112835887A (en) | Database management method, database management device, computing equipment and storage medium | |
| US11720551B1 (en) | Method and system for streaming data from portable storage devices | |
| US11500886B2 (en) | Finding locations of tabular data across systems | |
| WO2024108638A1 (en) | Adaptive query method based on sharding indexes, and apparatus | |
| CN117093580A (en) | Data storage method, system, equipment and medium based on label calculation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |