CN115129781A - Data processing method and device, electronic equipment and storage medium - Google Patents
Data processing method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN115129781A CN115129781A CN202210424812.3A CN202210424812A CN115129781A CN 115129781 A CN115129781 A CN 115129781A CN 202210424812 A CN202210424812 A CN 202210424812A CN 115129781 A CN115129781 A CN 115129781A
- Authority
- CN
- China
- Prior art keywords
- data
- service
- hash value
- storing
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F16/273—Asynchronous replication or reconciliation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
- G06F16/2255—Hash tables
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2471—Distributed queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F16/278—Data partitioning, e.g. horizontal or vertical partitioning
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Computing Systems (AREA)
- Software Systems (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Probability & Statistics with Applications (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to the technical field of data processing, and provides a data processing method, a data processing device, electronic equipment and a storage medium, wherein the method comprises the following steps: determining service original data of data to be stored, determining hash value data of the service original data, and recording the corresponding relation between the service original data and the hash value data to a local database; carrying out block chain storage on the hash value data; and storing the service original data to a distributed storage system in an asynchronous processing mode. According to the data processing method provided by the invention, the data needing to be stored and certified is divided into the original service data and the hash value data, the block link points only carry out chain link certification on the hash value data, and the original service data is stored in the distributed storage system in an asynchronous processing mode, so that the storage efficiency of the block link points is improved, the processing capacity of the whole service data is improved, and the data processing requirement of a high-concurrency scene is met.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a data processing method and apparatus, an electronic device, and a storage medium.
Background
The block chain data storage certificate is to store the service data into the block chain, so that the purposes of tamper resistance, traceability and trustable data source are achieved. The service data can be in the form of any file such as text, video, audio pictures and the like.
At present, a scheme of local storage is basically adopted for block chain storage, and a disk can face a large reading and writing pressure when the situation of massive data and large messages is faced. Therefore, the existing block link evidence storing technology cannot meet the data processing requirements of high-concurrency scenes such as industrial internet, logistics and the like.
Disclosure of Invention
The present invention is directed to solving at least one of the problems of the prior art. Therefore, the invention provides a data processing method, which improves the data processing capacity of the block chain storage certificate and meets the data processing requirement of a high-concurrency scene.
The invention also provides a data processing device.
The invention further provides the electronic equipment.
The invention also provides a storage medium.
The data processing method according to the embodiment of the first aspect of the invention comprises the following steps:
determining service original data of data to be stored, determining hash value data of the service original data, and recording the corresponding relation between the service original data and the hash value data to a local database;
carrying out block chain storage on the hash value data;
and storing the service original data to a distributed storage system in an asynchronous processing mode.
According to the data processing method provided by the embodiment of the invention, the data needing to be stored and certified is divided into the original service data and the hash value data, the block link point only carries out chain storage and certification on the hash value data, and the original service data is stored in the distributed storage system in an asynchronous processing mode, so that the storage efficiency of the block link point is improved, the processing capacity of the whole service data is improved, and the data processing requirement of a high-concurrency scene is met.
According to an embodiment of the present invention, the block chaining certification of the hash value data includes:
acquiring the service characteristics of the data to be stored and certified, and matching a target certificate storing link corresponding to the service characteristics from the candidate certificate storing links;
and carrying out block chain certification on the hash value data based on the target certification storing link.
According to the data processing method provided by the embodiment of the invention, the link matched with the service characteristic is selected from the candidate evidence storing links to store the evidence in the block chain according to the service characteristic of the data to be stored and the preset matching rule, so that the flexibility and the reliability of the evidence storage are improved.
According to one embodiment of the invention, the number of the target authentication links is at least two;
the block chain crediting the hash value data based on the target crediting link includes:
and based on at least two target evidence storing links, performing block chain evidence storing on the hash value data by utilizing message middleware in a concurrent processing mode.
According to the data processing method provided by the embodiment of the invention, cross-chain storage and multi-element storage of data are realized by storing the same hash value data by using a plurality of links, the expandability of service is improved, the concurrent storage of a plurality of block chains is realized by using message middleware, and the hash storage decoupling of an upper-layer service and a bottom-layer block chain is realized.
According to an embodiment of the invention, said certifying the hash value data as blockchains comprises:
and determining that the data volume of the service original data is less than a preset threshold value, and performing block chaining evidence storage after packing the service original data and the hash value data.
According to the data processing method provided by the embodiment of the invention, if the data volume of the service original data is less than a certain threshold value, the data to be stored is directly packed for block chain storage, and the problem of overlarge data processing pressure of a block chain node is not caused.
According to an embodiment of the present invention, after the block chain crediting the hash value data, the method further includes:
and obtaining evidence storage record data generated when the hash value data completes the block chain evidence storage, storing the evidence storage record data to the local database, and recording the business node information of the evidence storage record data.
According to the data processing method provided by the embodiment of the invention, after the chain certification storing operation of the hash value data of a certain service node is completed, corresponding certification storing record data is generated for storing, and the information of the service node to which the corresponding data to be certified belongs is recorded at the same time, so that the subsequent data query service is provided, and the service expansibility is improved.
According to an embodiment of the present invention, after the obtaining of the evidence-storing record data generated when the hash value data completes the block chain evidence storage, storing the evidence-storing record data in the local database, and recording the service node information to which the evidence-storing record data belongs, the method further includes:
determining the affiliated service node of the evidence storing record data as a final service node of a target service according to the affiliated service node information, and packaging and combining all the evidence storing record data corresponding to the target service into archival data of the target service;
acquiring the service configuration attribute of the target service, matching a target filing link corresponding to the service configuration attribute from candidate filing links, and determining a target notarization mode of the target service according to the service configuration attribute;
and performing block link evidence storage on the archived data of the target service based on the target archival link, and acquiring a notarization confirmation letter of the target service based on the target notarization mode.
According to the data processing method provided by the embodiment of the invention, the business data of all business nodes is subjected to data archiving, the proper link is selected according to the configuration attribute of the business for archiving the chain link certificate of the data, and the proper notarization mode is matched for obtaining the notarization confirmation letter, so that the archiving certificate and the notarization traceability of the business data are pertinently carried out according to the business scene, and the flexibility of the business data certificate is improved.
According to an embodiment of the present invention, after the storing the service raw data to the distributed storage system by the asynchronous processing mode, the method further includes:
and acquiring service synchronous data of the service original data, and storing the service synchronous data to a distributed retrieval engine cluster in an asynchronous disk refreshing mode.
According to the data processing method provided by the embodiment of the invention, synchronous data is stored in the distributed retrieval engine cluster in an asynchronous disk refreshing mode for the data needing to be provided with service retrieval, so that the multifunctional retrieval capability of the service data is provided.
A data processing apparatus according to an embodiment of the second aspect of the present invention includes:
the determining module is used for determining the original service data of the data to be stored, determining the hash value data of the original service data and recording the corresponding relation between the original service data and the hash value data to a local database;
the evidence storing module is used for carrying out block chain evidence storing on the hash value data;
and the storage module is used for storing the service original data to a distributed storage system in an asynchronous processing mode.
An electronic device according to an embodiment of the third aspect of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the data processing method as described in any of the above when executing the program.
A non-transitory computer-readable storage medium according to a fourth aspect of the present invention, having stored thereon a computer program which, when executed by a processor, implements the data processing method as any one of the above.
One or more technical solutions in the embodiments of the present invention have at least one of the following technical effects:
the data needing to be stored with the certificate is divided into the original business data and the hash value data, the block node points only carry out chain link certificate storage on the hash value data, and the original business data are stored in the distributed storage system in an asynchronous processing mode, so that the storage efficiency of the block node points is improved, the processing capacity of the whole business data is improved, and the data processing requirement of a high concurrency scene is met.
Additional aspects and advantages of the invention will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the invention.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
FIG. 1 is a flow chart of a data processing method according to an embodiment of the present invention;
FIG. 2 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present invention;
fig. 3 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
The embodiments of the present invention will be described in further detail with reference to the drawings and examples. The following examples are intended to illustrate the invention but are not intended to limit the scope of the invention.
Fig. 1 is a schematic flowchart of a data processing method according to an embodiment of the present invention. Referring to fig. 1, an embodiment of the present invention provides a data processing method, including:
s1, determining the original business data of the data to be stored, determining the hash value data of the original business data, and recording the corresponding relation between the original business data and the hash value data to a local database.
It should be noted that the block chain data storage certificate is to store the service data onto the block chain, so as to achieve the purposes of tamper resistance, traceability and trustable data source. The service data can be in the form of any file such as text, video, audio pictures and the like. At present, a scheme of local storage is basically adopted for block chain storage, when the situation of massive data and large messages is faced, block chain link points cannot provide good concurrent processing capacity, and a locally stored disk can face great read-write pressure. Therefore, the existing block link evidence storing technology cannot meet the data processing requirements of high-concurrency scenes such as industrial internet, logistics and the like.
In view of this, the embodiments of the present invention provide a big data processing scheme based on a block chain. Firstly, data to be certified is divided into service raw data and hash value data, specifically, which service raw data with a large data volume is determined according to data to be certified, and the specific service raw data can be defined according to actual requirements, for example, different certification templates are defined according to different service systems, and the service raw data of the data to be certified is acquired through the certification templates. And then, determining corresponding hash value data based on the original service data, for example, extracting key information or parameters of the original service data to perform hash value calculation, and determining to obtain the hash value data corresponding to the original service data. And recording the corresponding relation between the service original data and the hash value data so as to associate the hash value data of the block chain certificate with the service original data.
And S2, performing block chain verification on the hash value data.
It can be understood that, in the embodiment of the present invention, the block link node itself is only used as a trusted storage portion to perform hash storage on hash value data corresponding to the service raw data, and the service raw data does not need to be stored, so that the storage pressure of the block link node is reduced, the storage efficiency of the node is improved, and further, the data processing capability of the whole service system is greatly improved.
And S3, storing the service raw data to a distributed storage system in an asynchronous processing mode.
It should be noted that, in the case of industrial internet, logistics and other scenes, data generated by the business system is very large and highly concurrent, and if the highly concurrent data cannot be stored by the traditional block chain storage technology, the embodiment of the present invention may deploy a data service system between the business system and the block chain service, and divide data storage into business original data distributed storage and trusted data hash storage. For the storage of the service original data, the flexible use of the high concurrency middleware is supported, the high concurrency massive data can be flexibly processed, and the transverse expansion of the data storage node can be realized under the supporting of distributed storage of the complete service original data through the data service.
In the embodiment of the invention, for the storage of the service original data, the data can be stored in the distributed storage system in an asynchronous processing mode. The service original data can be further encrypted and then stored in the distributed storage system. By way of example, the distributed storage system may use an HDFS-based NOSQL (non-relational database) database as a main database for storage, such as an Hbase distributed database, and based on the columnar storage characteristics of the distributed storage NOSQL database, a good foundation is laid for service expandability, and meanwhile, it provides a data foundation support for operations of massive data with respect to the high concurrency insertion and query capabilities of row key.
It can be understood that, by dividing the acquired data to be certified into service raw data and corresponding hash value data to be respectively stored (by calling corresponding interfaces), and recording the corresponding relationship between the two data, when one data is acquired in subsequent query, it can be known which data is corresponding to the other data according to the recorded corresponding relationship, and then the data is queried in a corresponding system (a block chain or a distributed storage system).
According to the data processing method provided by the embodiment of the invention, the data needing to be stored with the certificate is divided into the original service data and the hash value data, the block chain node only carries out chain link certificate storage on the hash value data, the original service data is stored in the distributed storage system in an asynchronous processing mode, and the block chain certificate storage service and the distributed storage system are decoupled, so that the storage efficiency of the block chain node is improved, the processing capacity of the whole service data is improved, and the data processing requirement of a high-concurrency scene is met.
In one embodiment, step S2 may include the steps of:
s21, acquiring the service characteristics of the data to be stored and obtaining a target storage link corresponding to the service characteristics from the candidate storage links in a matching manner;
s22, carrying out block chain certification on the hash value data based on the target certification storage link.
It can be understood that, for data generated by different service systems or for different service data in the same service system, there are usually different evidence storage requirements, even though different service nodes of the same service may also store different evidence storage requirements. In contrast, in the embodiment of the present invention, an appropriate target evidence storing link may be matched from multiple candidate evidence storing links according to the service characteristics of the data to be stored for uplink evidence storing.
For example, the service characteristic may be an identifier carried in the data to be certified, each service characteristic identifier may correspond to one or more candidate certificate-deposit links, and when the data to be certified needs to be processed, the block link certification for the hash value data corresponding to the data to be certified by the target certificate-deposit link may be determined according to the obtained service characteristic identifier of the data to be certified and a preset corresponding relationship; the specific identifier type and the corresponding relationship may be set according to actual application, for example, if a certain service needs to provide a tracing service, the identifier of the service is a target evidence storing link capable of providing the tracing service.
For another example, the service characteristic may be a feature value calculated according to a specific parameter (such as a data amount) in the data to be stored, and then the calculated feature value is compared with a preset threshold value to determine the target storage link.
According to the data processing method provided by the embodiment of the invention, the link matched with the service characteristic is selected from the candidate link for storing the evidence according to the service characteristic of the data to be stored and the preset matching rule, so that the flexibility and the reliability of the evidence storing service are improved, and if the link for storing the evidence is selected according to the data volume characteristic, the storage efficiency of the block link node can be further improved.
In one embodiment, the target authentication links are at least two;
step S21 may include the steps of:
s211, based on at least two target evidence storing links, the hash value data is subjected to block chain evidence storing in a concurrent processing mode by using message middleware.
It should be noted that, for more important service data, there may be a need for storing certificates through multiple block chains at the same time. In the embodiment of the invention, when two or more target certificate storing links are matched, the block chain certificate storing can be carried out on the hash value data by utilizing the message middleware in a concurrent processing mode, different subjects can be divided by the message middleware to provide support for different certificate storing parties and block chains, the advantages of the message middleware in peak clipping and valley filling and service logic decoupling can be fully exerted, and the transverse processing capability of data service is improved.
According to the data processing method provided by the embodiment of the invention, based on the message-driven characteristic, the configurable data link is added, the multiple links are used for storing the same hash value data, the cross-link storage and the multivariate storage of the data are realized, the service expansibility is improved, the concurrent storage of the multiple block chains is realized by using the message middleware, and the hash storage decoupling of the upper-layer service and the bottom-layer block chain is realized.
In one embodiment, step S2 may further include the steps of:
s23, determining that the data volume of the service original data is less than a preset threshold value, and packing the service original data and the hash value data and then performing block chain evidence storage.
In the embodiment of the present invention, after dividing the original service data and the corresponding hash value data, if the data size of the original service data is determined to be less than the preset threshold, in order to fully exert the function of the block chain service, the original service data and the hash value data may be packed together and then block chain crediting may be performed. The preset threshold may be a fixed threshold, or a threshold that is adaptively adjusted in real time according to the current data processing pressure of the block link point, so as to better ensure the storage efficiency of the block link node.
According to the data processing method provided by the embodiment of the invention, if the data volume of the original service data is less than a certain threshold value, the data to be stored is directly packed for block chain storage, and the current original service data is judged to be light enough to be packed for storage, so that the reliability of data storage is improved, the problem of overlarge data processing pressure of block chain nodes is avoided, the storage efficiency of the block chain nodes is ensured, and the processing capacity of the whole service data is further improved.
In one embodiment, after step S2, the method may further include the steps of:
s24, obtaining the certificate storing record data generated when the hash value data completes the block chain certificate storing, storing the certificate storing record data to the local database, and recording the business node information of the certificate storing record data.
It should be noted that, for the certified data, corresponding certified recording data can be generated for storage backup. For example, when a certain service exists, and the time span occupied by the service nodes is large, it is necessary to complete certificate storage for each service node of the service, and generate corresponding certificate storage record data until the final node of the service is completed.
According to the data processing method provided by the embodiment of the invention, after the chain evidence storing operation of the hash value data of a certain service node is completed, the corresponding evidence storing record data is generated for storage, and the information of the service node to which the corresponding data to be stored belongs is recorded at the same time, so that the subsequent data query service is provided, and the service expansibility is improved.
In one embodiment, after step S24, the method may further include the steps of:
s25, determining the business node of the evidence storing record data as the final business node of the target business according to the business node information, and packaging and combining all the evidence storing record data corresponding to the target business into the filing data of the target business;
s26, acquiring the service configuration attribute of the target service, matching a target filing link corresponding to the service configuration attribute from candidate filing links, and determining a target notarization mode of the target service according to the service configuration attribute;
s27, performing block chain evidence storage on the archived data of the target service based on the target archival link, and acquiring a notarization confirmation letter of the target service based on the target notarization mode.
In the embodiment of the present invention, when it is determined that a certain target service has completed all service nodes, all certificate-storing record data corresponding to the target service may be packaged into archived data, and the archived data is further subjected to chain certificate-storing and a specific notarization manner to obtain a notarization confirmation.
For example, for a personal loan service, the service node may include a plurality of service nodes such as registration, loan application, real-name authentication, contract preview, contract signing, contract validation, repayment, and the like, the block chain service returns the certificate-storing record data (such as hash data) when performing certificate storage for each node, and when completing the repayment, the user is regarded as all service nodes completing the service, and at this time, all the certificate-storing record data related to the service stored before are packaged and combined into the filing data of the service, so as to implement uplink certificate storage and notary calling of the filing data. The evidence of the archived data can select an evidence storage link matched with the service scene and an adaptive notarization mode based on the service configuration attribute corresponding to the service.
According to the data processing method provided by the embodiment of the invention, the business data of all business nodes is subjected to data archiving, the proper link is selected according to the configuration attribute of the business for archiving the chain link certificate of the data, and the proper notarization mode is matched for obtaining the notarization confirmation letter, so that the archiving certificate and the notarization traceability of the business data are pertinently carried out according to the business scene, and the flexibility of the business data certificate is improved.
In one embodiment, after step S3, the method may further include the steps of:
and S31, acquiring the service synchronous data of the service original data, and storing the service synchronous data to the distributed retrieval engine cluster in an asynchronous disk refreshing mode.
It should be noted that, for the data service of the business original data, elasticsearch may be adopted as a data source responsible for data retrieval, and further, a distributed relational database of the TiDB may be adopted as a data analysis source. For data needing service retrieval, data can be synchronized to a multi-node distributed elastic search cluster in an asynchronous batch disk refreshing mode, front-end retrieval is defined through a data object of a configurable statement, indexes of retrieved data are accurately calculated, and complex data retrieval capabilities such as paging query, grouping query, aggregation query and the like are provided. In addition, on key basic data processing, cache middleware such as redis can be utilized to provide query support of hot-point data.
The data processing method provided by the embodiment of the invention stores the synchronous data to the distributed retrieval engine cluster in an asynchronous disk refreshing mode for the data needing to be provided with the service retrieval, and provides the multifunctional retrieval capability of the service data, thereby further improving the processing capability of the whole service data.
The following describes a data processing apparatus provided in an embodiment of the present invention, and the data processing apparatus described below and the data processing method described above may be referred to correspondingly.
Referring to fig. 2, fig. 2 is a schematic block diagram of a data processing apparatus according to an embodiment of the present invention, where the data processing apparatus includes a determining module 1, a storage module 2, and a storage module 3; wherein the content of the first and second substances,
the determining module 1 is configured to determine original service data of data to be stored, determine hash value data of the original service data, and record a corresponding relationship between the original service data and the hash value data in a local database;
the evidence storing module 2 is used for performing block chain evidence storing on the hash value data;
the storage module 3 is configured to store the service raw data in a distributed storage system in an asynchronous processing manner.
In one embodiment, the evidence storage module 2 specifically includes:
the link matching unit is used for acquiring the service characteristics of the data to be stored and certified, and matching a target certificate storing link corresponding to the service characteristics from the candidate certificate storing links;
and the data evidence storing unit is used for storing the block chain of the hash value data based on the target evidence storing link.
In one embodiment, the target authentication links are at least two;
the data evidence storing unit is specifically configured to:
and based on at least two target evidence storing links, performing block chain evidence storing on the hash value data by utilizing message middleware in a concurrent processing mode.
In one embodiment, the evidence storage module 2 is further specifically configured to:
and determining that the data volume of the service original data is less than a preset threshold value, and performing block chaining evidence storage after packing the service original data and the hash value data.
In one embodiment, the data processing apparatus further comprises a data recording module for:
obtaining evidence storing record data generated when the hash value data completes the block chain evidence storing, storing the evidence storing record data to the local database, and recording the business node information of the evidence storing record data.
In one embodiment, the data processing apparatus further comprises a data archiving module for:
determining the affiliated service node of the evidence storing record data as a final service node of a target service according to the affiliated service node information, and packaging and combining all the evidence storing record data corresponding to the target service into archival data of the target service;
acquiring the service configuration attribute of the target service, matching a target filing link corresponding to the service configuration attribute from candidate filing links, and determining a target notarization mode of the target service according to the service configuration attribute;
and performing block chain certification storage on the filing data of the target service based on the target filing link, and acquiring a notarization confirmation letter of the target service based on the target notarization mode.
In one embodiment, the data processing apparatus further comprises a data synchronization module for:
and acquiring service synchronous data of the service original data, and storing the service synchronous data to a distributed retrieval engine cluster in an asynchronous disk refreshing mode.
It should be understood that the foregoing apparatus item embodiments correspond to the method item embodiments of the present invention, and the data processing apparatus provided in the embodiments of the present invention may implement the data processing method provided in any method item embodiment of the present invention.
Fig. 3 illustrates a physical structure diagram of an electronic device, which may include, as shown in fig. 3: a processor (processor)310, a communication Interface (communication Interface)320, a memory (memory)330 and a communication bus 340, wherein the processor 310, the communication Interface 320 and the memory 330 communicate with each other via the communication bus 340. The processor 310 may call logic instructions in the memory 330 to perform the following method:
s1, determining the original service data of the data to be stored, determining the hash value data of the original service data, and recording the corresponding relation between the original service data and the hash value data to a local database;
s2, carrying out block chain verification on the hash value data;
and S3, storing the service raw data to a distributed storage system in an asynchronous processing mode.
In addition, the logic instructions in the memory 330 may be implemented in the form of software functional units and stored in a computer readable storage medium when the software functional units are sold or used as independent products. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
Further, embodiments of the present invention disclose a computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, which when executed by a computer, the computer is capable of performing the methods provided by the above-mentioned method embodiments, for example, comprising:
s1, determining the original service data of the data to be stored, determining the hash value data of the original service data, and recording the corresponding relation between the original service data and the hash value data to a local database;
s2, carrying out block chain verification on the hash value data;
and S3, storing the service raw data to a distributed storage system in an asynchronous processing mode.
In another aspect, an embodiment of the present invention further provides a non-transitory computer-readable storage medium, on which a computer program is stored, where the computer program is implemented to perform the transmission method provided in the foregoing embodiments when executed by a processor, for example, the method includes:
s1, determining the original business data of the data to be stored, determining the hash value data of the original business data, and recording the corresponding relation between the original business data and the hash value data to a local database;
s2, carrying out block chain verification on the hash value data;
and S3, storing the service raw data to a distributed storage system in an asynchronous processing mode.
The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. Based on the understanding, the above technical solutions substantially or otherwise contributing to the prior art may be embodied in the form of a software product, which may be stored in a computer-readable storage medium, such as ROM/RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method according to the various embodiments or some parts of the embodiments.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
The above embodiments are merely illustrative of the present invention and are not to be construed as limiting the invention. Although the present invention has been described in detail with reference to the embodiments, it should be understood by those skilled in the art that various combinations, modifications and equivalents may be made to the technical solution of the present invention without departing from the spirit and scope of the technical solution of the present invention, and the technical solution of the present invention is covered by the claims of the present invention.
Claims (10)
1. A data processing method, comprising:
determining service original data of data to be stored, determining hash value data of the service original data, and recording the corresponding relation between the service original data and the hash value data to a local database;
carrying out block chain storage on the hash value data;
and storing the service original data to a distributed storage system in an asynchronous processing mode.
2. The data processing method of claim 1, wherein said block chaining the hash value data comprises:
acquiring the service characteristics of the data to be stored and certified, and matching a target certificate storing link corresponding to the service characteristics from the candidate certificate storing links;
and carrying out block chain evidence storage on the hash value data based on the target evidence storage link.
3. The data processing method of claim 2, wherein the number of target authentication links is at least two;
the block chain crediting the hash value data based on the target crediting link includes:
and based on at least two target evidence storing links, performing block chain evidence storing on the hash value data by utilizing message middleware in a concurrent processing mode.
4. The data processing method of claim 1, wherein the block chaining the hash value data comprises:
and determining that the data volume of the service original data is less than a preset threshold value, and performing block chaining evidence storage after packing the service original data and the hash value data.
5. The data processing method according to claim 1, further comprising, after said block chain crediting said hash value data:
obtaining evidence storing record data generated when the hash value data completes the block chain evidence storing, storing the evidence storing record data to the local database, and recording the business node information of the evidence storing record data.
6. The data processing method according to claim 5, wherein after the obtaining of the evidence storing record data generated when the hash value data completes the block chain evidence storing, storing the evidence storing record data in the local database, and recording the service node information of the evidence storing record data, the method further comprises:
determining the affiliated service node of the evidence storing record data as a final service node of a target service according to the affiliated service node information, and packaging and combining all the evidence storing record data corresponding to the target service into archival data of the target service;
acquiring the service configuration attribute of the target service, matching a target filing link corresponding to the service configuration attribute from candidate filing links, and determining a target notarization mode of the target service according to the service configuration attribute;
and performing block link evidence storage on the archived data of the target service based on the target archival link, and acquiring a notarization confirmation letter of the target service based on the target notarization mode.
7. The data processing method according to any one of claims 1 to 6, wherein after the storing the service raw data to the distributed storage system by the asynchronous processing manner, further comprising:
and acquiring service synchronous data of the service original data, and storing the service synchronous data to a distributed retrieval engine cluster in an asynchronous disk refreshing mode.
8. A data processing apparatus, comprising:
the determining module is used for determining the original business data of the data to be stored, determining the hash value data of the original business data and recording the corresponding relation between the original business data and the hash value data to a local database;
the certificate storing module is used for performing block chain certificate storing on the hash value data;
and the storage module is used for storing the service original data to a distributed storage system in an asynchronous processing mode.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the data processing method according to any of claims 1 to 7 when executing the program.
10. A non-transitory computer-readable storage medium on which a computer program is stored, the computer program, when being executed by a processor, implementing the data processing method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210424812.3A CN115129781A (en) | 2022-04-21 | 2022-04-21 | Data processing method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210424812.3A CN115129781A (en) | 2022-04-21 | 2022-04-21 | Data processing method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115129781A true CN115129781A (en) | 2022-09-30 |
Family
ID=83375882
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210424812.3A Pending CN115129781A (en) | 2022-04-21 | 2022-04-21 | Data processing method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115129781A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116668451A (en) * | 2023-04-10 | 2023-08-29 | 中泰证券股份有限公司 | Data storage method, equipment and medium for securities business |
-
2022
- 2022-04-21 CN CN202210424812.3A patent/CN115129781A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116668451A (en) * | 2023-04-10 | 2023-08-29 | 中泰证券股份有限公司 | Data storage method, equipment and medium for securities business |
| CN116668451B (en) * | 2023-04-10 | 2024-01-23 | 中泰证券股份有限公司 | Data storage method, equipment and medium for securities business |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110689349B (en) | Transaction hash value storage and searching method and device in blockchain | |
| US20120011101A1 (en) | Integrating client and server deduplication systems | |
| CN106874348B (en) | File storage and index method and device and file reading method | |
| CN106649828B (en) | Data query method and system | |
| CN104794123A (en) | Method and device for establishing NoSQL database index for semi-structured data | |
| CN105530272A (en) | Method and device for application data synchronization | |
| CN111858520B (en) | Method and device for separately storing block chain node data | |
| CN112785408A (en) | Account checking method and device based on Hash | |
| CN105975352A (en) | Cache data processing method and server | |
| CN103414762A (en) | Cloud backup method and cloud backup device | |
| CN110719526B (en) | Video playing method and device | |
| CN115858488A (en) | Parallel migration method and device based on data governance and readable medium | |
| CN115129781A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN108230487A (en) | The method and apparatus of shared camera resource | |
| CN112241396B (en) | Spark-based method and system for merging small files of Delta | |
| CN116756253B (en) | Data storage and query methods, devices, equipment and media of relational database | |
| CN112181302A (en) | Data multilevel storage and access method and system | |
| CN104732438A (en) | Transaction reversing method and transaction processing system | |
| CN116910820A (en) | Data report processing method, device, computer equipment and storage medium | |
| CN107436848B (en) | Method and device for realizing conversion between user data and compressed data | |
| CN112035471B (en) | Transaction processing method and computer equipment | |
| CN115129787A (en) | Method and device for maintaining block chain data, electronic equipment and storage medium | |
| CN111444194B (en) | Method, device and equipment for clearing indexes in block chain type account book | |
| CN111858609A (en) | Fuzzy query method and device for block chain | |
| CN113590636A (en) | Online transaction system and method based on block chain |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |