CN118227836A - Block chain-based data processing method, device and storage medium - Google Patents
Block chain-based data processing method, device and storage medium Download PDFInfo
- Publication number
- CN118227836A CN118227836A CN202410155510.XA CN202410155510A CN118227836A CN 118227836 A CN118227836 A CN 118227836A CN 202410155510 A CN202410155510 A CN 202410155510A CN 118227836 A CN118227836 A CN 118227836A
- Authority
- CN
- China
- Prior art keywords
- data
- block
- processing
- hash
- blocks
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000003672 processing method Methods 0.000 title abstract description 12
- 238000003860 storage Methods 0.000 title abstract description 12
- 238000012545 processing Methods 0.000 claims abstract description 108
- 238000000034 method Methods 0.000 claims abstract description 91
- 230000008569 process Effects 0.000 claims abstract description 26
- 230000002159 abnormal effect Effects 0.000 claims abstract description 21
- 238000007781 pre-processing Methods 0.000 claims abstract description 19
- 230000011218 segmentation Effects 0.000 claims abstract description 13
- 230000009467 reduction Effects 0.000 claims abstract description 12
- 238000011946 reduction process Methods 0.000 claims abstract description 6
- 238000004422 calculation algorithm Methods 0.000 claims description 16
- 238000012216 screening Methods 0.000 claims description 9
- 238000012549 training Methods 0.000 claims description 9
- 238000013507 mapping Methods 0.000 claims description 8
- 238000013479 data entry Methods 0.000 claims description 7
- 238000001514 detection method Methods 0.000 claims description 7
- 238000012937 correction Methods 0.000 claims description 6
- 239000011159 matrix material Substances 0.000 claims description 6
- 238000013528 artificial neural network Methods 0.000 claims description 5
- 238000013450 outlier detection Methods 0.000 claims description 4
- 238000012935 Averaging Methods 0.000 claims description 3
- 238000000354 decomposition reaction Methods 0.000 claims description 3
- 238000002360 preparation method Methods 0.000 claims description 3
- 230000010076 replication Effects 0.000 claims description 3
- 230000009897 systematic effect Effects 0.000 abstract 1
- 238000005516 engineering process Methods 0.000 description 11
- 230000006870 function Effects 0.000 description 9
- 230000007246 mechanism Effects 0.000 description 7
- 230000005540 biological transmission Effects 0.000 description 4
- 238000011161 development Methods 0.000 description 3
- 230000018109 developmental process Effects 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000007405 data analysis Methods 0.000 description 2
- 238000007418 data mining Methods 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 238000013459 approach Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 230000000052 comparative effect Effects 0.000 description 1
- 230000006378 damage Effects 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 238000013524 data verification Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 238000005538 encapsulation Methods 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000006855 networking Effects 0.000 description 1
- 238000003909 pattern recognition Methods 0.000 description 1
- 238000012797 qualification Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000013515 script Methods 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000012706 support-vector machine Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/901—Indexing; Data structures therefor; Storage structures
- G06F16/9014—Indexing; Data structures therefor; Storage structures hash tables
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/10—Pre-processing; Data cleansing
- G06F18/15—Statistical pre-processing, e.g. techniques for normalisation or restoring missing data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/602—Providing cryptographic facilities or services
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Software Systems (AREA)
- Evolutionary Computation (AREA)
- Bioethics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Computational Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Artificial Intelligence (AREA)
- Health & Medical Sciences (AREA)
- Evolutionary Biology (AREA)
- General Health & Medical Sciences (AREA)
- Computer Hardware Design (AREA)
- Computer Security & Cryptography (AREA)
- Life Sciences & Earth Sciences (AREA)
- Probability & Statistics with Applications (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention belongs to the technical field of blockchains, and particularly relates to a blockchain-based data processing method, a blockchain-based data processing device and a storage medium, wherein the method comprises the following steps of: a method of blockchain-based data processing, the method performing the steps of: step 1: data preprocessing: carrying out data preprocessing on the data to be processed to obtain preprocessed data; the data preprocessing process comprises the following steps: data noise reduction processing and data segmentation processing; the data noise reduction processing at least comprises: removing unique data attributes, processing data missing values, detecting data abnormal values and processing data protocols; and the data segmentation process is used for carrying out data segmentation on each piece of data subjected to the data noise reduction process so as to obtain a data body representing the data content and a data head representing the data attribute characteristics. By constructing a systematic hash tree in the blockchain, each node in the blockchain is connected, so that target data can be efficiently acquired when data processing is performed.
Description
Technical Field
The invention belongs to the technical field of blockchains, and particularly relates to a blockchain-based data processing method, a blockchain-based data processing device and a storage medium.
Background
Blockchains are a term of art in information technology. Essentially, the system is a shared database, and data or information stored in the shared database has the characteristics of 'non-falsifiability', 'whole-course trace', 'traceability', 'disclosure transparency', 'collective maintenance', and the like. Based on the characteristics, the blockchain technology lays a solid 'trust' foundation, creates a reliable 'cooperation' mechanism and has wide application prospect.
Generally, blockchain systems consist of a data layer, a network layer, a consensus layer, an incentive layer, a contract layer, and an application layer. The data layer encapsulates the underlying data blocks and related basic data such as data encryption and time stamps and basic algorithms; the network layer comprises a distributed networking mechanism, a data transmission mechanism, a data verification mechanism and the like; the consensus layer mainly encapsulates various consensus algorithms of the network node; the incentive layer integrates economic factors into a blockchain technology system and mainly comprises an issuing mechanism, an allocation mechanism and the like of economic incentives; the contract layer mainly encapsulates various scripts, algorithms and intelligent contracts, and is the basis of programmable characteristics of the block chain; the application layer encapsulates various application scenarios and cases of the blockchain. In the model, chain block structure based on time stamp, consensus mechanism of distributed nodes, economic incentive based on consensus force and flexible programmable intelligent contract are the most representative innovation points of block chain technology.
Data (Data) is a representation of facts, concepts, or instructions that may be processed by manual or automated means. After the data is interpreted and given a certain meaning, the data becomes information. Data processing (dataprocessing) is the collection, storage, retrieval, processing, transformation, and transmission of data. The basic purpose of data processing is to extract and derive data that is valuable and meaningful to some particular person from a large, possibly unorganized, unintelligible, data. Data processing is a fundamental link of system engineering and automatic control. Data processing extends throughout various areas of social production and social life. The development of data processing technology and the breadth and depth of application thereof greatly influence the progress of human society development.
Therefore, the block chain technology and the data processing are combined, so that the data processing efficiency can be remarkably improved, and the data security can be improved.
Disclosure of Invention
The invention mainly aims to provide a data processing method, a device and a storage medium based on a blockchain, which are used for connecting all nodes in the blockchain by constructing a system hash tree in the blockchain so as to efficiently acquire target data when data processing is performed, and meanwhile, the data are connected through the hash tree in all the nodes in the blockchain, so that the data acquisition efficiency is further improved, and meanwhile, the data acquisition efficiency is improved by using a path planning algorithm when the data processing is performed, and the data processing efficiency is greatly improved.
In order to achieve the above purpose, the technical scheme of the invention is realized as follows:
A voiceprint recognition method based on multi-feature matching, the method performing the steps of:
A method of blockchain-based data processing, the method performing the steps of:
Step 1: data preprocessing: carrying out data preprocessing on the data to be processed to obtain preprocessed data; the data preprocessing process comprises the following steps: data noise reduction processing and data segmentation processing; the data noise reduction processing at least comprises: removing unique data attributes, processing data missing values, detecting data abnormal values and processing data protocols; the data segmentation process is carried out to segment each piece of data after the data noise reduction process so as to obtain a data body representing the data content and a data head representing the data attribute characteristics;
step 2: constructing a blockchain system, wherein the blockchain system comprises a plurality of blocks; establishing a system hash tree of a block chain system, wherein each block has a hash value in the system hash tree, and the hash value is a root hash of the block;
step 3: constructing a hash tree in each block, wherein each node in the block has a hash value in the hash tree, and the hash value is the root hash of the node in the block; adding a mark for each block at the same time; the mark comprises at least: a time stamp; the root hash, the hash tree and the mark form a block head of the block; the data blocks stored in each block and the mapping data blocks form a block main body of the block;
Step 4: inputting the data body and the data head into a block chain system, wherein each block in the block chain system performs data entry on the received data body and the corresponding data head, each data head and the corresponding data body are used as data blocks in a block main body, and each data block corresponds to a hash value in a hash tree; then updating the data record and updating the flag in the block header;
Step 5: each block performs data copying on the data body and the data head after data entry, and performs data self-encryption on the data body and the data head after data copying;
Step 6: after each block copies the data in the block, and the data from the encrypted data body is written into the next block by block, the data from the encrypted data body is written into the next block by block; the next block is defined as a block corresponding to another hash value which is mutually mapped with the hash value of the block in the system hash tree; the data body written into the next block constitutes the present block
Step 7: when data processing is carried out, a block chain system firstly receives a data request, firstly transmits a data acquisition command according to the received data request, and the data acquisition command carries out block screening through a system hash tree so as to determine a block required during the data processing, wherein the blocks are related with each other through hash values in the system hash tree;
Step 8: data screening is carried out in the screened blocks based on the hash trees in the blocks so as to determine the data in the blocks required during data processing, and the Zhejiang data are related through the hash values of the hash trees in the blocks;
Step 9: and planning a data processing path by screening the selected block and the data in the screened block so as to minimize the path during data processing.
Further, the method for processing the data missing value in the step 1 includes: preparing data, namely acquiring data without missing values to perform data preprocessing, constructing a random mask according to a given missing rate, and taking the newly generated random mask and corresponding data as a new data set to perform model training; model training, which is to use the new data set generated in the data preparation step to perform model training so as to construct a model based on neural network decomposition; and constructing a corresponding mask for the data to be processed with the missing value by using the model, and filling the missing value of the data to be processed by using the trained model.
Further, the method for detecting the abnormal value of the data in the step 1 includes: sorting the die processing data according to a time sequence, filling the missing data with randomly generated data, and obtaining primary sorting data; performing outlier detection processing on the preliminary arrangement data, and filling the detected outlier with randomly generated data; and carrying out abnormal value correction processing on the data after the abnormal value detection processing, namely filling the 0 value in the data, and finishing the abnormal value detection and correction of the data to be processed.
Further, the method for performing data protocol processing in the step 1 includes: removing average value, calculating covariance matrix, calculating eigenvalue and eigenvector of covariance matrix, sorting eigenvalue from big to small, reserving maximum 5 eigenvectors, and converting data into new space constructed by 5 eigenvectors; and finally, obtaining processed new data, wherein the data are irrelevant in pairs.
Further, the method for copying the data of the data body and the data head after the data is input by each block in the step 5 includes: and directly copying the data head, averaging the data in the data body with a randomly generated numerical value, and then copying.
Further, the method for performing data self-encryption on the data body and the data head after the data copying in the step 5 includes: performing data blurring on the copied data, including dividing the data into a plurality of blocks, and exchanging data between the blocks; encrypting the data using an encryption algorithm; known information from the copied data is used as an encryption key; each block is individually encrypted, and for each block, known information from another block is used as the encryption key; the replication data determines a hash value and uses the determined hash value to determine the size of the block and/or the number of blocks.
Further, the encryption algorithm is a symmetric encryption algorithm.
Further, the method in step 9 of planning a data processing path by using the screened block and the data in the screened block so as to minimize the path during the data processing includes: firstly, carrying out path connection on screened blocks through a system hash tree, and determining a contact sequence according to a mapping sequence of hash values corresponding to each block in the system hash tree; and regarding each block as a node, and planning a data path under each node based on the screened data.
Further, the method for data path planning performs the following steps: determining a path based on the screened data according to a random sequence, and calculating the complexity of the determined path by using the following formula: w i =lg (1-Wherein N is the number of the screened data, L is the weight ratio of the screened data, the weight ratio=the number of the screened data/the total number of the data in the block, U kj represents the screened data, and J i represents the hash value corresponding to the screened data; if the calculated complexity is within the set threshold range, the path is used as a path planning result; if the calculated complexity exceeds the set threshold, the path is discarded, and a path is determined, and the calculated complexity is known to be within the set threshold.
A data processing apparatus based on a blockchain.
The data processing method, the device and the storage medium based on the blockchain have the following beneficial effects: according to the method, the system hash tree is built in the block chain, all nodes in the block chain are connected, so that target data can be efficiently obtained when data processing is performed, meanwhile, in all the nodes in the block chain, the data are connected through the hash tree, the data obtaining efficiency is further improved, and meanwhile, when the data processing is performed, the data obtaining efficiency is improved by using a path planning algorithm, and the data processing efficiency is greatly improved. The method is mainly realized through the following steps: the method is mainly realized through the following steps: 1. pretreatment of data: the invention firstly carries out data preprocessing on the data to be processed, and the data preprocessing process comprises the following steps: data noise reduction processing and data segmentation processing; the data noise reduction processing at least comprises: removing unique data attributes, processing data missing values, detecting data abnormal values and processing data protocols; the data segmentation process is carried out on each piece of data after the data noise reduction process to obtain a data body representing the data content and a data head representing the data attribute characteristics, and the data is processed after the process, so that the data processing efficiency can be remarkably improved; 2. correlation of data: in the process of data processing, the invention carries out the association of the blocks and the association of the data through the hash tree, and the efficiency of the associated data and the blocks is higher when the associated data and the blocks are processed; 3. planning of data processing paths: when the data is processed, the path planning is performed based on the associated block and the data, and when the planned path is processed, the time for calling the data can be saved, and the efficiency of the data processing is further improved.
Drawings
FIG. 1 is a block chain based data processing method according to an embodiment of the present invention;
FIG. 2 is a block diagram illustrating a block of a blockchain-based data processing method according to an embodiment of the present invention;
FIG. 3 is a flowchart illustrating a method for performing data processing within a block of a blockchain-based data processing method according to an embodiment of the present invention;
fig. 4 is a schematic diagram of experimental effects of efficiency improvement percentage and error ratio changing along with the number of experiments according to the voiceprint recognition method and device based on multi-feature matching according to the embodiment of the present invention, and a schematic diagram of comparative experimental effects of the manual data processing method and the conventional data processing method in the prior art.
Detailed Description
The technical scheme of the invention is further described in detail below with reference to the specific embodiments and the attached drawings:
Example 1
As shown in fig. 1, a voiceprint recognition method based on multi-feature matching performs the steps of:
A method of blockchain-based data processing, the method performing the steps of:
Step 1: data preprocessing: carrying out data preprocessing on the data to be processed to obtain preprocessed data; the data preprocessing process comprises the following steps: data noise reduction processing and data segmentation processing; the data noise reduction processing at least comprises: removing unique data attributes, processing data missing values, detecting data abnormal values and processing data protocols; the data segmentation process is carried out to segment each piece of data after the data noise reduction process so as to obtain a data body representing the data content and a data head representing the data attribute characteristics;
step 2: constructing a blockchain system, wherein the blockchain system comprises a plurality of blocks; establishing a system hash tree of a block chain system, wherein each block has a hash value in the system hash tree, and the hash value is a root hash of the block;
step 3: constructing a hash tree in each block, wherein each node in the block has a hash value in the hash tree, and the hash value is the root hash of the node in the block; adding a mark for each block at the same time; the mark comprises at least: a time stamp; the root hash, the hash tree and the mark form a block head of the block; the data blocks stored in each block and the mapping data blocks form a block main body of the block;
Step 4: inputting the data body and the data head into a block chain system, wherein each block in the block chain system performs data entry on the received data body and the corresponding data head, each data head and the corresponding data body are used as data blocks in a block main body, and each data block corresponds to a hash value in a hash tree; then updating the data record and updating the flag in the block header;
Step 5: each block performs data copying on the data body and the data head after data entry, and performs data self-encryption on the data body and the data head after data copying;
Step 6: after each block copies the data in the block, and the data from the encrypted data body is written into the next block by block, the data from the encrypted data body is written into the next block by block; the next block is defined as a block corresponding to another hash value which is mutually mapped with the hash value of the block in the system hash tree; the data body written into the next block forms the block;
Step 7: when data processing is carried out, a block chain system firstly receives a data request, firstly transmits a data acquisition command according to the received data request, and the data acquisition command carries out block screening through a system hash tree so as to determine a block required during the data processing, wherein the blocks are related with each other through hash values in the system hash tree;
Step 8: data screening is carried out in the screened blocks based on the hash trees in the blocks so as to determine the data in the blocks required during data processing, and the Zhejiang data are related through the hash values of the hash trees in the blocks;
Step 9: and planning a data processing path by screening the selected block and the data in the screened block so as to minimize the path during data processing.
By adopting the technical scheme, each node in the block chain is connected by constructing the system hash tree in the block chain, so that target data can be efficiently acquired when data processing is performed, meanwhile, the data is connected by the hash tree in each node in the block chain, the data acquisition efficiency is further improved, and meanwhile, the data acquisition efficiency is improved by using a path planning algorithm when the data processing is performed, and the data processing efficiency is greatly improved. The method is mainly realized through the following steps: the method is mainly realized through the following steps: 1. pretreatment of data: the invention firstly carries out data preprocessing on the data to be processed, and the data preprocessing process comprises the following steps: data noise reduction processing and data segmentation processing; the data noise reduction processing at least comprises: removing unique data attributes, processing data missing values, detecting data abnormal values and processing data protocols; the data segmentation process is carried out on each piece of data after the data noise reduction process to obtain a data body representing the data content and a data head representing the data attribute characteristics, and the data is processed after the process, so that the data processing efficiency can be remarkably improved; 2. correlation of data: in the process of data processing, the invention carries out the association of the blocks and the association of the data through the hash tree, and the efficiency of the associated data and the blocks is higher when the associated data and the blocks are processed; 3. planning of data processing paths: when the data is processed, the path planning is performed based on the associated block and the data, and when the planned path is processed, the time for calling the data can be saved, and the efficiency of the data processing is further improved.
Example 2
On the basis of the above embodiment, the method for processing the data missing value in step 1 includes: preparing data, namely acquiring data without missing values to perform data preprocessing, constructing a random mask according to a given missing rate, and taking the newly generated random mask and corresponding data as a new data set to perform model training; model training, which is to use the new data set generated in the data preparation step to perform model training so as to construct a model based on neural network decomposition; and constructing a corresponding mask for the data to be processed with the missing value by using the model, and filling the missing value of the data to be processed by using the trained model.
In particular, with the development of modern information technology and internet technology, people acquire more and more data through different channels, and the data is the core of big data analysis, machine learning and knowledge discovery, and the high-quality data is the basic requirement of knowledge discovery, modeling and rule extraction. But due to the factors of poor quality of the data source, environmental errors and human interference in the data collection process, a large amount of data for data mining and analysis has missing values. Data loss refers to data that should be obtained at the time of data acquisition but that is not obtained for some reason. The backfill missing values which are as accurate as possible are beneficial to improving the data quality and the effectiveness of data mining and pattern recognition, so that the effective backfill missing values have very important significance.
Example 3
On the basis of the above embodiment, the method for detecting the abnormal value of the data in the step 1 includes: sorting the die processing data according to a time sequence, filling the missing data with randomly generated data, and obtaining primary sorting data; performing outlier detection processing on the preliminary arrangement data, and filling the detected outlier with randomly generated data; and carrying out abnormal value correction processing on the data after the abnormal value detection processing, namely filling the 0 value in the data, and finishing the abnormal value detection and correction of the data to be processed.
Specifically, for the research of the real-time data outlier detection method, the former has proposed methods such as neural network, support vector machine, etc., but the outlier processing of the historical data is not considered, because the accurate prediction of the real-time data needs to be based on the reliable historical data. For detecting abnormal values of historical data, common methods include abnormal value detection methods based on statistics, clustering, distance, density and the like, but these methods do not consider time sequence change characteristics of time sequence data, but consider a data corpus to conceal local abnormal values, which are difficult to detect. The invention can effectively detect the local outlier of the time sequence by adopting a median-EEMD method for the historical data, and can more effectively detect the outlier of the real-time data by combining a neural network method.
Example 4
On the basis of the above embodiment, the method for performing data protocol processing in step 1 includes: removing average value, calculating covariance matrix, calculating eigenvalue and eigenvector of covariance matrix, sorting eigenvalue from big to small, reserving maximum 5 eigenvectors, and converting data into new space constructed by 5 eigenvectors; and finally, obtaining processed new data, wherein the data are irrelevant in pairs.
Example 5
On the basis of the above embodiment, the method for copying the data of the data body and the data head after the data is input by each block in the step 5 includes: and directly copying the data head, averaging the data in the data body with a randomly generated numerical value, and then copying.
Example 6
On the basis of the above embodiment, the method for performing data self-encryption on the data body and the data head after the data copying in the step 5 includes: performing data blurring on the copied data, including dividing the data into a plurality of blocks, and exchanging data between the blocks; encrypting the data using an encryption algorithm; known information from the copied data is used as an encryption key; each block is individually encrypted, and for each block, known information from another block is used as the encryption key; the replication data determines a hash value and uses the determined hash value to determine the size of the block and/or the number of blocks.
In particular, the purpose of the data transmission encryption technique is to encrypt the data stream in transmission, and there are generally two types of line encryption and end-to-end encryption. Line encryption is focused on lines regardless of source and destination, and is used to provide security protection for confidential information by using different encryption keys for each line. End-to-end encryption refers to the automatic encryption of information by the sender and the encapsulation of packets by TCP/IP, then traversing the internet as unreadable and unrecognizable data, which when arriving at the destination, will be automatically reassembled, decrypted, and made readable.
The purpose of the data storage encryption technology is to prevent data decryption in the storage link, and the data storage encryption technology can be divided into ciphertext storage and access control. The former is generally realized by methods of encryption algorithm conversion, password addition, encryption module and the like; the latter is to examine and limit the qualification and authority of the user to prevent the illegal user from accessing the data or prevent the legal user from accessing the data by unauthorized.
The purpose of data integrity authentication techniques is to verify the identity and associated data content of the person involved in the transfer, access and processing of information, typically including the authentication of passwords, keys, identities, data and the like. The system realizes the safety protection of the data by comparing whether the characteristic value input by the verification object accords with the preset parameter.
The key management technology comprises secret measures in various links such as key generation, distribution, storage, replacement, destruction and the like.
Example 7
On the basis of the above embodiment, the encryption algorithm is a symmetric encryption algorithm.
In particular, the method comprises the steps of,
Example 8
On the basis of the above embodiment, the method in step 9 of planning a data processing path by using the screened block and the data in the screened block so as to minimize the path during the data processing includes: firstly, carrying out path connection on screened blocks through a system hash tree, and determining a contact sequence according to a mapping sequence of hash values corresponding to each block in the system hash tree; and regarding each block as a node, and planning a data path under each node based on the screened data.
Specifically, the hash table (Hashtable, also called hash table) is a data structure that is directly accessed according to the key value (Keyvalue). That is, it accesses the record by mapping the key value to a location in the table to speed up the lookup. This mapping function is called a hash function and the array in which the records are stored is called a hash table.
Given a table M, there is a function f (key), and if an address recorded in the table containing the key can be obtained after substituting the function into any given key value key, the table M is referred to as a Hash (Hash) table, and the function f (key) is a Hash (Hash) function.
The hash table lookup process is essentially the same as the tabulation process. Some keys can be directly found through the addresses converted by the hash function, and other keys generate conflicts on the addresses obtained by the hash function, so that the keys need to be searched according to a conflict processing method. In the three methods of conflict handling described, the search after the conflict is still the process of comparing the given value with the key. Therefore, the measure of hash table lookup efficiency is still measured by the average lookup length.
Example 9
On the basis of the above embodiment, the method of data path planning performs the steps of: determining a path based on the screened data according to a random sequence, and calculating the complexity of the determined path by using the following formula: wherein N is the number of the screened data, L is the weight ratio of the screened data, the weight ratio=the number of the screened data/the total number of the data in the block, U kj represents the screened data, and J i represents the hash value corresponding to the screened data; if the calculated complexity is within the set threshold range, the path is used as a path planning result; if the calculated complexity exceeds the set threshold, the path is discarded, and a path is determined, and the calculated complexity is known to be within the set threshold.
Example 10
A data processing apparatus based on a blockchain.
The foregoing is merely an example of the present invention and is not intended to limit the scope of the present invention, and all changes made in the structure according to the present invention should be considered as falling within the scope of the present invention without departing from the gist of the present invention.
It will be clear to those skilled in the art that, for convenience and brevity of description, the specific working process of the system described above and the related description may refer to the corresponding process in the foregoing method embodiment, which is not repeated here.
It should be noted that, in the system provided in the foregoing embodiment, only the division of the foregoing functional modules is illustrated, in practical application, the foregoing functional allocation may be performed by different functional modules, that is, the modules or steps in the embodiment of the present invention are further decomposed or combined, for example, the modules in the foregoing embodiment may be combined into one module, or may be further split into multiple sub-modules, so as to complete all or part of the functions described above. The names of the modules and steps related to the embodiments of the present invention are merely for distinguishing the respective modules or steps, and are not to be construed as unduly limiting the present invention.
It will be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working process of the storage device and the processing device described above and the related description may refer to the corresponding process in the foregoing method embodiment, which is not repeated herein.
Those of skill in the art will appreciate that the various illustrative modules, method steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the program(s) corresponding to the software modules, method steps, may be embodied in Random Access Memory (RAM), memory, read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art. To clearly illustrate this interchangeability of electronic hardware and software, various illustrative components and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as electronic hardware or software depends upon the particular application and design constraints imposed on the solution. Those skilled in the art may implement the described functionality using different approaches for each particular application, but such implementation is not intended to be limiting.
The terms "first," "second," and the like, are used for distinguishing between similar objects and not for describing a particular sequential or chronological order.
The terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus/apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus/apparatus.
Thus far, the technical solution of the present invention has been described in connection with the preferred embodiments shown in the drawings, but it is easily understood by those skilled in the art that the scope of protection of the present invention is not limited to these specific embodiments. Equivalent modifications and substitutions for related technical features may be made by those skilled in the art without departing from the principles of the present invention, and such modifications and substitutions will fall within the scope of the present invention.
The foregoing description is only of the preferred embodiments of the present invention, and is not intended to limit the scope of the present invention.
Claims (10)
1. A method of blockchain-based data processing, the method performing the steps of:
Step 1: data preprocessing: carrying out data preprocessing on the data to be processed to obtain preprocessed data; the data preprocessing process comprises the following steps: data noise reduction processing and data segmentation processing; the data noise reduction processing at least comprises: removing unique data attributes, processing data missing values, detecting data abnormal values and processing data protocols; the data segmentation process is carried out to segment each piece of data after the data noise reduction process so as to obtain a data body representing the data content and a data head representing the data attribute characteristics;
step 2: constructing a blockchain system, wherein the blockchain system comprises a plurality of blocks; establishing a system hash tree of a block chain system, wherein each block has a hash value in the system hash tree, and the hash value is a root hash of the block;
step 3: constructing a hash tree in each block, wherein each node in the block has a hash value in the hash tree, and the hash value is the root hash of the node in the block; adding a mark for each block at the same time; the mark comprises at least: a time stamp; the root hash, the hash tree and the mark form a block head of the block; the data blocks stored in each block and the mapping data blocks form a block main body of the block;
Step 4: inputting the data body and the data head into a block chain system, wherein each block in the block chain system performs data entry on the received data body and the corresponding data head, each data head and the corresponding data body are used as data blocks in a block main body, and each data block corresponds to a hash value in a hash tree; then updating the data record and updating the flag in the block header;
Step 5: each block performs data copying on the data body and the data head after data entry, and performs data self-encryption on the data body and the data head after data copying;
Step 6: after each block copies the data in the block, and the data from the encrypted data body is written into the next block by block, the data from the encrypted data body is written into the next block by block; the next block is defined as a block corresponding to another hash value which is mutually mapped with the hash value of the block in the system hash tree; the data body written into the next block constitutes the present block
Step 7: when data processing is carried out, a block chain system firstly receives a data request, firstly transmits a data acquisition command according to the received data request, and the data acquisition command carries out block screening through a system hash tree so as to determine a block required during the data processing, wherein the blocks are related with each other through hash values in the system hash tree;
step 8: and then data screening is carried out in the screened blocks based on the hash tree in the blocks to determine the data in the blocks required by data processing,
The Zhejiang data are related through hash values of hash trees in the blocks;
Step 9: and planning a data processing path by screening the selected block and the data in the screened block so as to minimize the path during data processing.
2. The method of claim 1, wherein the method of processing the data loss value in step 1 comprises: preparing data, namely acquiring data without missing values to perform data preprocessing, constructing a random mask according to a given missing rate, and taking the newly generated random mask and corresponding data as a new data set to perform model training; model training, which is to use the new data set generated in the data preparation step to perform model training so as to construct a model based on neural network decomposition; and constructing a corresponding mask for the data to be processed with the missing value by using the model, and filling the missing value of the data to be processed by using the trained model.
3. The method of claim 2, wherein the method of detecting the abnormal value of the data in step1 comprises: sorting the die processing data according to a time sequence, filling the missing data with randomly generated data, and obtaining primary sorting data; performing outlier detection processing on the preliminary arrangement data, and filling the detected outlier with randomly generated data; and carrying out abnormal value correction processing on the data after the abnormal value detection processing, namely filling the 0 value in the data, and finishing the abnormal value detection and correction of the data to be processed.
4. The method of claim 3, wherein the method of performing data protocol processing in step 1 comprises: removing average value, calculating covariance matrix, calculating eigenvalue and eigenvector of covariance matrix, sorting eigenvalue from big to small, reserving maximum 5 eigenvectors, and converting data into new space constructed by 5 eigenvectors; and finally, obtaining processed new data, wherein the data are irrelevant in pairs.
5. The method of claim 4, wherein the method of copying the data of the data body and the data header after the data entry by each block in step 5 comprises: and directly copying the data head, averaging the data in the data body with a randomly generated numerical value, and then copying.
6. The method as set forth in claim 5, wherein the method for performing data self-encryption on the copied data body and data header in step 5 includes: performing data blurring on the copied data, including dividing the data into a plurality of blocks, and exchanging data between the blocks; encrypting the data using an encryption algorithm; known information from the copied data is used as an encryption key; each block is individually encrypted, and for each block, known information from another block is used as the encryption key; the replication data determines a hash value and uses the determined hash value to determine the size of the block and/or the number of blocks.
7. The method of claim 6, wherein the encryption algorithm is a symmetric encryption algorithm.
8. The method of claim 6, wherein the step 9 of planning the data processing path by the sorted blocks and the data in the sorted blocks so that the path during the data processing is the shortest comprises: firstly, carrying out path connection on screened blocks through a system hash tree, and determining a contact sequence according to a mapping sequence of hash values corresponding to each block in the system hash tree; and regarding each block as a node, and planning a data path under each node based on the screened data.
9. The method of claim 7, wherein the method of data path planning performs the steps of: determining a path based on the screened data according to a random sequence, and calculating the complexity of the determined path by using the following formula: wherein N is the number of the screened data, L is the weight ratio of the screened data, the weight ratio=the number of the screened data/the total number of the data in the block, U kj represents the screened data, and J i represents the hash value corresponding to the screened data; if the calculated complexity is within the set threshold range, the path is used as a path planning result; if the calculated complexity exceeds the set threshold, the path is discarded, and a path is determined, and the calculated complexity is known to be within the set threshold.
10. A blockchain-based data processing device for implementing the method of any of claims 1 to 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410155510.XA CN118227836B (en) | 2024-02-04 | 2024-02-04 | Block chain-based data processing method, device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410155510.XA CN118227836B (en) | 2024-02-04 | 2024-02-04 | Block chain-based data processing method, device and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN118227836A true CN118227836A (en) | 2024-06-21 |
| CN118227836B CN118227836B (en) | 2024-09-17 |
Family
ID=91503749
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410155510.XA Active CN118227836B (en) | 2024-02-04 | 2024-02-04 | Block chain-based data processing method, device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN118227836B (en) |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110717832A (en) * | 2019-10-21 | 2020-01-21 | 腾讯科技（深圳）有限公司 | Clean energy trading method, system, device and storage medium based on block chain |
| WO2020125839A1 (en) * | 2018-12-18 | 2020-06-25 | GRID INVENT gGmbH | Electronic element and electrically controlled display element |
| WO2021184580A1 (en) * | 2020-03-17 | 2021-09-23 | 平安科技（深圳）有限公司 | Intelligent domain name resolution method and apparatus, electronic device and computer-readable storage medium |
| JP2021158548A (en) * | 2020-03-27 | 2021-10-07 | 株式会社日立製作所 | Information sharing management method and information sharing management apparatus |
| CN113822263A (en) * | 2021-06-18 | 2021-12-21 | 腾讯科技（深圳）有限公司 | Image annotation method and device, computer equipment and storage medium |
| KR20220032884A (en) * | 2020-09-08 | 2022-03-15 | 서강대학교산학협력단 | Batch processing method for storing continuous IoT data on the blockchain and system for storing IoT data using the method |
| CN115408718A (en) * | 2022-09-02 | 2022-11-29 | 中国银行股份有限公司 | Information auditing method and device, electronic equipment and storage medium |
| CN115480714A (en) * | 2022-10-25 | 2022-12-16 | 中国空间技术研究院 | Satellite big data distributed storage system based on block chain |
| WO2023050620A1 (en) * | 2021-09-29 | 2023-04-06 | 西安交通大学 | Method and system for monitoring abnormal user behavior in distributed blockchain system |
-
2024
- 2024-02-04 CN CN202410155510.XA patent/CN118227836B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2020125839A1 (en) * | 2018-12-18 | 2020-06-25 | GRID INVENT gGmbH | Electronic element and electrically controlled display element |
| CN110717832A (en) * | 2019-10-21 | 2020-01-21 | 腾讯科技（深圳）有限公司 | Clean energy trading method, system, device and storage medium based on block chain |
| WO2021184580A1 (en) * | 2020-03-17 | 2021-09-23 | 平安科技（深圳）有限公司 | Intelligent domain name resolution method and apparatus, electronic device and computer-readable storage medium |
| JP2021158548A (en) * | 2020-03-27 | 2021-10-07 | 株式会社日立製作所 | Information sharing management method and information sharing management apparatus |
| KR20220032884A (en) * | 2020-09-08 | 2022-03-15 | 서강대학교산학협력단 | Batch processing method for storing continuous IoT data on the blockchain and system for storing IoT data using the method |
| CN113822263A (en) * | 2021-06-18 | 2021-12-21 | 腾讯科技（深圳）有限公司 | Image annotation method and device, computer equipment and storage medium |
| WO2023050620A1 (en) * | 2021-09-29 | 2023-04-06 | 西安交通大学 | Method and system for monitoring abnormal user behavior in distributed blockchain system |
| CN115408718A (en) * | 2022-09-02 | 2022-11-29 | 中国银行股份有限公司 | Information auditing method and device, electronic equipment and storage medium |
| CN115480714A (en) * | 2022-10-25 | 2022-12-16 | 中国空间技术研究院 | Satellite big data distributed storage system based on block chain |
Non-Patent Citations (2)
| Title |
|---|
| 周润东: "基于时序数据的复杂石化过程关键指标预测方法研究", 《CNKI优秀硕士学位论文全文库工程科技Ⅰ辑》, 15 February 2023 (2023-02-15), pages 1 - 75 * |
| 邴吉帅等: "海洋多糖复合水凝胶的制备及性能研究", 《江苏海洋大学学报(自然科学版)》, 15 December 2022 (2022-12-15), pages 28 - 34 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN118227836B (en) | 2024-09-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106503574B (en) | Block chain safe storage method | |
| CN109417479A (en) | The rewritable block chain of cryptologic | |
| CN107508812A (en) | A kind of industry control network date storage method, call method and system | |
| CN108629193A (en) | A kind of encryption protection system and method for artificial nerve network model | |
| CN111698241A (en) | Internet of things cloud platform system, verification method and data management method | |
| CN114996755B (en) | Cache data safety protection system based on cloud computing | |
| CN114785563B (en) | Encryption malicious traffic detection method of soft voting strategy | |
| CN110717197A (en) | Credit investigation data sharing method based on block chain | |
| CN117527265B (en) | Internet of things data acquisition system and method based on distributed digital identity | |
| CN116032464A (en) | Property data encryption system based on quantum communication | |
| CN107426148A (en) | A kind of anti-reptile method and system based on running environment feature recognition | |
| Saravanan et al. | Privacy preserving using enhanced shadow honeypot technique for data retrieval in cloud computing | |
| CN115189966A (en) | Block chain private data encryption and decryption service system | |
| Zhao et al. | Block cipher identification scheme based on Hamming weight distribution | |
| CN110570303A (en) | Service information processing method and device, storage medium and server cluster | |
| CN118227836B (en) | Block chain-based data processing method, device and storage medium | |
| CN107392042A (en) | Electric network data monitoring method and device | |
| CN114666063A (en) | Traditional Hash algorithm-based digital asset tracing method | |
| Siddamsetti et al. | Anomaly detection in blockchain using machine learning | |
| CN113259122A (en) | Full-scene network security intelligent decision handling method based on artificial intelligence | |
| CN112235418B (en) | Cross-blockchain stable access authentication system and method | |
| CN118174956B (en) | DoH detection method based on self-attention BiLSTM | |
| CN118504037B (en) | Block chain-based electric energy meter data security management method | |
| CN117648704B (en) | Block chain-based data security interaction method, medium and system | |
| CN117726421B (en) | Rights management method applied to bank |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |