CN106572172A - Multi-process data migration method based on Hash algorithm - Google Patents
Multi-process data migration method based on Hash algorithm Download PDFInfo
- Publication number
- CN106572172A CN106572172A CN201610975800.4A CN201610975800A CN106572172A CN 106572172 A CN106572172 A CN 106572172A CN 201610975800 A CN201610975800 A CN 201610975800A CN 106572172 A CN106572172 A CN 106572172A
- Authority
- CN
- China
- Prior art keywords
- data
- data migration
- hash algorithm
- migration method
- hash
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/563—Data redirection of data network streams
Abstract
The invention discloses a multi-process data migration method based on the Hash algorithm; the method comprises the following steps: S1, determining a Hash value; S2, using the Hash algorithm to group the migration data according to the Hash value; S3, finishing the multi-process data migration. The migration method can effectively solve the low efficiency and long time consuming problems when mass data are downloaded.
Description
Technical field
The invention belongs to database data migration technical field, and in particular to a kind of multi-process data based on hash algorithm
Migration.
Background technology
Data Migration refers to the operation that data are moved to specified storage device according to specified strategy from former storage device.
Traditional data moving method typically by the way of " DB loadings ", is mainly shown as the modes such as IMPORT, LOAD.This realization side
Formula is the realization based on database self-technique, but is limited to the restriction of number of processes and cannot realize loaded in parallel data.
The use that wherein IMPORT modes are is the medium after being derived based on EXPORT, carries out importing action.Due to deriving
Memory space is huge needed for medium, and cause needs frequently to carry out transferring to other use for memory space during enforcement, when wasting substantial amounts of
Between.And when inserting to mass data, execution efficiency is reduced, server overhead increase causes service disconnection under extreme case.
The importing efficiency of LOAD is more much higher than IMPORT/INSERT, and reason is the page that the instrument is directed to database
Operated, therefore saved the importing expense of the overwhelming majority.But in data loading procedure, col widths of the LOAD to table object
There is strict demand, if the width for having a column data is 9, but the width of the row is 6 defined in table, therefore for each line number
According to load instruments have all carried out break-in operation to the row, are greatly reduced so as to cause overall data to load speed, and data are complete
Property it is destroyed.
The content of the invention
Present invention seek to address that one of technical problem present in prior art, for this purpose, it is an object of the present invention to
There is provided a kind of multi-process data migration method based on hash algorithm is proposed, effectively loading timeliness can be carried out to mass data
Rate is low, the problems such as time-consuming.
It should be noted that the present invention is completed based on the following discovery of inventor：
According to an aspect of the present invention, the invention provides a kind of multi-process Data Migration side based on hash algorithm
Method, comprises the following steps:
S1, determine cryptographic Hash：Extract the cryptographic Hash of data to be migrated；
S2, packet：According to cryptographic Hash, migrating data is grouped by hash algorithm；
S3, Data Migration：Multi-process Data Migration is completed to the data after packet.
In addition, a kind of multi-process data migration method based on hash algorithm according to the above embodiment of the present invention, may be used also
With with following additional technical characteristic：
Step S1 includes：
Check bit in S101, extraction customer ID；
S102, increase weighting digit；
S103, basis estimate the quantity of distribution process, and by hash algorithm cryptographic Hash is extracted.
Embodiments in accordance with the present invention, the cryptographic Hash is maximum up to 128.
Embodiments in accordance with the present invention, step S2 includes：According to the cryptographic Hash that step S1 is extracted, by hash algorithm
Data are grouped, it is ensured that duplicate data will not be produced between each group of data, and the record of same customer ID can be assigned to
In same group, accounting standard is met.
Embodiments in accordance with the present invention, step S3 also includes：
Start multi-process data parallel Data Migration.
Embodiments in accordance with the present invention, the multi-process is 128 to the maximum.
Implement the multi-process data migration method based on hash algorithm that the present invention is provided, have the advantages that：And
Row Data Migration, it is ensured that adopt hash algorithm, n numbers that duplicate data will not be produced according between while data integrity, and with visitor
The record at family number can be placed in same data file.While transport efficiency is lifted, accounting calculation principle has been observed.
The additional aspect and advantage of the present invention will be set forth in part in the description, and partly will become from the following description
Obtain substantially, or recognized by the practice of the present invention.
Description of the drawings
Fig. 1 is the schematic flow sheet based on the multi-process data migration method of hash algorithm.
Specific embodiment
Embodiments of the invention are described below in detail.Below with reference to Description of Drawings embodiment be it is exemplary, only
For explaining the present invention, and it is not considered as limiting the invention.
Comprised the steps of based on the multi-process data migration method of hash algorithm：
S1, determine cryptographic Hash；
S2, according to cryptographic Hash, migrating data is grouped by hash algorithm；
S3, complete multi-process Data Migration.
In the multi-process data migration method based on hash algorithm of the present invention, step S1 includes：
Some check bit (check bit computational methods are banking secrecy information) in by extracting customer ID, and by increasing certain
A little weighting digits, and the quantity of distribution process is estimated, cryptographic Hash is extracted according to hash algorithm, cryptographic Hash is maximum up to 128.
In the multi-process data migration method based on hash algorithm of the present invention, step S2 includes：
Data are grouped according to cryptographic Hash.Assume that user logging quantity has M (having many with customer ID record), breathe out
Uncommon value is N number of, and last all of user logging can be assigned in N number of group by hash algorithm, there is ≈ M/N bars visitor in each group
Family information record.And the record with customer ID is bound to be assigned in same group, meets accounting standard,.
In the multi-process data migration method based on hash algorithm of the present invention, step S3 includes：
Start the migration of multi-process data parallel.
In the description of this specification, reference term " one embodiment ", " some embodiments ", " example ", " specifically show
The description of example " or " some examples " etc. means to combine specific features, structure, material or spy that the embodiment or example are described
Point is contained at least one embodiment of the present invention or example.In this manual, to the schematic representation of above-mentioned term not
Necessarily refer to identical embodiment or example.And, the specific features of description, structure, material or feature can be any
One or more embodiments or example in combine in an appropriate manner.
Although an embodiment of the present invention has been shown and described, it will be understood by those skilled in the art that：Not
These embodiments can be carried out with various changes, modification, replacement and modification in the case of the principle and objective that depart from the present invention, this
The scope of invention is limited by claim and its equivalent.
Claims (6)
1. a kind of multi-process data migration method based on hash algorithm, it is characterised in that comprise the following steps:
S1, determine cryptographic Hash：Extract the cryptographic Hash of data to be migrated；
S2, packet：According to cryptographic Hash, migrating data is grouped by hash algorithm；
S3, Data Migration：Multi-process Data Migration is completed to the data after packet.
2. data migration method according to claim 1, it is characterised in that step S1 includes：
Check bit in S101, extraction customer ID；
S102, increase weighting digit；
S103, basis estimate the quantity of distribution process, and by hash algorithm cryptographic Hash is extracted.
3. data migration method according to claim 2, it is characterised in that the cryptographic Hash is maximum up to 128.
4. data migration method according to claim 1, it is characterised in that step S2 includes：
According to the cryptographic Hash that step S1 is extracted, data are grouped by hash algorithm, it is ensured that will not produce between each group of data
Duplicate data, and the record of same customer ID can be assigned in same group, meet accounting standard.
5. data migration method according to claim 1, it is characterised in that step S3 also includes：
Start multi-process data parallel Data Migration.
6. data migration method according to claim 5, it is characterised in that the multi-process is 128 to the maximum.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610975800.4A CN106572172A (en) | 2016-11-07 | 2016-11-07 | Multi-process data migration method based on Hash algorithm |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610975800.4A CN106572172A (en) | 2016-11-07 | 2016-11-07 | Multi-process data migration method based on Hash algorithm |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN106572172A true CN106572172A (en) | 2017-04-19 |
Family
ID=58540146
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610975800.4A Pending CN106572172A (en) | 2016-11-07 | 2016-11-07 | Multi-process data migration method based on Hash algorithm |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106572172A (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103914458A (en) * | 2012-12-29 | 2014-07-09 | 中国移动通信集团河北有限公司 | Mass data migration method and device |
| CN105279280A (en) * | 2015-11-16 | 2016-01-27 | 天津南大通用数据技术股份有限公司 | Method and tool for quickly migrating oracle data to MPP database |
-
2016
- 2016-11-07 CN CN201610975800.4A patent/CN106572172A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103914458A (en) * | 2012-12-29 | 2014-07-09 | 中国移动通信集团河北有限公司 | Mass data migration method and device |
| CN105279280A (en) * | 2015-11-16 | 2016-01-27 | 天津南大通用数据技术股份有限公司 | Method and tool for quickly migrating oracle data to MPP database |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103136243B (en) | File system duplicate removal method based on cloud storage and device | |
| CN105468298B (en) | A kind of key assignments storage method based on log-structured merging tree | |
| CN106407224A (en) | Method and device for file compaction in KV (Key-Value)-Store system | |
| JP2012531674A5 (en) | ||
| CN103617097B (en) | File access pattern method and device | |
| CN107102819A (en) | The method and apparatus of data is write to solid state hard disc | |
| CN104536847B (en) | It is a kind of to improve the method that data write integrality | |
| CN109324758A (en) | Data migration method, device and storage equipment | |
| CN107239569A (en) | A kind of distributed file system subtree storage method and device | |
| CN104408190A (en) | Spark based data processing method and device | |
| CN114708133B (en) | Universal text watermarking method and device | |
| CN106682215A (en) | Data processing method and management node | |
| CN102708183A (en) | Method and device for data compression | |
| CN110880143A (en) | System and method for processing transaction verification operations in decentralized applications | |
| CN107665219A (en) | A kind of blog management method and device | |
| CN103617124B (en) | Flash memory management method and device | |
| CN104991741B (en) | A kind of situation adaptation power network big data storage method based on key-value model | |
| CN106775470A (en) | A kind of method and system of data storage | |
| CN107391040A (en) | A kind of method and device of storage array disk I O scheduling | |
| CN108646987A (en) | A kind of management method of file volume, device, storage medium and terminal | |
| CN106572172A (en) | Multi-process data migration method based on Hash algorithm | |
| WO2015087651A1 (en) | Device, program, recording medium, and method for extending service life of memory, | |
| CN105260130A (en) | Read-write method for Seagate hard disk system file | |
| CN108009041A (en) | A kind of flash array checksum update method known based on data correlation sexuality | |
| CN105117169A (en) | Optimized disk space management method and apparatus |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20170419 |
|
| RJ01 | Rejection of invention patent application after publication |