CN107038188A - A kind of Global ID's generation method and system based on business - Google Patents
A kind of Global ID's generation method and system based on business Download PDFInfo
- Publication number
- CN107038188A CN107038188A CN201610955454.3A CN201610955454A CN107038188A CN 107038188 A CN107038188 A CN 107038188A CN 201610955454 A CN201610955454 A CN 201610955454A CN 107038188 A CN107038188 A CN 107038188A
- Authority
- CN
- China
- Prior art keywords
- business
- global
- generation
- data
- operator
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2358—Change logging, detection, and notification
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Financial Or Insurance-Related Operations Such As Payment And Settlement (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
A kind of Global ID's generation method based on business, it can make the ID of generation be unique, and have business meaning, allow Data Analyst to can be obtained by many important informations according to ID, many such as join etc other nuisance operations are avoided, data analysis efficiency is improved.This method comprises the following steps：(1) digital coding is carried out to business datum, and hadoop cluster is arrived in the dictionary file storage of generation；(2) carrier data is obtained in real time, and writes hadoop cluster；(3) configuration schedules task, is periodically executed program, parses carrier data, and generates ID, imports data warehouse.Also a kind of Global ID's generation system based on business.
Description
Technical field
The present invention relates to the technical field of big data processing, more particularly to a kind of Global ID's generation method based on business,
And Global ID's generation system based on business.
Background technology
Prior art uses the generating mode of timestamp+random number mostly.Timestamp is usually to take system current
13 millisecond timestamps, random number is then the random number generation function according to built in system, and passes through certain processing, returns to one
The numeric string of individual fixed digit, is stitched together as whole ID.
Existing scheme has no problem when unit is performed, but in a distributed system, due to being parallel
Operation, in same millisecond, many machines can all perform same instructions, so as to cause the situation for duplicating ID, by test, 40
Hundred million or so the order of magnitude, can produce the repetition ID of 10,000 or so, and repetitive rate is about 40 a ten thousandths.
The content of the invention
To overcome the defect of prior art, the technical problem to be solved in the present invention is to provide a kind of overall situation based on business
ID generation methods, it can make the ID of generation be unique, and have business meaning, allow Data Analyst just can be with according to ID
Obtain many important informations, it is to avoid many such as join etc other nuisance operations, improve data analysis efficiency.
The technical scheme is that：This Global ID's generation method based on business, this method comprises the following steps：
(1) digital coding is carried out to business datum, and hadoop cluster is arrived in the dictionary file storage of generation；
(2) carrier data is obtained in real time, and writes hadoop cluster；
(3) configuration schedules task, is periodically executed program, parses carrier data, and generates ID, imports data warehouse.
The present invention is perfectly determined unique by the way that the machine ID in distributed system and counter are added in parameter
Property, and business datum is encoded, be unique so as to the ID that makes generation as generation ID substring, and had
Business meaning, allow Data Analyst to can be obtained by many important informations according to ID, it is to avoid many such as join etc its
His nuisance operation, improves data analysis efficiency.
A kind of Global ID's generation system based on business is additionally provided, the system includes：
Operation code module, it, which is configured, to carry out digital coding to business datum, and the dictionary file storage of generation is arrived
Hadoop cluster；
Carrier data acquisition module, it configures to obtain carrier data in real time, and write hadoop cluster；
ID generation modules, it, which is configured, carrys out configuration schedules task, is periodically executed program, parses carrier data, and generates ID,
Import data warehouse.
Brief description of the drawings
Fig. 1 show the flow chart of Global ID's generation method based on business according to the present invention.
Embodiment
As shown in figure 1, this Global ID's generation method based on business, this method comprises the following steps：
(1) digital coding is carried out to business datum, and hadoop cluster is arrived in the dictionary file storage of generation；
(2) carrier data is obtained in real time, and writes hadoop cluster；
(3) configuration schedules task, is periodically executed program, parses carrier data, and generates ID, imports data warehouse.
The present invention is perfectly determined unique by the way that the machine ID in distributed system and counter are added in parameter
Property, and business datum is encoded, be unique so as to the ID that makes generation as generation ID substring, and had
Business meaning, allow Data Analyst to can be obtained by many important informations according to ID, it is to avoid many such as join etc its
His nuisance operation, improves data analysis efficiency.
In addition, in the step (1), to the most concerned account type of user, operator and province data, having carried out numeral
Coding.
In addition, in the step (1), digital coding is represented using 2 ten's digits, account type, operator and province
Number is separately encoded according to separate.
Generation ID program is realized based on MapReduce frameworks.When Mapreduce is run, it can call in cluster
Many machines are performed parallel, wherein each machine has a taskid, different machines taskid is different, and we just utilize here
This characteristic.In addition during each machine processing data, a counter is had, i.e. counter, program often handles one
Data, operation that counter execution Jia one.
In addition, in the step (3), ID composition is：Timestamp adds industry plus counter plus machine node ID
Business coding.Generation ID uniqueness so can be both completely secured, and ID contains important business information, facilitates follow-up
Data analysis requirements.
In addition, in the step (3), parsing UID, operator, province, account type from carrier data, judging phase
Whether the UID answered had generated ID, was, skipped, and otherwise obtained the ID of current 13 timestamps and current machine node；
Then corresponding code conversion is carried out in order according to dictionary file to operator, province, account type, generates a digit groups
Into character string, and add one to counter, if counter exceed threshold value (threshold value can be 9999, also can according to it is actual voluntarily
Adjustment), then counter clear, and so on.
Can be with it will appreciated by the skilled person that realizing that all or part of step in above-described embodiment method is
The hardware of correlation is instructed to complete by program, described program can be stored in a computer read/write memory medium,
The program upon execution, includes each step of above-described embodiment method, and described storage medium can be：ROM/RAM, magnetic
Dish, CD, storage card etc..Therefore, the method with the present invention is corresponding, and the present invention is also simultaneously including a kind of based on the complete of business
Office ID generation systems, the system is generally represented in the form of the functional module corresponding with each step of method.Use this method
System includes：
Operation code module, it, which is configured, to carry out digital coding to business datum, and the dictionary file storage of generation is arrived
Hadoop cluster；
Carrier data acquisition module, it configures to obtain carrier data in real time, and write hadoop cluster；
ID generation modules, it, which is configured, carrys out configuration schedules task, is periodically executed program, parses carrier data, and generates ID,
Import data warehouse.
In addition, in the operation code module, to the most concerned account type of user, operator and province data, carrying out
Digital coding.
In addition, in the operation code module, digital coding is represented using 2 ten's digits, account type, operator
It is separate with province data, it is separately encoded.
In addition, in the ID generation modules, ID composition is：Timestamp is added plus counter plus machine node ID
Operation code.
In addition, in the ID generation modules, UID, operator, province, account type are parsed from carrier data, judge
Whether corresponding UID had generated ID, was to skip, and otherwise obtained current 13 timestamps and current machine node
ID；Then corresponding code conversion is carried out in order according to dictionary file to operator, province, account type, generates a number
The character string of word composition, and adds one to counter, if counter exceedes threshold value, counter clear, and so on.
Beneficial effects of the present invention are as follows：
1. it may insure id global uniqueness；
2. business datum add id, the resource utilization of the data storage of raising, reduce data parsing link and when
Between cost.
It is described above, be only presently preferred embodiments of the present invention, any formal limitation not made to the present invention, it is every according to
Any simple modification, equivalent variations and the modification made according to the technical spirit of the present invention to above example, still belong to the present invention
The protection domain of technical scheme.
Claims (10)
1. a kind of Global ID's generation method based on business, it is characterised in that：This method comprises the following steps：
(1) digital coding is carried out to business datum, and hadoop cluster is arrived in the dictionary file storage of generation；
(2) carrier data is obtained in real time, and writes hadoop cluster；
(3) configuration schedules task, is periodically executed program, parses carrier data, and generates ID, imports data warehouse.
2. Global ID's generation method according to claim 1 based on business, it is characterised in that：It is right in the step (1)
The most concerned account type of user, operator and province data, have carried out digital coding.
3. Global ID's generation method according to claim 2 based on business, it is characterised in that：In the step (1), number
Word coding represents that account type, operator and province data are separate, are separately encoded using 2 ten's digits.
4. Global ID's generation method according to claim 3 based on business, it is characterised in that：In the step (3), ID
Composition be：Timestamp adds operation code plus counter plus machine node ID.
5. Global ID's generation method according to claim 4 based on business, it is characterised in that：In the step (3), from
Carrier data parses UID, operator, province, account type, judges whether corresponding UID had generated ID, is, jumps
Cross, otherwise obtain the ID of current 13 timestamps and current machine node；Then to operator, province, account type according to
Dictionary file carries out corresponding code conversion in order, generates the character string of a numeral composition, and adds one to counter, if
Counter exceedes threshold value, then counter clear, and so on.
6. a kind of Global ID's generation system based on business, it is characterised in that：The system includes：
Operation code module, it, which is configured, to carry out digital coding to business datum, and the dictionary file storage of generation is arrived
Hadoop cluster；
Carrier data acquisition module, it configures to obtain carrier data in real time, and write hadoop cluster；
ID generation modules, it, which is configured, carrys out configuration schedules task, is periodically executed program, parses carrier data, and generates ID, imports
Data warehouse.
7. Global ID's generation system according to claim 6 based on business, it is characterised in that：The operation code module
In, to the most concerned account type of user, operator and province data, carry out digital coding.
8. Global ID's generation system according to claim 7 based on business, it is characterised in that：The operation code module
In, digital coding represents that account type, operator and province data are separate, are separately encoded using 2 ten's digits.
9. Global ID's generation system according to claim 8 based on business, it is characterised in that：The ID generation modules
In, ID composition is：Timestamp adds operation code plus counter plus machine node ID.
10. Global ID's generation system according to claim 9 based on business, it is characterised in that：The ID generation modules
In, UID, operator, province, account type are parsed from carrier data, judges whether corresponding UID had generated ID,
It is to skip, otherwise obtains the ID of current 13 timestamps and current machine node；Then to operator, province, account class
Type carries out corresponding code conversion in order according to dictionary file, generates the character string of a numeral composition, and counter is added
One, if counter exceed threshold value, counter clear, and so on.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610955454.3A CN107038188A (en) | 2016-11-03 | 2016-11-03 | A kind of Global ID's generation method and system based on business |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610955454.3A CN107038188A (en) | 2016-11-03 | 2016-11-03 | A kind of Global ID's generation method and system based on business |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107038188A true CN107038188A (en) | 2017-08-11 |
Family
ID=59530316
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610955454.3A Pending CN107038188A (en) | 2016-11-03 | 2016-11-03 | A kind of Global ID's generation method and system based on business |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107038188A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110109911A (en) * | 2018-01-10 | 2019-08-09 | 武汉斗鱼网络科技有限公司 | Distributed Global ID's generation method, storage medium, electronic equipment and method |
| CN112905703A (en) * | 2021-03-12 | 2021-06-04 | 北京格瑞拉科技有限公司 | User information management method and system |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103703724A (en) * | 2013-08-15 | 2014-04-02 | 华为技术有限公司 | Resource payment method |
| CN104907259A (en) * | 2015-05-20 | 2015-09-16 | 深圳深若科技有限公司 | Unified coding-based express delivery object sorting method and system |
| CN105979021A (en) * | 2016-04-26 | 2016-09-28 | 北京金山安全软件有限公司 | Information processing method and user terminal |
-
2016
- 2016-11-03 CN CN201610955454.3A patent/CN107038188A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103703724A (en) * | 2013-08-15 | 2014-04-02 | 华为技术有限公司 | Resource payment method |
| CN104907259A (en) * | 2015-05-20 | 2015-09-16 | 深圳深若科技有限公司 | Unified coding-based express delivery object sorting method and system |
| CN105979021A (en) * | 2016-04-26 | 2016-09-28 | 北京金山安全软件有限公司 | Information processing method and user terminal |
Non-Patent Citations (2)
| Title |
|---|
| WHTHOMAS: "全局唯一ID设计", 《BBSMAX》 * |
| 王延炯-极客时间: "生成全局唯一ID的3个思路，来自一个资深架构师的总结", 《微信公众号》 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110109911A (en) * | 2018-01-10 | 2019-08-09 | 武汉斗鱼网络科技有限公司 | Distributed Global ID's generation method, storage medium, electronic equipment and method |
| CN112905703A (en) * | 2021-03-12 | 2021-06-04 | 北京格瑞拉科技有限公司 | User information management method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN100489879C (en) | Method, system and server for checking page data | |
| CN107562556B (en) | Failure recovery method, recovery device and storage medium | |
| CN103092621B (en) | The method and device that a kind of chart generates | |
| CN104504084A (en) | Method and device for determining user retention rate | |
| CN109542997B (en) | Motion trail display method and device | |
| CN108228166A (en) | A kind of back-end code generation method and system based on template | |
| CN112286772B (en) | Attribution analysis method and device and electronic equipment | |
| CN111431926B (en) | Data association analysis method, system, equipment and readable storage medium | |
| CN108694214A (en) | Generation method, generating means, readable medium and the electronic equipment of data sheet | |
| CN111126019B (en) | Report generation method and device based on mode customization and electronic equipment | |
| CN110781180B (en) | Data screening method and data screening device | |
| CN111400170B (en) | Data authority testing method and device | |
| CN106250310A (en) | A kind of method for generating test case and device | |
| CN110502520A (en) | A kind of method of data loading, system, equipment and computer readable storage medium | |
| CN115567607A (en) | Processing method, device and system for calling link, electronic equipment and storage medium | |
| CN107038188A (en) | A kind of Global ID's generation method and system based on business | |
| CN114820080A (en) | User grouping method, system, device and medium based on crowd circulation | |
| CN107479866A (en) | The method that open terminal applies data and function are realized based on reconfiguration technique | |
| CN115328759A (en) | Form verification method and device | |
| CN114493634A (en) | Method, device and medium for generating product traceability information based on identification analysis | |
| CN111949720A (en) | Data analysis method based on big data and artificial intelligence and cloud data server | |
| CN104599092A (en) | Order business monitoring method and equipment | |
| CN108132783A (en) | A kind of configuration method and device of Web App | |
| CN104102672A (en) | Data information processing system and method | |
| CN109213821B (en) | Data processing method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20170811 |