CN116627969A - Method, system and computer storage medium for constructing IDmapping based on Flink - Google Patents
Method, system and computer storage medium for constructing IDmapping based on Flink Download PDFInfo
- Publication number
- CN116627969A CN116627969A CN202310513852.XA CN202310513852A CN116627969A CN 116627969 A CN116627969 A CN 116627969A CN 202310513852 A CN202310513852 A CN 202310513852A CN 116627969 A CN116627969 A CN 116627969A
- Authority
- CN
- China
- Prior art keywords
- data
- registration
- idmapping
- generated
- user
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 25
- 238000010586 diagram Methods 0.000 claims abstract description 16
- 238000004140 cleaning Methods 0.000 claims description 3
- 238000013507 mapping Methods 0.000 description 3
- 101100289995 Caenorhabditis elegans mac-1 gene Proteins 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
- G06F16/2272—Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/283—Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The invention provides a method for constructing IDmapping based on a Flink, which comprises the following steps: s1, acquiring equipment identification ID data of different mobile equipment login of a user in a mobile terminal; s2, acquiring different registration ID data registered by a server side user; s3, intersecting the equipment identification ID data acquired in the step S1 with the registration ID data acquired in the step S2 to create a first data set; s4, eliminating the first data set in the step S3, and creating a second data set; s5, merging the first data set and the second data set to obtain a third data set; s6, reading a third data set by using the Flink, generating a point set and an edge set, and performing graph calculation through the generated point set and the generated edge set to generate a large-pass graph; and S7, reading all points in the large-pass diagram by using the Flink, converting all points in the large-pass diagram into data in a plaintext format through a unique function which is generated at will, outputting the data, storing the data in a Hive table, and constructing IDmapping. The invention can realize the association of different mobile devices and different registration accounts of the same user.
Description
Technical Field
The invention relates to the technical field of data processing, in particular to a method, a system and a computer storage medium for constructing IDmapping based on a Flink.
Background
With the continuous development and progress of society, each person (user) has a plurality of mobile phones and registers different account numbers, and the same person (user) has a plurality of mobile phones and a plurality of registration account numbers. It is now necessary to consider a user who has as many handsets or accounts the same person (user) as one user. The user unique identification mapping is required based on the device identification information and registration information.
Disclosure of Invention
The invention provides a method, a system and a computer storage medium for constructing IDmapping based on a Flink, which are used for solving the technical problem that different mobile terminal devices and different registration information of the same user are difficult to be associated and identified in the prior art.
One aspect of the present invention provides a method for constructing IDMapping based on a link, the method comprising the following method steps:
s1, acquiring equipment identification ID data of different mobile equipment logins of a user on a mobile terminal, and synchronizing the equipment identification ID data to a data warehouse;
s2, acquiring different registration ID data registered by a server user, and synchronizing the different registration ID data to a data warehouse;
s3, intersecting the equipment identification ID data acquired in the step S1 with the registration ID data acquired in the step S2 to create a first data set;
s4, eliminating the first data set in the step S3 from the equipment identification ID data acquired in the step S1 and the registration ID data acquired in the step S2, and creating a second data set;
s5, merging the first data set created in the step S3 with the second data set created in the step S4 to obtain a third data set;
s6, reading a third data set by using the Flink, generating a point set and an edge set, and performing graph calculation through the generated point set and the generated edge set to generate a large-pass graph;
and S7, reading all points in the large-pass diagram generated in the step S6 by using the Flink, converting all points in the large-pass diagram into data in a plaintext format through a unique function which is generated at will, outputting the data, and storing the data in a Hive table to construct IDmapping.
In a preferred embodiment, in step S1, field deduplication is performed on the obtained device identification ID data of the user logged in by different mobile devices of the mobile terminal, and the deduplicated data is stored in the Hive table.
In a preferred embodiment, in step S2, field deduplication is performed on different registration ID data registered by the acquiring server-side user, and the deduplicated data is stored in the Hive table.
In a preferred embodiment, when new device identification ID data or registration ID data is added, the newly added device identification ID data or registration ID data is inserted into the third data set generated in step S5;
repeating the steps S6 to S7, and constructing IDmapping.
In a preferred embodiment, before storing in the Hive table in step S7, the method further comprises: and performing data cleaning on the output plaintext format data by using an ETL program.
Another aspect of the present invention is to provide a system for constructing IDMapping based on a link, wherein the system includes: the data acquisition module is used for acquiring equipment Identification (ID) data of different mobile equipment logins of a user on the mobile terminal, acquiring different registration ID data of the user registration of the server side and synchronizing the different registration ID data to the data warehouse;
and the graph calculation module is used for reading the third data set by using the Flink, generating a point set and an edge set, and performing graph calculation through the generated point set and the generated edge set to generate a large-pass graph.
And the IDmapping generation module is used for reading all points in the large-pass diagram by using the Flink, converting all the points in the large-pass diagram into data in a plaintext format through an arbitrarily generated unique function, outputting the data, and storing the data in a Hive table to construct IDmapping.
Yet another aspect of the present invention is to provide a computer storage medium for storing a computer-implemented program for executing a method of constructing IDMapping based on a link provided by the present invention.
Compared with the prior art, the invention has the following beneficial effects:
the method, the system and the computer storage medium for constructing IDmapping based on the Flink can realize the association of different mobile devices and different registration accounts of the same user and realize the mapping of the user equipment identification information and the registration information with the unique identification of the user.
The method, the system and the computer storage medium for constructing IDmapping based on the Flink can realize the new equipment identification information of the user, and the registration information is updated in time and mapped with the unique identification of the user.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are needed in the description of the embodiments or the prior art will be briefly described, and it is obvious that the drawings in the description below are some embodiments of the present invention, and other drawings can be obtained according to the drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of a method for constructing IDmapping based on a Flink.
FIG. 2 is a schematic flow chart of data acquisition according to the present invention.
Detailed Description
To further clarify the above and other features and advantages of the present invention, a further description of the invention will be rendered by reference to the appended drawings. It should be understood that the specific embodiments presented herein are for purposes of explanation to those skilled in the art and are intended to be illustrative only and not limiting.
In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", "axial", "radial", "circumferential", etc. indicate orientations or positional relationships based on the orientations or positional relationships shown in the drawings are merely for convenience in describing the present invention and simplifying the description, and do not indicate or imply that the device or element being referred to must have a specific orientation, be configured and operated in a specific orientation, and therefore should not be construed as limiting the present invention.
Furthermore, the terms "first," "second," and the like, are used for descriptive purposes only and are not to be construed as indicating or implying a relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defining "a first" or "a second" may explicitly or implicitly include at least one such feature. In the description of the present invention, the meaning of "plurality" means at least two, for example, two, three, etc., unless specifically defined otherwise.
As shown in fig. 1, which is a schematic flow chart of a method for constructing IDMapping based on a link, and as shown in fig. 2, which is a schematic flow chart of data acquisition of the present invention, according to an embodiment of the present invention, a method for constructing IDMapping based on a link is provided, including the following method steps:
and S1, acquiring mobile terminal data.
Device identification ID data of user login at different mobile devices of the mobile terminal 100 is acquired and synchronized to the data repository 300.
And S2, acquiring server-side data.
Different registration ID data for the server side 200 user registration is obtained and synchronized to the data repository 300.
Step S1 and step S2 are processes of data collection, in step S1, field deduplication is performed on the obtained device identification ID data of the user logged in by different mobile devices of the mobile terminal 100, and the deduplicated data is stored in the Hive table of the data warehouse 300.
For example, the device ID data logged in by the user on different mobile devices of the mobile terminal 100 is imei\mac\android_id\registration ID, deduplication is performed based on the imei\mac\android_id\registration ID field, and the device ID data after deduplication is stored in the Hive table of the data repository 300.
In step S2, field deduplication is performed on different registration ID data registered by the user of the acquisition server 200, and the deduplicated data is stored in the Hive table of the data repository 300.
For example, the different registration ID data registered by the user is registration id\registration phone number\mailbox, duplication removal is performed based on the registration id\registration phone number\mailbox field, and the duplicated registration ID data is stored in the table of the database 300 Hive.
And step S3, creating a first data set.
The device identification ID data acquired in step S1 is intersected with the registration ID data acquired in step S2, a first data set is created, and the data in the first data set is stored in the Hive table of the data warehouse 300.
And S4, creating a second data set.
In the device identification ID data acquired in step S1 and the registration ID data acquired in step S2, the first data set in step S3 is rejected, a second data set is created, and the data in the second data set is stored in the Hive table of the data warehouse 300.
Step S5, creating a third data set.
Combining the first data set created in step S3 with the second data set created in step S4 to obtain a third data set, and storing the data in the third data set in the Hive table of the data warehouse 300.
If there is a data information field in the third data set that has no value, the data field with no value is stored in the Hive table of the data warehouse 300 in an empty manner.
And S6, generating a large-pass diagram.
And reading the third data set by using the Flink, generating a point set and an edge set, and performing graph calculation through the generated point set and the generated edge set to generate a large-pass graph.
The core of graph calculation is that data is expressed as ' points ', edges ' are established between the points, and data relations are found from the points and edges: such as connectivity, e.g., shortest path planning, last target of id_mapping, etc.
The large through diagram is used for constructing IDmapping, the data in the third data set is used as a point, and the value of the data is used as an edge.
And S7, converting into a plaintext format, storing the plaintext format into a Hive table, and constructing IDmapping.
And (3) reading all points in the large-pass diagram generated in the step S6 by using the Flink, converting all points in the large-pass diagram into plaintext format data through an arbitrarily generated unique function, outputting the plaintext format data, cleaning the output plaintext format data by using an ETL program, storing the cleaned plaintext format data into a Hive table, and constructing IDmapping.
In the embodiment of the present invention, the format of the plaintext format data to be output is as follows:
unique ID→imei1\imei2\mac\registration ID 1\registration ID 2\registration telephone number.
The IDmapping constructed in the Hive table is finally stored as follows:
unique ID | Device identification ID/registration ID |
A0001 | Imei1 |
A0001 | Imei2 |
A0001 | Mac1 |
A0001 | Registration ID1 |
According to the embodiment of the invention, when the user adds new device identification ID data or registration ID data, the new device identification ID data or registration ID data is inserted into the third data set generated in step S5, and steps S6 to S7 are repeated to construct IDMapping.
Further, after the newly added device identification ID data or registration ID data is added into the third data set, comparing the newly added data with the data in the third data set, and when the newly added data already exists in the third data set, not executing the steps S6 to S7; and when the third data set does not have the newly added data, executing the steps S6 to S7 again to construct IDmapping.
For example, the user adds data: register ID2 and register ID3, then the IDMapping constructed in the Hive table is finally stored as follows:
unique ID | Device identification ID/registration ID |
A0001 | Imei1 |
A0001 | Imei2 |
A0001 | Mac1 |
A0001 | Registration ID1 |
A0001 | Registration ID2 |
A0002 | Registration ID3 |
According to an embodiment of the present invention, there is provided a system for constructing IDMapping based on a link, including: the data acquisition module is used for acquiring equipment Identification (ID) data of different mobile equipment logins of a user on the mobile terminal, acquiring different registration ID data of the user registration of the server side and synchronizing the different registration ID data to the data warehouse;
and the graph calculation module is used for reading the third data set by using the Flink, generating a point set and an edge set, and performing graph calculation through the generated point set and the generated edge set to generate a large-pass graph.
And the IDmapping generation module is used for reading all points in the large-pass diagram by using the Flink, converting all the points in the large-pass diagram into data in a plaintext format through an arbitrarily generated unique function, outputting the data, and storing the data in a Hive table to construct IDmapping.
According to an embodiment of the present invention, a computer storage medium is configured to store a computer-implemented program, where the computer-implemented program is configured to implement a method for constructing IDMapping based on a link.
While embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and not to be construed as limiting the invention, and that variations, modifications, alternatives and variations may be made to the above embodiments by one of ordinary skill in the art within the scope of the invention.
Claims (7)
1. A method for constructing IDMapping based on a link, which is characterized by comprising the following method steps:
s1, acquiring equipment identification ID data of different mobile equipment logins of a user on a mobile terminal, and synchronizing the equipment identification ID data to a data warehouse;
s2, acquiring different registration ID data registered by a server user, and synchronizing the different registration ID data to a data warehouse;
s3, intersecting the equipment identification ID data acquired in the step S1 with the registration ID data acquired in the step S2 to create a first data set;
s4, eliminating the first data set in the step S3 from the equipment identification ID data acquired in the step S1 and the registration ID data acquired in the step S2, and creating a second data set;
s5, merging the first data set created in the step S3 with the second data set created in the step S4 to obtain a third data set;
s6, reading a third data set by using the Flink, generating a point set and an edge set, and performing graph calculation through the generated point set and the generated edge set to generate a large-pass graph;
and S7, reading all points in the large-pass diagram generated in the step S6 by using the Flink, converting all points in the large-pass diagram into data in a plaintext format through a unique function which is generated at will, outputting the data, and storing the data in a Hive table to construct IDmapping.
2. The method according to claim 1, wherein in step S1, field deduplication is performed on the obtained device identification ID data of the user logged in at different mobile devices of the mobile terminal, and the deduplicated data is stored in the Hive table.
3. The method according to claim 1, wherein in step S2, field deduplication is performed on different registration ID data registered by the acquiring server-side user, and the deduplicated data is stored in the Hive table.
4. The method according to claim 1, wherein when new device identification ID data or registration ID data is added, the newly added device identification ID data or registration ID data is inserted into the third data set generated in step S5;
repeating the steps S6 to S7, and constructing IDmapping.
5. The method according to claim 1, further comprising, before storing in the Hive table in step S7: and performing data cleaning on the output plaintext format data by using an ETL program.
6. A system for constructing IDMapping based on a link, the system comprising: the data acquisition module is used for acquiring equipment Identification (ID) data of different mobile equipment logins of a user on the mobile terminal, acquiring different registration ID data of the user registration of the server side and synchronizing the different registration ID data to the data warehouse;
and the graph calculation module is used for reading the third data set by using the Flink, generating a point set and an edge set, and performing graph calculation through the generated point set and the generated edge set to generate a large-pass graph.
And the IDmapping generation module is used for reading all points in the large-pass diagram by using the Flink, converting all the points in the large-pass diagram into data in a plaintext format through an arbitrarily generated unique function, outputting the data, and storing the data in a Hive table to construct IDmapping.
7. A computer storage medium for storing a computer-executable program for performing the method of any one of claims 1 to 5.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202310513852.XA CN116627969A (en) | 2023-05-08 | 2023-05-08 | Method, system and computer storage medium for constructing IDmapping based on Flink |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202310513852.XA CN116627969A (en) | 2023-05-08 | 2023-05-08 | Method, system and computer storage medium for constructing IDmapping based on Flink |
Publications (1)
Publication Number | Publication Date |
---|---|
CN116627969A true CN116627969A (en) | 2023-08-22 |
Family
ID=87620446
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202310513852.XA Pending CN116627969A (en) | 2023-05-08 | 2023-05-08 | Method, system and computer storage medium for constructing IDmapping based on Flink |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN116627969A (en) |
-
2023
- 2023-05-08 CN CN202310513852.XA patent/CN116627969A/en active Pending
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN107977396B (en) | Method and device for updating data table of KeyValue database | |
CN107622102A (en) | Entity card number generation method and terminal device | |
CN105630803B (en) | The method and apparatus that Document image analysis establishes index | |
CN107862047B (en) | Natural person data processing method and system based on multiple data sources | |
CN110765272A (en) | Knowledge graph-based signing method and device, computer equipment and computer storage medium | |
CN107992492A (en) | A kind of storage method of data block, read method, its device and block chain | |
CN109522332A (en) | Customer profile data merging method, device, equipment and readable storage medium storing program for executing | |
CN109902087A (en) | For the data processing method and device of question and answer, server | |
CN103997557A (en) | Address list management method and mobile terminal | |
CN104408183B (en) | The data lead-in method and device of data system | |
CN106131134A (en) | A kind of message content merges De-weight method and system | |
CN116627969A (en) | Method, system and computer storage medium for constructing IDmapping based on Flink | |
CN109739859B (en) | Relation map drawing method, system and related device | |
US20200192938A1 (en) | Graph creation from multiple databases | |
CN106250243B (en) | The processing method and processing device of banking system application based on poll tupe | |
CN104239307A (en) | User information storage method and system | |
CN102982422B (en) | The processing method and processing device of technology document data | |
CN112906091A (en) | Geographic information drawing method, device, terminal and medium based on CAD | |
CN107515867A (en) | The generation method and device that data storage, querying method and the device and a kind of rowKey of a kind of NoSQL databases combine entirely | |
CN111488345A (en) | Storage optimization method and device for service data block chain | |
CN114547184A (en) | Personnel information synchronization method, terminal device and storage medium | |
CN112257284A (en) | Design knowledge tracing method and system in CAD system based on block chain | |
CN110414984A (en) | Auth method and Related product based on block chain | |
CN110569362B (en) | Ontology instantiation method and device | |
CN111949839B (en) | Data association method, electronic device and medium |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination |