CN115952150A - Multi-source heterogeneous data fusion method and device - Google Patents
Multi-source heterogeneous data fusion method and device Download PDFInfo
- Publication number
- CN115952150A CN115952150A CN202211610209.0A CN202211610209A CN115952150A CN 115952150 A CN115952150 A CN 115952150A CN 202211610209 A CN202211610209 A CN 202211610209A CN 115952150 A CN115952150 A CN 115952150A
- Authority
- CN
- China
- Prior art keywords
- data
- fused
- field
- association
- fusion
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Abstract
The application relates to a multi-source heterogeneous data fusion method and device, relating to the technical field of data processing, wherein the method comprises the following steps: acquiring data to be fused, endowing different weight operators to each field in the data to be fused, performing association fusion on the data to be fused according to a preset data association rule and the weight operators of each field in the data to be fused to obtain data to be processed, and finally performing standardized processing on the data to be processed. In the method, various factors are comprehensively considered, different weight operators are given to each field in the data to be fused, for example, a field with high expected accuracy is given a relatively high weight operator, a field with low expected accuracy is given a relatively low weight operator, the close dependence degree of the fused data on the data is higher during the association fusion, and the obtained fused data is more accurate and reliable.
Description
Technical Field
The application relates to the technical field of data processing, in particular to a multi-source heterogeneous data fusion method and device.
Background
In the information construction process, due to the influence of factors such as the stage, the technology, other economic factors and human factors of the construction and implementation of the data management system of each business system, a large amount of business data adopting different storage modes are accumulated in the development process, and the data management system also comprises a simple file database and a complex network database, so that a large amount of multi-source heterogeneous data exists.
The multi-source heterogeneous data has the characteristics of structural data and non-structural data mixed, low data quality and non-uniform data standard, so that the data analysis is difficult to complete by using a uniform data model or a data algorithm.
The multi-source heterogeneous data fusion is to synthesize various data information, extract the characteristics of different data sources and extract unified information which is better and richer than single data, thereby assisting various government decision analysis.
The existing data fusion means generally comprise three types, namely data-level fusion, model-level fusion and decision-level fusion, but the data fusion means have the problems of low data accuracy and reliability when multi-source heterogeneous data fusion is carried out.
Disclosure of Invention
In order to overcome the problem of low data accuracy and reliability existing in the fusion of multi-source heterogeneous data by a data fusion means in the related technology at least to a certain extent, the application provides a multi-source heterogeneous data fusion method and device.
The scheme of the application is as follows:
according to a first aspect of embodiments of the present application, a multi-source heterogeneous data fusion method is provided, including:
acquiring data to be fused;
different weight operators are given to each field in the data to be fused;
performing association fusion on the data to be fused according to a preset data association rule and a weight operator of each field in the data to be fused to obtain data to be processed;
and carrying out standardization processing on the data to be processed.
Preferably, the acquiring the data to be fused includes:
and acquiring the data to be fused from a plurality of data sources.
Preferably, the assigning different weight operators to each field in the data to be fused includes:
dividing the data to be fused into a plurality of types of fields, wherein each type of field comprises a plurality of subfields;
determining the level of each data source and determining the service attribution of each subfield;
and determining the weight operator of each sub-field under each type of field according to the level of each data source and the service attribution of each sub-field.
Preferably, the method further comprises:
and determining the subjective factor influence value of each subfield, and calibrating the weight operator of each subfield according to the subjective factor influence value of each subfield.
Preferably, the performing association fusion on the data to be fused according to a preset data association rule and a weight operator of each field in the data to be fused includes:
calculating the accuracy value of each subfield in the data to be fused relative to the field category to which each subfield belongs according to the weight operator of each subfield in the data to be fused;
and obtaining the fused uniform data value of each field as fused data according to the accuracy value of each subfield relative to the field category to which the subfield belongs.
Preferably, the performing association fusion on the data to be fused according to a preset data association rule and a weight operator of each field in the data to be fused includes:
determining a unique association identifier in the fused data;
associating the data which has relevance with the unique association identifier in the fused data according to the same data structure; the data structure includes at least: library tables and data items;
establishing a data association model, wherein the data association model is used for constructing an association relation between data structures;
acquiring specific numerical information of the data to be fused, and verifying the data association model according to the specific numerical information of the data to be fused;
and when the verification is passed, using the associated data as the data to be processed.
Preferably, the method further comprises: and when the verification is passed, issuing the data association model.
Preferably, the method further comprises: and cleaning and reprocessing the data to be processed.
Preferably, the normalizing the data to be processed includes:
and converting the fused data of various fields in the data to be processed into corresponding coded values in a corresponding data conversion mode.
According to a second aspect of the embodiments of the present application, there is provided a multi-source heterogeneous data fusion apparatus, including:
the acquisition module is used for acquiring data to be fused;
the weighting module is used for giving different weight operators to each field in the data to be fused;
the association fusion module is used for performing association fusion on the data to be fused according to a preset data association rule and a weight operator of each field in the data to be fused to obtain data to be processed;
and the processing module is used for carrying out standardized processing on the data to be processed.
The technical scheme provided by the application can comprise the following beneficial effects: the multi-source heterogeneous data fusion method in the application comprises the following steps: acquiring data to be fused, endowing different weight operators to each field in the data to be fused, performing association fusion on the data to be fused according to a preset data association rule and the weight operators of each field in the data to be fused to obtain data to be processed, and finally performing standardized processing on the data to be processed. In the method, various factors are comprehensively considered, different weight operators are given to each field in the data to be fused, for example, a field with high expected accuracy is given a relatively high weight operator, a field with low expected accuracy is given a relatively low weight operator, the close dependence degree of the fused data on the data is higher during the association fusion, and the obtained fused data is more accurate and reliable.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and, together with the description, serve to explain the principles of the application.
Fig. 1 is a schematic flowchart of a multi-source heterogeneous data fusion method according to an embodiment of the present application;
fig. 2 is a schematic flowchart of a multi-source heterogeneous data fusion method according to another embodiment of the present application;
fig. 3 is a block diagram of a multi-source heterogeneous data fusion apparatus according to an embodiment of the present application.
Reference numerals: an acquisition module-31; an empowerment module-32; a correlation fusion module-33; a processing module-34.
Detailed Description
Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.
Example one
A multi-source heterogeneous data fusion method comprises the following steps:
s11: acquiring data to be fused;
s12: different weight operators are given to each field in the data to be fused;
s13: performing association fusion on the data to be fused according to a preset data association rule and a weight operator of each field in the data to be fused to obtain the data to be processed;
s14: and carrying out standardized processing on the data to be processed.
It should be noted that, acquiring data to be fused includes:
data to be fused are obtained from a plurality of data sources.
In this embodiment, the data to be fused is acquired as data, and the data to be fused is taken as natural human data as an example, data resources related to natural human needs to be screened, and multiple data sources are involved.
It should be noted that, assigning different weight operators to each field in the data to be fused includes:
dividing data to be fused into a plurality of types of fields, wherein each type of field comprises a plurality of subfields;
determining the level of each data source and determining the service attribution of each subfield;
and determining the weight operator of each sub-field under each type of field according to the level of each data source and the service attribution of each sub-field.
Taking the address class data field as an example, the address class field includes subfields such as a place of birth, a current address, a place of residence, and the like, and needs to be merged into a uniform address class field.
Since different levels of data source exist, for example, the accuracy of data at the township level may be lower than that at the city and county level, the weighting operators of the sub-fields under each type of field are determined according to the level of the data source, or the importance of the data source in the embodiment.
Different subfields correspond to different service attributions or professional attributions, and the data source and the service attribution belong to government service characteristics of data to be fused as data, so that in the embodiment, the weight operator of each subfield under each type of field is determined according to the level of each data source and the service attribution of each subfield.
Preferably, the method further comprises:
and determining the subjective factor influence value of each subfield, and calibrating the weight operator of each subfield according to the subjective factor influence value of each subfield.
It will be appreciated that in addition to taking into account the level of the data source and the business attribution of each sub-field, the determination of the weighting operator also needs to take into account the influence of subjective factors, such as the address class field being affected by time constraints, and the most recently updated address class sub-field data information being closer to the exact address data, so it needs to be given a higher weighting operator.
It should be noted that, performing association fusion on the data to be fused according to a preset data association rule and a weight operator of each field in the data to be fused includes:
calculating the accuracy value of each subfield in the data to be fused relative to the field type according to the weight operator of each subfield in the data to be fused;
and obtaining a fused uniform data value of each field as fused data according to the accuracy value of each subfield relative to the field type to which the subfield belongs.
The embodiment provides a data fusion function:
x j representing fused uniform data values, w, of various types of fields i Is the weight of subfield i, t ij Is the accuracy value of the subfield i relative to the belonging field class j.
In the embodiment, according to the data fusion function, the accuracy value of each subfield in the data to be fused relative to the field type to which the subfield belongs is calculated; obtaining the fused uniform data value of various fields as fused data according to the accuracy value of each subfield relative to the field category to which the subfield belongs
It should be noted that, performing association fusion on the data to be fused according to a preset data association rule and a weight operator of each field in the data to be fused, further includes:
determining a unique associated identifier in the fused data;
associating the data which has relevance with the unique association identifier in the fused data according to the same data structure; the data structure includes at least: library tables and data items;
establishing a data association model, wherein the data association model is used for constructing an association relation between data structures;
acquiring specific numerical information of the data to be fused, and verifying the data association model according to the specific numerical information of the data to be fused;
and when the verification is passed, the associated data is taken as the data to be processed.
Taking the fused data as the natural person data as an example, when the natural person data is associated, the data can be associated and fused by using the related certificate number, such as an identity card number, a military officer certificate, a passport and the like, as a unique association identifier. Specifically, according to a data association rule, an association relation is established between the same data structures such as a base table and the same base table, a data item and the same data item, a data association model is established, the data association model is verified according to the specific numerical information of the data to be fused by acquiring the specific numerical information of the data to be fused, when the verification is passed, the associated data is used as the data to be processed, and preferably, the finally formed associated data is issued in a data wide table mode.
Further, the method further comprises: and when the verification is passed, releasing the data association model.
In this embodiment, the verified data association model is published for use in communication.
Referring to fig. 2, the method further includes:
s21: and (4) carrying out cleaning and re-processing on the data to be processed.
In this embodiment, data inspection, null value processing, repeated data processing, record missing processing, data normalization, value domain normalization, and the like are performed on the data to be processed after the association fusion, that is, the data wide table. According to the attributes of the data and the related business attributes, operations such as 'dirty data' and 'empty data' cleaning, duplicate removal and the like are carried out on the data, data cleaning conversion rules and workflows are defined, cleaning verification can be carried out on data samples of a data source, and when the cleaning requirements are not met, the cleaning conversion rules, the workflows and system parameters need to be adjusted and improved.
It should be noted that, the normalizing process performed on the data to be processed includes:
and converting the fused data of various fields in the data to be processed into corresponding coded values in a corresponding data conversion mode.
In this embodiment, the fused multi-source data is subjected to the standardization operation of the corresponding dictionary, and the attribute values are converted into a consistent and uniform format. For example, in the associated natural human width table, the standardization of the address field requires that the fused data be returned to the corresponding code, such as the longitude and latitude code, through the spatial address conversion. Based on this, it is also possible to standardize fields such as marital status, national status, academic degree information, and the like.
It can be understood that the multi-source heterogeneous data fusion method in this embodiment includes: acquiring data to be fused, endowing different weight operators to each field in the data to be fused, performing association fusion on the data to be fused according to a preset data association rule and the weight operators of each field in the data to be fused to obtain data to be processed, and finally performing standardized processing on the data to be processed. In this embodiment, a plurality of factors are considered comprehensively, different weight operators are assigned to each field in the data to be fused, for example, a relatively higher weight operator is assigned to a field with high expected accuracy, a relatively lower weight operator is assigned to a field with low expected accuracy, and the degree of close dependence of the fused data on the data is higher during the association and fusion, so that the obtained fused data is more accurate and reliable.
Example two
A multi-source heterogeneous data fusion apparatus, comprising:
an obtaining module 31, configured to obtain data to be fused;
the weighting module 32 is configured to assign different weight operators to each field in the data to be fused;
the association fusion module 33 is configured to perform association fusion on the data to be fused according to a preset data association rule and a weight operator of each field in the data to be fused to obtain data to be processed;
and the processing module 34 is used for performing standardization processing on the data to be processed.
It can be understood that the multi-source heterogeneous data fusion device in the embodiment includes: the system comprises an acquisition module 31, an empowerment module 32, an association fusion module 33 and a processing module 34. In implementation, the obtaining module 31 obtains data to be fused, the weighting module 32 assigns different weight operators to fields in the data to be fused, the association fusion module 33 performs association fusion on the data to be fused according to a preset data association rule and the weight operators of the fields in the data to be fused to obtain data to be processed, and finally the processing module 34 performs standardized processing on the data to be processed. In the embodiment, various factors are comprehensively considered, different weight operators are assigned to each field in the data to be fused, for example, a relatively higher weight operator is assigned to a field with high expected accuracy, a relatively lower weight operator is assigned to a field with low expected accuracy, and the close dependence degree of the fused data on the data is higher during the association fusion, so that the obtained fused data is more accurate and reliable.
It should be noted that the processing module 34 is also used for performing a cleaning and re-processing on the data to be processed.
It should be noted that the obtaining module 31 is specifically configured to obtain the data to be fused from multiple data sources.
The weighting module 32 is specifically configured to divide the data to be fused into multiple types of fields, where each type of field includes multiple subfields; determining the level of each data source and determining the service attribution of each subfield; and determining the weight operator of each sub-field under each type of field according to the level of each data source and the service attribution of each sub-field.
The weighting module 32 is further configured to determine a subjective factor influence value of each sub-field, and calibrate a weight operator of each sub-field according to the subjective factor influence value of each sub-field.
The association fusion module 33 is specifically configured to calculate, according to the weight operator of each subfield in the data to be fused, an accuracy value of each subfield in the data to be fused relative to the field category to which the subfield belongs; and obtaining the fused uniform data value of each field as fused data according to the accuracy value of each subfield relative to the field category to which the subfield belongs.
The association fusion module 33 is further configured to determine a unique association identifier in the fused data; associating the data which has relevance with the unique association identifier in the fused data according to the same data structure; the data structure includes at least: library tables and data items; establishing a data association model, wherein the data association model is used for constructing an association relation between data structures; acquiring specific numerical information of the data to be fused, and verifying the data association model according to the specific numerical information of the data to be fused; and when the verification is passed, taking the associated data as the data to be processed.
And the processing module 34 converts the fused data of the various fields in the data to be processed into corresponding coded values in a corresponding data conversion mode.
Preferably, the multi-source heterogeneous data fusion device further comprises a model issuing module, which is used for issuing the data association model when the data association model passes the verification according to the specific numerical information of the data to be fused.
It is understood that the same or similar parts in the above embodiments may be mutually referred to, and the same or similar parts in other embodiments may be referred to for the content which is not described in detail in some embodiments.
It should be noted that, in the description of the present application, the terms "first", "second", etc. are used for descriptive purposes only and are not to be construed as indicating or implying relative importance. Further, in the description of the present application, the meaning of "a plurality" means at least two unless otherwise specified.
Any process or method descriptions in flow charts or otherwise described herein may be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or steps of the process, and the scope of the preferred embodiments of the present application includes other implementations in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those reasonably skilled in the art of the present application.
It should be understood that portions of the present application may be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the various steps or methods may be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one or combination of the following techniques, which are known in the art, may be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application specific integrated circuit having an appropriate combinational logic gate circuit, a Programmable Gate Array (PGA), a Field Programmable Gate Array (FPGA), or the like.
It will be understood by those skilled in the art that all or part of the steps carried by the method for implementing the above embodiments may be implemented by hardware related to instructions of a program, which may be stored in a computer readable storage medium, and when the program is executed, the program includes one or a combination of the steps of the method embodiments.
In addition, functional units in the embodiments of the present application may be integrated into one processing module, or each unit may exist alone physically, or two or more units are integrated into one module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode. The integrated module, if implemented in the form of a software functional module and sold or used as a stand-alone product, may also be stored in a computer readable storage medium.
The storage medium mentioned above may be a read-only memory, a magnetic or optical disk, etc.
In the description herein, reference to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the application. In this specification, the schematic representations of the terms used above do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
While embodiments of the present application have been shown and described above, it will be understood that the above embodiments are exemplary and should not be construed as limiting the present application and that changes, modifications, substitutions and alterations in the above embodiments may be made by those of ordinary skill in the art within the scope of the present application.
Claims (10)
1. A multi-source heterogeneous data fusion method is characterized by comprising the following steps:
acquiring data to be fused;
different weight operators are given to each field in the data to be fused;
performing association fusion on the data to be fused according to a preset data association rule and a weight operator of each field in the data to be fused to obtain data to be processed;
and carrying out standardization processing on the data to be processed.
2. The method according to claim 1, wherein the obtaining the data to be fused comprises:
and acquiring the data to be fused from a plurality of data sources.
3. The method according to claim 2, wherein the assigning different weight operators to the fields in the data to be fused comprises:
dividing the data to be fused into a plurality of types of fields, wherein each type of field comprises a plurality of subfields;
determining the level of each data source and determining the service attribution of each subfield;
and determining the weight operator of each sub-field under each type of field according to the level of each data source and the service attribution of each sub-field.
4. The method according to any one of claims 1-3, further comprising:
and determining the subjective factor influence value of each subfield, and calibrating the weight operator of each subfield according to the subjective factor influence value of each subfield.
5. The method according to claim 4, wherein the performing association fusion on the data to be fused according to a preset data association rule and a weight operator of each field in the data to be fused comprises:
calculating the accuracy value of each subfield in the data to be fused relative to the field category to which the subfield belongs according to the weight operator of each subfield in the data to be fused;
and obtaining the fused uniform data value of each field as fused data according to the accuracy value of each subfield relative to the field category to which the subfield belongs.
6. The method according to claim 5, wherein the performing association fusion on the data to be fused according to a preset data association rule and a weight operator of each field in the data to be fused comprises:
determining a unique association identifier in the fused data;
associating the data which has relevance with the unique association identifier in the fused data according to the same data structure; the data structure includes at least: library tables and data items;
establishing a data association model, wherein the data association model is used for constructing an association relation between data structures;
acquiring specific numerical information of the data to be fused, and verifying the data association model according to the specific numerical information of the data to be fused;
and when the verification is passed, using the associated data as the data to be processed.
7. The method of claim 5, further comprising: and when the verification is passed, issuing the data association model.
8. The method of claim 1, further comprising: and cleaning and reprocessing the data to be processed.
9. The method of claim 1, wherein the normalizing the data to be processed comprises:
and converting the fused data of various fields in the data to be processed into corresponding coded values in a corresponding data conversion mode.
10. A multi-source heterogeneous data fusion device is characterized by comprising:
the acquisition module is used for acquiring data to be fused;
the weighting module is used for giving different weight operators to each field in the data to be fused;
the association fusion module is used for performing association fusion on the data to be fused according to a preset data association rule and a weight operator of each field in the data to be fused to obtain data to be processed;
and the processing module is used for carrying out standardized processing on the data to be processed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211610209.0A CN115952150A (en) | 2022-12-14 | 2022-12-14 | Multi-source heterogeneous data fusion method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211610209.0A CN115952150A (en) | 2022-12-14 | 2022-12-14 | Multi-source heterogeneous data fusion method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115952150A true CN115952150A (en) | 2023-04-11 |
Family
ID=87281687
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211610209.0A Pending CN115952150A (en) | 2022-12-14 | 2022-12-14 | Multi-source heterogeneous data fusion method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115952150A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116089907A (en) * | 2023-04-13 | 2023-05-09 | 民航成都信息技术有限公司 | Fusion method and device of aviation multi-source data, electronic equipment and storage medium |
-
2022
- 2022-12-14 CN CN202211610209.0A patent/CN115952150A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116089907A (en) * | 2023-04-13 | 2023-05-09 | 民航成都信息技术有限公司 | Fusion method and device of aviation multi-source data, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107169768B (en) | Method and device for acquiring abnormal transaction data | |
| CN110554958B (en) | Graph database testing method, system, device and storage medium | |
| CN109919781A (en) | Case recognition methods, electronic device and computer readable storage medium are cheated by clique | |
| CN108595657B (en) | Data table classification mapping method and device of HIS (hardware-in-the-system) | |
| CN109947881B (en) | POI weight judging method and device, mobile terminal and computer readable storage medium | |
| KR101953190B1 (en) | A multidimensional recursive learning process and system used to discover complex dyadic or multiple counterparty relationships | |
| CN110109908B (en) | Analysis system and method for mining potential relationship of person based on social basic information | |
| JP2002032773A (en) | Device and method for processing map data | |
| CN115952150A (en) | Multi-source heterogeneous data fusion method and device | |
| CN112308727A (en) | Insurance claim settlement service processing method and device | |
| CN111767277A (en) | Data processing method and device | |
| CN113918705A (en) | Contribution auditing method and system with early warning and recommendation functions | |
| CN114049016A (en) | Index similarity judgment method, system, terminal device and computer storage medium | |
| CN116881430B (en) | Industrial chain identification method and device, electronic equipment and readable storage medium | |
| CN111858236B (en) | Knowledge graph monitoring method and device, computer equipment and storage medium | |
| Khan et al. | Analysis of Tree-Family Machine Learning Techniques for Risk Prediction in Software Requirements | |
| US11227288B1 (en) | Systems and methods for integration of disparate data feeds for unified data monitoring | |
| CN107679174A (en) | Construction method, device and the server of Knowledge Organization System | |
| CN111582313A (en) | Sample data generation method and device and electronic equipment | |
| CN115587333A (en) | Failure analysis fault point prediction method and system based on multi-classification model | |
| CN115599840A (en) | Complex service data management method and system | |
| CN114595216A (en) | Data verification method and device, storage medium and electronic equipment | |
| CN113962216A (en) | Text processing method and device, electronic equipment and readable storage medium | |
| CN113792114A (en) | Credible evaluation method and system for urban field knowledge graph | |
| CN112308466A (en) | Enterprise qualification auditing method and device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |