CN112711626A - Unified convergence method and system for multi-source heterogeneous data - Google Patents
Unified convergence method and system for multi-source heterogeneous data Download PDFInfo
- Publication number
- CN112711626A CN112711626A CN202011591495.1A CN202011591495A CN112711626A CN 112711626 A CN112711626 A CN 112711626A CN 202011591495 A CN202011591495 A CN 202011591495A CN 112711626 A CN112711626 A CN 112711626A
- Authority
- CN
- China
- Prior art keywords
- data
- unit
- module
- fusion
- client
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/252—Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2465—Query processing support for facilitating data mining operations in structured databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2468—Fuzzy queries
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Probability & Statistics with Applications (AREA)
- Computational Linguistics (AREA)
- Automation & Control Theory (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
A unified convergence method and a system for multi-source heterogeneous data relate to the field of multi-source heterogeneous data processing, and the system comprises a client, a cloud server and a memory; the client is provided with a data acquisition module; the cloud server is provided with a data processing module and a data fusion module. According to the method, the data acquisition module is used for acquiring multi-source data information, the data processing module is used for grouping data, eliminating isomerism and manufacturing a data pool, the data decision model is used for detecting the fusion degree of the data in the data pool and adjusting the data in the data pool until the estimated fusion degree is reached, and then fusion is performed, so that the unified convergence of multi-source heterogeneous data is realized, the converged fusion degree is high, invalid data are few, the classified storage of a large amount of data is facilitated, and the efficiency of subsequent query is effectively improved.
Description
Technical Field
The invention relates to the field of multi-source heterogeneous data processing, in particular to a method and a system for uniformly converging multi-source heterogeneous data.
Background
At present, with the high-speed development of the internet of things technology, the number and the types of various terminals and basic acquisition equipment are continuously increased, a large amount of data can be generated at every moment, the data are various in types and sources, namely, the data are multi-source heterogeneous data, the multi-source heterogeneous data are large in number, the data fusion degree is low due to the fact that the data are processed by a traditional unified convergence method, and the subsequent data use is not facilitated.
Disclosure of Invention
Objects of the invention
In order to solve the technical problems in the background art, the invention provides a method and a system for uniformly converging multi-source heterogeneous data. According to the method, the data acquisition module is used for acquiring multi-source data information, the data processing module is used for grouping data, eliminating isomerism and manufacturing a data pool, the data decision model is used for detecting the fusion degree of the data in the data pool and adjusting the data in the data pool until the estimated fusion degree is reached, and then fusion is performed, so that the unified convergence of multi-source heterogeneous data is realized, the converged fusion degree is high, invalid data are few, the classified storage of a large amount of data is facilitated, and the efficiency of subsequent query is effectively improved.
(II) technical scheme
In order to solve the problems, the invention provides a multi-source heterogeneous data uniform aggregation system which comprises a client, a cloud server and a memory; the client, the cloud server and the memory are connected through a network; the client is provided with a data acquisition module; the cloud server is provided with a data processing module and a data fusion module; the data acquisition module acquires multi-source data information through a client, and comprises a character acquisition unit, a picture acquisition unit and an audio-video acquisition unit; the data processing module is connected with the data acquisition module and comprises a data classification unit, a data processing unit and a data matching unit, the data processing module groups data in the data acquisition module and eliminates isomerism, corresponding data characteristic groups are respectively extracted according to categories, and then the data characteristic groups are matched with the existing data characteristic groups to find out data characteristics reaching set relevancy; the data fusion module comprises a data aggregation unit, a data decision model, a data adjustment unit and a data fusion unit, the data fusion module is used for manufacturing a data pool by aggregating the data characteristics of new data and original data, the data decision model is used for checking the fusion degree of the data in the data pool and adjusting the data in the data pool at the same time until the estimated fusion degree is reached, then the fusion is carried out, and the fusion result is returned to the memory.
Preferably, the system further comprises a monitoring module; the monitoring module monitors and records all data behaviors in the system, and stores the data behaviors in a memory, wherein the memory comprises a monitoring unit, a recording unit and a data transmission unit.
Preferably, the memory includes a data storage unit, a backup unit, an encryption unit, and an update unit.
Preferably, the client is provided with a query module; the query module comprises an identity verification unit, a query application unit and a result feedback unit.
Preferably, the cloud server is provided with a query processing module; the query processing module is used for data circulation of the query module and the memory.
Preferably, the data decision model adopts an OWA weight vector calculation method to process fuzzy data in the data pool; the treatment method comprises the following steps:
setting: f: rn→ R, there is an n-dimensional weighting vector w associated with F ═ w (w)1,w2,...,wn)，wi∈[0，1]I is not less than 1 and not more than n, and
such that:
wherein: biIs aiF is called an n-dimensional OWA operator if the ith element is the largest; OWA weight vector w ═ w1,w2,...,wn) Is determined by the following formula:
wif (i/n) -f ((i-1)/n), wherein: 1, 2.. n, f is a fuzzy semantic quantizer defined as:
wherein: x, a, b ∈ [0, 1 ]]。
The invention also provides a unified convergence method for multi-source heterogeneous data, which comprises the following steps:
s1, inputting all multi-source heterogeneous data by a user through a client;
s2, classifying multi-source heterogeneous data, eliminating heterogeneity, extracting corresponding data features, and forming a new data feature group according to categories;
s3, carrying out correlation degree query on the new data characteristic group and the original data characteristic group, and finding out the original data characteristic group with the correlation degree reaching a set value;
s4, placing the new data characteristic cluster and the original data characteristic cluster into a data pool;
s5, the data decision model checks and adjusts the data in the data pool until the estimated fusion degree is reached, and then fusion is carried out;
s6, returning the fused data to a memory;
s7, when the client inquires the information, the client is firstly authenticated, and after the authentication is passed, the inquiry processing module carries out butt joint on the client and the memory;
and S8, the client acquires the query information.
Preferably, in S5, the data decision model eliminates overlapping data and error data in the data pool, and quantifies the fuzzy data.
The technical scheme of the invention has the following beneficial technical effects:
the method comprises the steps of collecting multi-source data information through a data collection module, wherein the collected types are comprehensive and various, then grouping the data through a data processing module, eliminating isomerism, respectively extracting corresponding data characteristic groups according to categories, matching the data characteristic groups with existing data characteristic groups, finding out data characteristics reaching set relevance, then converging new data and original data through a data fusion module to manufacture a data pool, detecting the fusion degree of the data in the data pool through a data decision model, adjusting the data in the data pool until the estimated fusion degree is reached, and then fusing, so that unified convergence of multi-source heterogeneous data is realized, the converged fusion degree is high, invalid data are few, classified storage of a large amount of data is facilitated, and the efficiency of subsequent query is effectively improved.
Drawings
Fig. 1 is a block diagram of a unified convergence system for multi-source heterogeneous data according to the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be described in further detail with reference to the accompanying drawings in conjunction with the following detailed description. It should be understood that the description is intended to be exemplary only, and is not intended to limit the scope of the present invention. Moreover, in the following description, descriptions of well-known structures and techniques are omitted so as to not unnecessarily obscure the concepts of the present invention.
Example 1
As shown in fig. 1, the system for uniformly converging multi-source heterogeneous data provided by the present invention includes a client, a cloud server, and a memory; the client, the cloud server and the memory are connected through a network; the client is provided with a data acquisition module; the cloud server is provided with a data processing module and a data fusion module; the data acquisition module acquires multi-source data information through a client, and comprises a character acquisition unit, a picture acquisition unit and an audio-video acquisition unit; the data processing module is connected with the data acquisition module and comprises a data classification unit, a data processing unit and a data matching unit, the data processing module groups data in the data acquisition module and eliminates isomerism, corresponding data characteristic groups are respectively extracted according to categories, and then the data characteristic groups are matched with the existing data characteristic groups to find out data characteristics reaching set relevancy; the data fusion module comprises a data aggregation unit, a data decision model, a data adjustment unit and a data fusion unit, the data fusion module is used for manufacturing a data pool by aggregating the data characteristics of new data and original data, the data decision model is used for checking the fusion degree of the data in the data pool and adjusting the data in the data pool at the same time until the estimated fusion degree is reached, then the fusion is carried out, and the fusion result is returned to the memory.
In an optional embodiment, the system further comprises a monitoring module; the monitoring module monitors and records all data behaviors in the system, and stores the data behaviors in a memory, wherein the memory comprises a monitoring unit, a recording unit and a data transmission unit.
In an alternative embodiment, the memory includes a data storage unit, a backup unit, an encryption unit, and an update unit.
In an optional embodiment, the client is provided with a query module; the query module comprises an identity verification unit, a query application unit and a result feedback unit.
In an optional embodiment, the cloud server is provided with a query processing module; the query processing module is used for data circulation of the query module and the memory.
In an optional embodiment, the data decision model processes the fuzzy data in the data pool by adopting an OWA weight vector calculation method; the treatment method comprises the following steps:
setting: f: rn→ R, there is an n-dimensional weighting vector w associated with F ═ w (w)1,w2,...,wn)，wi∈[0，1]I is not less than 1 and not more than n, and
such that:
wherein: biIs aiF is called an n-dimensional OWA operator if the ith element is the largest; OWA weight vector w ═ w1,w2,...,wn) Is determined by the following formula:
wif (i/n) -f ((i-1)/n), wherein: 1, 2.. n, f is a fuzzy semantic quantizer defined as:
wherein: x, a, b ∈ [0, 1 ]]。
Example 2
The invention also provides a unified convergence method for multi-source heterogeneous data, which comprises the following steps:
s1, inputting all multi-source heterogeneous data by a user through a client;
s2, classifying multi-source heterogeneous data, eliminating heterogeneity, extracting corresponding data features, and forming a new data feature group according to categories;
s3, carrying out correlation degree query on the new data characteristic group and the original data characteristic group, and finding out the original data characteristic group with the correlation degree reaching a set value;
s4, placing the new data characteristic cluster and the original data characteristic cluster into a data pool;
s5, the data decision model checks and adjusts the data in the data pool until the estimated fusion degree is reached, and then fusion is carried out;
s6, returning the fused data to a memory;
s7, when the client inquires the information, the client is firstly authenticated, and after the authentication is passed, the inquiry processing module carries out butt joint on the client and the memory;
and S8, the client acquires the query information.
In an alternative embodiment, in S5, the data decision model eliminates overlapping data and error data in the data pool, and quantifies the fuzzy data.
The method comprises the steps of collecting multi-source data information through a data collection module, wherein the collected types are comprehensive and various, then grouping the data through a data processing module, eliminating isomerism, respectively extracting corresponding data characteristic groups according to categories, matching the data characteristic groups with existing data characteristic groups, finding out data characteristics reaching set relevance, then converging new data and original data through a data fusion module to manufacture a data pool, detecting the fusion degree of the data in the data pool through a data decision model, adjusting the data in the data pool until the estimated fusion degree is reached, and then fusing, so that unified convergence of multi-source heterogeneous data is realized, the converged fusion degree is high, invalid data are few, classified storage of a large amount of data is facilitated, and the efficiency of subsequent query is effectively improved.
It is to be understood that the above-described embodiments of the present invention are merely illustrative of or explaining the principles of the invention and are not to be construed as limiting the invention. Therefore, any modification, equivalent replacement, improvement and the like made without departing from the spirit and scope of the present invention should be included in the protection scope of the present invention. Further, it is intended that the appended claims cover all such variations and modifications as fall within the scope and boundaries of the appended claims or the equivalents of such scope and boundaries.
Claims (8)
1. A multi-source heterogeneous data uniform aggregation system comprises a client, a cloud server and a memory; the client, the cloud server and the memory are connected through a network; the system is characterized in that a data acquisition module is arranged on the client; the cloud server is provided with a data processing module and a data fusion module;
the data acquisition module acquires multi-source data information through a client, and comprises a character acquisition unit, a picture acquisition unit and an audio-video acquisition unit; the data processing module is connected with the data acquisition module and comprises a data classification unit, a data processing unit and a data matching unit, the data processing module groups data in the data acquisition module and eliminates isomerism, corresponding data characteristic groups are respectively extracted according to categories, and then the data characteristic groups are matched with the existing data characteristic groups to find out data characteristics reaching set relevancy; the data fusion module comprises a data aggregation unit, a data decision model, a data adjustment unit and a data fusion unit, the data fusion module is used for manufacturing a data pool by aggregating the data characteristics of new data and original data, the data decision model is used for checking the fusion degree of the data in the data pool and adjusting the data in the data pool at the same time until the estimated fusion degree is reached, then the fusion is carried out, and the fusion result is returned to the memory.
2. The system for uniformly converging multi-source heterogeneous data according to claim 1, further comprising a monitoring module; the monitoring module monitors and records all data behaviors in the system, and stores the data behaviors in a memory, wherein the memory comprises a monitoring unit, a recording unit and a data transmission unit.
3. The system for uniform convergence of multi-source heterogeneous data according to claim 1, wherein the memory comprises a data storage unit, a backup unit, an encryption unit and an update unit.
4. The system for uniformly converging multi-source heterogeneous data according to claim 1, wherein a query module is arranged on a client; the query module comprises an identity verification unit, a query application unit and a result feedback unit.
5. The system for uniformly converging multi-source heterogeneous data according to claim 4, wherein a query processing module is arranged on the cloud server; the query processing module is used for data circulation of the query module and the memory.
6. The system for uniformly converging multi-source heterogeneous data according to claim 1, wherein the data decision model adopts an OWA weight vector calculation method to process fuzzy data in the data pool; the treatment method comprises the following steps:
setting: f: rn→ R, there is an n-dimensional weighting vector w associated with F ═ w (w)1,w2,...,wn)，wi∈[0，1]I is not less than 1 and not more than n, and
such that:
wherein: biIs aiTo middleif the i maximum elements are, F is called an n-dimensional OWA operator; OWA weight vector w ═ w1,w2,...,wn) Is determined by the following formula:
wif (i/n) -f ((i-1)/n), wherein: 1, 2.. n, f is a fuzzy semantic quantizer defined as:
wherein: x, a, b ∈ [0, 1 ]]。
7. A method for unified convergence of multi-source heterogeneous data, comprising the system of any one of claims 1 to 6, wherein the method comprises the following steps:
s1, inputting all multi-source heterogeneous data by a user through a client;
s2, classifying multi-source heterogeneous data, eliminating heterogeneity, extracting corresponding data features, and forming a new data feature group according to categories;
s3, carrying out correlation degree query on the new data characteristic group and the original data characteristic group, and finding out the original data characteristic group with the correlation degree reaching a set value;
s4, placing the new data characteristic cluster and the original data characteristic cluster into a data pool;
s5, the data decision model checks and adjusts the data in the data pool until the estimated fusion degree is reached, and then fusion is carried out;
s6, returning the fused data to a memory;
s7, when the client inquires the information, the client is firstly authenticated, and after the authentication is passed, the inquiry processing module carries out butt joint on the client and the memory;
and S8, the client acquires the query information.
8. The method for uniformly converging multi-source heterogeneous data according to claim 7, wherein in S5, the data decision model eliminates overlapped data and error data in the data pool, and quantizes the fuzzy data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011591495.1A CN112711626A (en) | 2020-12-29 | 2020-12-29 | Unified convergence method and system for multi-source heterogeneous data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011591495.1A CN112711626A (en) | 2020-12-29 | 2020-12-29 | Unified convergence method and system for multi-source heterogeneous data |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112711626A true CN112711626A (en) | 2021-04-27 |
Family
ID=75546204
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011591495.1A Pending CN112711626A (en) | 2020-12-29 | 2020-12-29 | Unified convergence method and system for multi-source heterogeneous data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112711626A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113110351A (en) * | 2021-04-28 | 2021-07-13 | 广东省科学院智能制造研究所 | Industrial production field heterogeneous state data acquisition system and method |
| CN113626511A (en) * | 2021-08-12 | 2021-11-09 | 山东勤成健康科技股份有限公司 | Heterogeneous database fusion access system |
| CN116132548A (en) * | 2023-02-06 | 2023-05-16 | 北京思维实创科技有限公司 | Data acquisition method and system for distributed multi-source heterogeneous data |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109558966A (en) * | 2018-10-28 | 2019-04-02 | 西南电子技术研究所（中国电子科技集团公司第十研究所） | Intelligence sentences the processing system that card predicted events occur |
| US20190205477A1 (en) * | 2017-12-29 | 2019-07-04 | Guangdong Oppo Mobile Telecommunications Corp., Ltd. | Method for Processing Fusion Data and Information Recommendation System |
| CN109981772A (en) * | 2019-03-22 | 2019-07-05 | 西安电子科技大学 | A kind of multiple domain data share exchange platform architecture based on block chain |
| CN111680082A (en) * | 2020-04-30 | 2020-09-18 | 四川弘智远大科技有限公司 | Government financial data acquisition system and data acquisition method based on data integration |
| CN111836285A (en) * | 2020-07-10 | 2020-10-27 | 辽宁邮电规划设计院有限公司 | Device and method for evaluating 5G network structure based on 4G network OMC-R and MDT data |
| CN112037920A (en) * | 2020-08-31 | 2020-12-04 | 康键信息技术（深圳）有限公司 | Medical knowledge map construction method, device, equipment and storage medium |
-
2020
- 2020-12-29 CN CN202011591495.1A patent/CN112711626A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20190205477A1 (en) * | 2017-12-29 | 2019-07-04 | Guangdong Oppo Mobile Telecommunications Corp., Ltd. | Method for Processing Fusion Data and Information Recommendation System |
| CN109558966A (en) * | 2018-10-28 | 2019-04-02 | 西南电子技术研究所（中国电子科技集团公司第十研究所） | Intelligence sentences the processing system that card predicted events occur |
| CN109981772A (en) * | 2019-03-22 | 2019-07-05 | 西安电子科技大学 | A kind of multiple domain data share exchange platform architecture based on block chain |
| CN111680082A (en) * | 2020-04-30 | 2020-09-18 | 四川弘智远大科技有限公司 | Government financial data acquisition system and data acquisition method based on data integration |
| CN111836285A (en) * | 2020-07-10 | 2020-10-27 | 辽宁邮电规划设计院有限公司 | Device and method for evaluating 5G network structure based on 4G network OMC-R and MDT data |
| CN112037920A (en) * | 2020-08-31 | 2020-12-04 | 康键信息技术（深圳）有限公司 | Medical knowledge map construction method, device, equipment and storage medium |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113110351A (en) * | 2021-04-28 | 2021-07-13 | 广东省科学院智能制造研究所 | Industrial production field heterogeneous state data acquisition system and method |
| CN113626511A (en) * | 2021-08-12 | 2021-11-09 | 山东勤成健康科技股份有限公司 | Heterogeneous database fusion access system |
| CN116132548A (en) * | 2023-02-06 | 2023-05-16 | 北京思维实创科技有限公司 | Data acquisition method and system for distributed multi-source heterogeneous data |
| CN116132548B (en) * | 2023-02-06 | 2023-08-15 | 北京思维实创科技有限公司 | Data acquisition method and system for distributed multi-source heterogeneous data |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112711626A (en) | Unified convergence method and system for multi-source heterogeneous data | |
| CN108664480B (en) | Multi-data-source user information integration method and device | |
| CN105550222B (en) | A kind of images serve system and method based on distributed storage | |
| Shang et al. | Fedic: Federated learning on non-iid and long-tailed data via calibrated distillation | |
| US20200394448A1 (en) | Methods for more effectively moderating one or more images and devices thereof | |
| CN113949577A (en) | Data attack analysis method applied to cloud service and server | |
| US11843622B1 (en) | Providing machine learning models for classifying domain names for malware detection | |
| CN109829320B (en) | Information processing method and device | |
| Xu et al. | Redundancy avoidance for big data in data centers: A conventional neural network approach | |
| EP3716547A1 (en) | Data stream recognition method and apparatus | |
| CN110990498A (en) | Data fusion method based on FCM algorithm | |
| CN108319672A (en) | Mobile terminal malicious information filtering method and system based on cloud computing | |
| CN113364787A (en) | Botnet flow detection method based on parallel neural network | |
| CN111368867A (en) | Archive classification method and system and computer readable storage medium | |
| Liang et al. | Tackling long-tailed distribution issue in graph neural networks via normalization | |
| US11477161B1 (en) | Systems and methods for detecting DNS communications through time-to-live analyses | |
| CN104090885A (en) | Multi-view video object retrieval system based on local copy detection and method | |
| Zhuo et al. | Fedns: Improving federated learning for collaborative image classification on mobile clients | |
| CN112468498A (en) | Cross-mode polymerization method for multi-source heterogeneous safety monitoring data of power distribution terminal | |
| CN110290188B (en) | HTTPS (hypertext transfer protocol secure) stream service online identification method suitable for large-scale network environment | |
| Avrachenkov et al. | Change rate estimation and optimal freshness in web page crawling | |
| TW201435627A (en) | System and method for optimizing search results | |
| CN114925286B (en) | Public opinion data processing method and device | |
| CN113762040B (en) | Video identification method, device, storage medium and computer equipment | |
| Liu et al. | Optimizing federated unsupervised person re-identification via camera-aware clustering |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |