CN111858098A - Data exchange method based on mass data - Google Patents
Data exchange method based on mass data Download PDFInfo
- Publication number
- CN111858098A CN111858098A CN202010722793.3A CN202010722793A CN111858098A CN 111858098 A CN111858098 A CN 111858098A CN 202010722793 A CN202010722793 A CN 202010722793A CN 111858098 A CN111858098 A CN 111858098A
- Authority
- CN
- China
- Prior art keywords
- data
- slave node
- mirror image
- centralized
- message
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 20
- 238000004364 calculation method Methods 0.000 claims description 10
- 238000011156 evaluation Methods 0.000 claims description 7
- 230000008569 process Effects 0.000 claims description 5
- 230000004044 response Effects 0.000 claims description 4
- 230000009467 reduction Effects 0.000 claims description 3
- 230000005540 biological transmission Effects 0.000 abstract description 5
- 238000007405 data analysis Methods 0.000 abstract description 3
- 238000010606 normalization Methods 0.000 description 2
- 238000007781 pre-processing Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1448—Management of the data involved in backup or backup restore
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
- G06F2009/45562—Creating, deleting, cloning virtual machine instances
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
- G06F2009/45595—Network integration; Enabling network access in virtual machine instances
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data exchange method based on mass data.A centralized data center acquires the information of a data slave node which is least frequently used after receiving a message request; communicating with the data slave node which is used least frequently through a message queue, and sending a message and a corresponding image file to a subject queue where the message queue is located; the data slave node subscribes to the topics in the message queue; the data slave node automatically goes to a local mirror image warehouse to pull the needed algorithm mirror image; the pulled mirror image endows the data slave nodes with certain computing capacity, and the collected data is preprocessed; the data slave node sends the processing result to a queue corresponding to a receiving subject of the centralized data center; after obtaining the processing result from the receiving subject queue, the centralized data center records the processing result locally and then directly returns the processing result to the message requester. The data transmission pressure of the centralized data center is effectively solved, and the data transmission efficiency, the data analysis efficiency and the data safety are improved.
Description
Technical Field
The invention relates to a data exchange method, in particular to a data exchange method based on mass data, and belongs to the technical field of cloud computing.
Background
In the face of massive data, the problem to be considered first is how to improve the operation performance and processing efficiency of massive data. On the premise of mass data, as the data volume of slave nodes of cloud computing and centralized data centers increases, when each node transmits data to a cloud node or a data center, the physical distance is too far, so that great pressure is applied to a central service, the central service not only needs to receive the data transmitted from the slave nodes, but also needs to extract resources for computing for data analysis, so that data blockage is easily caused, and the problems of high service delay, network fluctuation, data loss, network attack and the like are caused.
Disclosure of Invention
The invention provides a data exchange method based on mass data, aiming at solving the problems of high service delay, network fluctuation, data loss, network attack and the like.
The technical scheme adopted by the invention is as follows:
the system comprises a centralized data center, a plurality of data slave nodes and a mirror image warehouse; the data slave node has a storage function, can store key metadata information, and can process data locally to quickly respond when a real-time response requirement exists.
A data exchange method based on mass data comprises the following steps:
s1: after receiving the message request, the centralized data center acquires the information of the data slave nodes which are least frequently used;
s2: the centralized data center communicates with the data slave nodes which are used least frequently through the message queues and sends messages and corresponding image files to the subject queues where the centralized data center is located;
s3: the data slave node subscribes to the topics in the message queue;
s4: the data slave node automatically goes to a local mirror image warehouse to pull the needed algorithm mirror image;
s5: the pulled mirror image endows the data slave nodes with certain computing capacity, and the collected data is preprocessed;
s6: the data slave node sends the processing result to a queue corresponding to a receiving subject of the centralized data center;
s7: after obtaining the processing result from the receiving subject queue, the centralized data center records the processing result locally and then directly returns the processing result to the message requester.
The step S5 includes the sub-steps of:
s501: providing a computing method for the data slave nodes in a mode of mirror image deployment of containers;
s502: the computing capacity of the data slave node is improved through the expansion and the reduction of the mirror image.
When a plurality of slave nodes need to run the coordination task with high reliability within a certain physical range, the step S2 further includes:
s21: hot-standby a plurality of container tasks on a plurality of data slave nodes;
s22: when one of the container tasks quits abnormally, other data are automatically elected and run from the container tasks of the nodes.
In some high-sensitivity tasks, the step S5 includes the sub-steps of:
s503: the data slave node obtains a specific early warning calculation function through a downloading algorithm mirror image;
s504: six evaluation indexes of the universally applicable data quality of the early warning calculation function are an accuracy index, an effectiveness index, a consistency index, a normalization index, a timeliness index and an integrity index, and the acquired data is evaluated according to the indexes;
s505: and judging whether the evaluation score meets a preset expected value, and if not, issuing the acquisition task to a user group through a message queue for early warning.
The invention achieves the following beneficial effects:
1. the invention provides algorithm support for the slave nodes by issuing the customized algorithm mirror image to the data slave node deployment container; that is, the acquisition node can acquire the calculation method through the container, preprocess the acquired data, and reduce the data volume transmitted to the data center.
2. The invention provides a multi-containerization hot standby function for the acquisition task and ensures the normal operation of the acquisition task.
3. The data exchange method based on mass data effectively solves the data transmission pressure of a centralized data center, and improves the data transmission and data analysis efficiency and data security.
Drawings
Other features and advantages of the invention will be apparent from the following description of the preferred embodiments of the invention, taken in conjunction with the accompanying drawings and from the claims. Individual features of the different embodiments shown in the figures may be combined in any desired manner in this case without going beyond the scope of the invention. In the drawings:
fig. 1 is a signal flow diagram of the present invention.
The implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is further described below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
Example (b):
as shown in fig. 1, in this embodiment, a data exchange method based on mass data includes a centralized data center, a plurality of data slave nodes, and a mirror repository; the data slave node has a storage function, can store key metadata information, and can process data locally to quickly respond when a real-time response requirement exists.
After receiving the message request, the centralized data center acquires the information of the data slave nodes which are least frequently used; the centralized data center communicates with the data slave nodes which are used least frequently through the message queues and sends messages and corresponding image files to the subject queues where the centralized data center is located; the data slave node subscribes to the topics in the message queue; the data slave node automatically goes to a local mirror image warehouse to pull the needed algorithm mirror image; the pulled mirror image endows the data slave node with certain computing capacity, preprocessing the acquired data, and sending the processing result to a queue corresponding to a receiving subject of a centralized data center from the data slave node; after obtaining the processing result from the receiving subject queue, the centralized data center records the processing result locally and then directly returns the processing result to the message requester.
It should be noted that, in order to enable the slave node to have the capability of data preprocessing, the centralized data center configures different images for different data slave nodes in a message queue theme manner, and may configure different themes to select different images to be sent to the data slave nodes. The expansion of the computing capacity of the slave nodes can be achieved through continuous expansion and reduction of the images. Therefore, the slave nodes can perform more local calculations, process local data, reduce data transmitted to the central service, reduce the pressure of the central service and accelerate the service response speed. And the private data is locally processed, so that the private data is prevented from being attacked and leaked during network transmission, and the data security is improved.
When the range data of the real position is subjected to cooperative processing, in order to ensure the synchronization and high reliability of the key data of each data slave node, the task can be made into a mirror image and issued to each data slave node in the area, and a plurality of container tasks are hot-prepared on a plurality of data slave nodes. When one of the container tasks quits abnormally, the container tasks on other nodes can automatically perform election operation. Therefore, the synchronization of key data and the normal execution of tasks are ensured.
In some high-sensitivity tasks, a data slave node obtains a specific early warning calculation function through a downloading algorithm mirror image; six evaluation indexes of the universally applicable data quality of the early warning calculation function are an accuracy index, an effectiveness index, a consistency index, a normalization index, a timeliness index and an integrity index, and the acquired data is evaluated according to the indexes; and finally obtaining the evaluation score of each index, judging whether the acquired data meet the set expected value according to the evaluation score, and if not, issuing the acquisition task to a user group through a message queue for early warning.
The above description is only a preferred embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications of equivalent structures and equivalent processes, which are made by using the contents of the present specification and the accompanying drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.
Claims (5)
1. A data exchange method based on mass data comprises a centralized data center, a plurality of data slave nodes and a mirror image warehouse; the method is characterized by comprising the following steps:
s1: after receiving the message request, the centralized data center acquires the information of the data slave nodes which are least frequently used;
s2: the centralized data center communicates with the data slave nodes which are used least frequently through the message queues and sends messages and corresponding image files to the subject queues where the centralized data center is located;
s3: the data slave node subscribes to the topics in the message queue;
s4: the data slave node automatically goes to a local mirror image warehouse to pull the needed algorithm mirror image;
s5: the pulled mirror image endows the data slave nodes with certain computing capacity, and the collected data is preprocessed;
s6: the data slave node sends the processing result to a queue corresponding to a receiving subject of the centralized data center;
s7: after obtaining the processing result from the receiving subject queue, the centralized data center records the processing result locally and then directly returns the processing result to the message requester.
2. A data exchange method based on mass data according to claim 1, wherein said step S5 includes the sub-steps of:
s501: providing a computing method for the data slave nodes in a mode of mirror image deployment of containers;
s502: the computing capacity of the data slave node is improved through the expansion and the reduction of the mirror image.
3. A data exchange method based on mass data according to claim 1, wherein in some high-sensitivity tasks, said step S5 includes the sub-steps of:
s503: the data slave node obtains a specific early warning calculation function through a downloading algorithm mirror image;
s504: evaluating the acquired data according to the evaluation index of the early warning calculation function;
s505: and judging whether the evaluation score meets a preset expected value, and if not, issuing the acquisition task to a user group through a message queue for early warning.
4. The mass data-based data exchange method according to claim 1, wherein when a plurality of slave nodes need to run the coordination task with high reliability within a certain physical range, step S2 is followed by further comprising:
s21: hot-standby a plurality of container tasks on a plurality of data slave nodes;
s22: when one of the container tasks quits abnormally, other data are automatically elected and run from the container tasks of the nodes.
5. The mass data-based data exchange method as claimed in claim 1, wherein the data slave node has a storage function, and can store key metadata information, and when there is a real-time response requirement, the data slave node can process data locally so as to respond quickly.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010722793.3A CN111858098B (en) | 2020-07-24 | 2020-07-24 | Data exchange method based on mass data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010722793.3A CN111858098B (en) | 2020-07-24 | 2020-07-24 | Data exchange method based on mass data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111858098A true CN111858098A (en) | 2020-10-30 |
| CN111858098B CN111858098B (en) | 2023-11-17 |
Family
ID=72950183
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010722793.3A Active CN111858098B (en) | 2020-07-24 | 2020-07-24 | Data exchange method based on mass data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111858098B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112433814A (en) * | 2020-11-24 | 2021-03-02 | 中信银行股份有限公司 | Local mirror image warehouse management method, device and system |
Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5787485A (en) * | 1996-09-17 | 1998-07-28 | Marathon Technologies Corporation | Producing a mirrored copy using reference labels |
| CN102521044A (en) * | 2011-12-30 | 2012-06-27 | 北京拓明科技有限公司 | Distributed task scheduling method and system based on messaging middleware |
| CN103257970A (en) * | 2012-02-17 | 2013-08-21 | 百度在线网络技术（北京）有限公司 | Method and device for testing primary node of HDFS (Hadoop Distributed File System) |
| JP2013214184A (en) * | 2012-04-02 | 2013-10-17 | Nec System Technologies Ltd | Mirroring system, node, mirroring method and program |
| CN105245373A (en) * | 2015-10-12 | 2016-01-13 | 天津市普迅电力信息技术有限公司 | Construction and operation method of container cloud platform system |
| CN105472024A (en) * | 2015-12-28 | 2016-04-06 | 北京赛思信安技术股份有限公司 | Cross-region data synchronizing method based on message pushing mode |
| CN106126349A (en) * | 2016-06-14 | 2016-11-16 | 浪潮电子信息产业股份有限公司 | A kind of remote invocation method, message-oriented middleware cluster and system |
| CN107818130A (en) * | 2017-09-15 | 2018-03-20 | 深圳市电陶思创科技有限公司 | The method for building up and system of a kind of search engine |
| CN108763026A (en) * | 2018-05-31 | 2018-11-06 | 郑州云海信息技术有限公司 | A kind of disk monitoring alarm method, system, equipment and computer storage media |
| CN108829787A (en) * | 2018-05-31 | 2018-11-16 | 郑州云海信息技术有限公司 | A kind of meta-data distribution formula system |
| CN109391664A (en) * | 2017-08-11 | 2019-02-26 | 中国电信股份有限公司 | System and method for the deployment of more cluster containers |
| CN109801479A (en) * | 2018-12-12 | 2019-05-24 | 西安电子科技大学 | A kind of monitoring of semiconductor processes and early warning system and its control method |
| CN110647580A (en) * | 2019-09-05 | 2020-01-03 | 南京邮电大学 | Distributed container cluster mirror image management main node, slave node, system and method |
| CN110889946A (en) * | 2018-12-28 | 2020-03-17 | 研祥智能科技股份有限公司 | Security monitoring and guiding system |
| CN110943872A (en) * | 2019-12-31 | 2020-03-31 | 上海擎感智能科技有限公司 | Data acquisition service configuration/performance analysis method/system, medium, and server |
-
2020
- 2020-07-24 CN CN202010722793.3A patent/CN111858098B/en active Active
Patent Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5787485A (en) * | 1996-09-17 | 1998-07-28 | Marathon Technologies Corporation | Producing a mirrored copy using reference labels |
| CN102521044A (en) * | 2011-12-30 | 2012-06-27 | 北京拓明科技有限公司 | Distributed task scheduling method and system based on messaging middleware |
| CN103257970A (en) * | 2012-02-17 | 2013-08-21 | 百度在线网络技术（北京）有限公司 | Method and device for testing primary node of HDFS (Hadoop Distributed File System) |
| JP2013214184A (en) * | 2012-04-02 | 2013-10-17 | Nec System Technologies Ltd | Mirroring system, node, mirroring method and program |
| CN105245373A (en) * | 2015-10-12 | 2016-01-13 | 天津市普迅电力信息技术有限公司 | Construction and operation method of container cloud platform system |
| CN105472024A (en) * | 2015-12-28 | 2016-04-06 | 北京赛思信安技术股份有限公司 | Cross-region data synchronizing method based on message pushing mode |
| CN106126349A (en) * | 2016-06-14 | 2016-11-16 | 浪潮电子信息产业股份有限公司 | A kind of remote invocation method, message-oriented middleware cluster and system |
| CN109391664A (en) * | 2017-08-11 | 2019-02-26 | 中国电信股份有限公司 | System and method for the deployment of more cluster containers |
| CN107818130A (en) * | 2017-09-15 | 2018-03-20 | 深圳市电陶思创科技有限公司 | The method for building up and system of a kind of search engine |
| CN108763026A (en) * | 2018-05-31 | 2018-11-06 | 郑州云海信息技术有限公司 | A kind of disk monitoring alarm method, system, equipment and computer storage media |
| CN108829787A (en) * | 2018-05-31 | 2018-11-16 | 郑州云海信息技术有限公司 | A kind of meta-data distribution formula system |
| CN109801479A (en) * | 2018-12-12 | 2019-05-24 | 西安电子科技大学 | A kind of monitoring of semiconductor processes and early warning system and its control method |
| CN110889946A (en) * | 2018-12-28 | 2020-03-17 | 研祥智能科技股份有限公司 | Security monitoring and guiding system |
| CN110647580A (en) * | 2019-09-05 | 2020-01-03 | 南京邮电大学 | Distributed container cluster mirror image management main node, slave node, system and method |
| CN110943872A (en) * | 2019-12-31 | 2020-03-31 | 上海擎感智能科技有限公司 | Data acquisition service configuration/performance analysis method/system, medium, and server |
Non-Patent Citations (1)
| Title |
|---|
| 何荣祥: ""面向移动用户的高可用服务器架构设计"", 《中国优秀硕士学位论文全文数据库-信息科技辑》 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112433814A (en) * | 2020-11-24 | 2021-03-02 | 中信银行股份有限公司 | Local mirror image warehouse management method, device and system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111858098B (en) | 2023-11-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10972565B2 (en) | Push notification delivery system with feedback analysis | |
| CN106210021A (en) | The method for real-time monitoring of financial application system online business and supervising device | |
| CN102630315B (en) | Method and system for processing data for preventing deadlock | |
| CN105511944A (en) | Anomaly detection method of internal virtual machine of cloud system | |
| US20230219605A1 (en) | Cloud platform-based comprehensive line network monitoring method and system | |
| CN109740038A (en) | Network data distributed parallel computing environment and method | |
| US11831746B2 (en) | Time consistency synchronization method for distributed simulation | |
| CN107423145A (en) | A kind of method and apparatus for avoiding information drop-out | |
| CN113722127A (en) | Efficient lightweight easy-to-use distributed network message middleware | |
| CN107579858A (en) | The alarm method and device of cloud main frame, communication system | |
| CN107509119A (en) | A kind of monitoring alarm method and device | |
| CN110620820A (en) | Ubiquitous power Internet of things intelligent management system | |
| CN111858098A (en) | Data exchange method based on mass data | |
| Dunne et al. | A comparison of data streaming frameworks for anomaly detection in embedded systems | |
| CN111460038A (en) | Quasi-real-time data synchronization method and device | |
| CN105681404A (en) | Metadata node management method and device of distributed cache system | |
| CN111368166A (en) | Resource search method, resource search apparatus, and computer-readable storage medium | |
| CN107730380B (en) | Method, system and server for processing joint account | |
| CN112950349B (en) | Method and system for processing base distributed system flushing quadrature easy time sequence exception | |
| CN113449628A (en) | Image processing system, image processing method, image processing apparatus, storage medium, and computer program product | |
| CN114021052A (en) | Inference service method, equipment and system | |
| Lu et al. | A fault tolerant election-based deadlock detection algorithm in distributed systems | |
| CN117240903B (en) | Internet of things offline message dynamic management configuration system | |
| JP6501007B1 (en) | OUTPUT MANAGEMENT DEVICE, OUTPUT MANAGEMENT METHOD, AND PROGRAM | |
| US20210227027A1 (en) | System and a method implementing a directed acyclic graph (dag) consensus algorithm via a gossip protocol |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |