CN114006883B - Cross-network data penetration interaction method, device, equipment and storage medium - Google Patents
Cross-network data penetration interaction method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN114006883B CN114006883B CN202111201791.0A CN202111201791A CN114006883B CN 114006883 B CN114006883 B CN 114006883B CN 202111201791 A CN202111201791 A CN 202111201791A CN 114006883 B CN114006883 B CN 114006883B
- Authority
- CN
- China
- Prior art keywords
- data
- center
- cross
- adapter
- interaction
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L61/00—Network arrangements, protocols or services for addressing or naming
- H04L61/09—Mapping addresses
- H04L61/10—Mapping addresses of different types
- H04L61/106—Mapping addresses of different types across networks, e.g. mapping telephone numbers to data network addresses
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/02—Standardisation; Integration
- H04L41/0226—Mapping or translating multiple network management protocols
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The application discloses a method, a device, equipment and a storage medium for cross-network data penetration interaction. On one hand, the cross-network data penetration interaction method provides the concept of an adapter for adapting different multi-class data sources with commonalities, and as all the data sources universal for markets are adapted, the data of each data island can be taken in real time; on the other hand, the forwarding center is arranged, the penetration strategy is simplified, the problem that the penetration strategy is increased due to the point-to-point increase is avoided, the problem of data circulation is mainly solved, fewer IPs are solidified, the change security strategy is reduced, the data circulation is safer, the latest data can be rapidly interacted across domains in real time, and the problem of data island is solved.
Description
Technical Field
The present disclosure relates to the field of information interaction technologies, and in particular, to a method, an apparatus, a device, and a storage medium for cross-network data penetration interaction.
Background
In the communication field, the domain represents a physically independent local area network, the domain is divided into security levels, and a domain with a high security level can access a domain with a low security level. Cross-domain penetration refers to the interaction of two physically isolated local area network data, such as: the network of a plurality of domains can be arranged in the industries such as banks, public security and the like, and the fields need the requirement of cross-domain data interaction, so that the safety of data is ensured. However, as the network is not enabled, the stored data cannot be associated, at present, a set of mature scheme is not available for supporting the mature scheme of cross-domain interaction, the traditional scheme is based on the function of importing and exporting, each time of cross-domain interaction needs multiple departments to examine and approve layer by layer to perform data interaction, the efficiency is low, and resource waste is caused, and the defects include:
1. adaptation problem of data sources
The multiple data stored in different domains, including traditional databases, large data platforms, libraries, etc., require different data sources to customize different adapters for adaptation.
2. Problem of network failure
The skip machine of the cross-network is required to be used as an agent, and the multi-layer support is required to be stable to normally transmit data to two domains of the network.
Therefore, it is necessary to quickly perform cross-domain interaction on the latest data in real time.
Disclosure of Invention
On one hand, the application provides a cross-network data penetration interaction method to solve the technical problems of low efficiency, high cost and information island caused by the existing cross-domain interaction.
The technical scheme adopted by the application is as follows:
a method of cross-network data penetration interaction, comprising the steps of:
after receiving the cross-network data penetration interaction task sent by the client, the dispatching center respectively sends the cross-network data penetration interaction task to an adapter and a forwarding center which are deployed in the domain, wherein the adapter is used for adapting the service of a large data center serving as a data destination or a data source in the domain, reading the data of the large data center, compressing the data into a compressed file, and transmitting the data;
the forwarding center transmits the cross-network data penetrating interaction task to an adapter deployed in a different domain to be interacted, and the adapter is used for adapting the service of a large data center serving as a data destination or a data source in the different domain, reading the data of the large data center, compressing the data into a compressed file and transmitting the data;
the adapter of the data destination acquires corresponding data from the big data center of the data destination according to the cross-network data penetration interaction task, and transmits the acquired corresponding data and state information to the forwarding center;
the forwarding center transmits the corresponding data acquired from the data destination to an adapter of the data source, the adapter of the data source performs memory-based MapReduce calculation on the acquired corresponding data and the corresponding data acquired from the data destination, and the calculation result is pushed to the forwarding center;
the forwarding center pushes the calculation result to an adapter of the data destination, and the adapter of the data destination writes the calculation result into a big data center of the data destination, so that the data destination obtains a desired interaction result.
Further, the big data center refers to a data source for storing data in a domain, the data source adopts a big data cluster, and the big data cluster comprises big data, CDH and HDP.
Further, the large data center refers to a data source storing data in a domain, and the data source adopts a traditional database, wherein the traditional database comprises oracle, mysql, mpp.
Further, the dispatching center receives a cross-network data penetration interaction task sent by the client through a rest interface, and the rest interface is used for standardizing how the application performs data interaction with the API provider in an HTTP layer.
Further, the content of the cross-network data penetration interaction task comprises a data source sql and a calculation result sql.
Further, after receiving the cross-network data transmission interaction task sent by the client, the dispatching center sends the cross-network data transmission interaction task to the message queue when respectively sending the cross-network data transmission interaction task to the adapter and the forwarding center deployed in the domain, and the forwarding center takes the task through the message queue.
In another aspect, the present application further provides a cross-network data penetration interaction device, including:
the system comprises a local domain task sending module, a dispatching center and a forwarding center, wherein the local domain task sending module is used for respectively sending cross-network data penetrating interaction tasks which are sent by a client through a rest interface to the adapter and the forwarding center which are deployed in the local domain after the cross-network data penetrating interaction tasks are received by the dispatching center, the content of the cross-network data penetrating interaction tasks comprises a data source sql and a calculation result sql, the adapter is used for adapting the service of a large data center which is used as a data destination or a data source in the local domain, reading the data of the large data center through the sql, compressing the data of the large data center into a compressed file and carrying out data transmission;
the system comprises a transmission center, a different domain task sending module and an adapter, wherein the transmission center is used for transmitting a cross-network data penetrating interaction task to the adapter deployed in the different domain to be interacted, and the adapter is used for adapting the service of a large data center serving as a data destination or a data source in the different domain, reading the data of the large data center through sql, compressing the data into a compressed file and transmitting the compressed file;
the data acquisition module is used for acquiring corresponding data from the big data center of the data destination according to the cross-network data penetration interaction task, transmitting the acquired corresponding data and state information to the forwarding center, and simultaneously, acquiring the corresponding data from the big data center of the data source according to the cross-network data penetration interaction task and transmitting the corresponding state information to the forwarding center;
the data calculation module is used for transmitting the corresponding data acquired from the data destination to the adapter of the data source, and the adapter of the data source carries out memory-based MapReduce calculation on the acquired corresponding data and the corresponding data acquired from the data destination and pushes a calculation result to the forwarding center;
and the data writing module is used for pushing the calculation result to the adapter of the data destination by the forwarding center, and writing the calculation result into the big data center of the data destination by the adapter of the data destination so as to enable the data destination to obtain the desired interaction result.
In another aspect, the present application further provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor implements the steps of the method for cross-network data penetration interaction when the program is executed.
In another aspect, the present application further provides a storage medium, where the storage medium includes a stored program, and when the program runs, the program controls a device where the storage medium is located to execute the steps of the method for cross-network data penetration interaction.
Compared with the prior art, the application has the following beneficial effects:
1. the method and the device start from the structure of the model, interface complex tasks in the process, and the needed party transmits parameters to the interface, so that a needed data result set can be quickly obtained.
2. The method and the device have the advantages that complicated scenes are processed, the practicability of the adapter is provided, the adapter can be adapted to various complicated multi-class data sources with commonalities, and the parallel development can be effectively realized;
3. the application sets up the forwarding center, the effect of forwarding center can be simplified data penetration tactics, avoids increasing the problem of penetration tactics complexity because of the increase of point-to-point, only opens effective port fixedly, has solidified less IP, has reduced the change security policy, improves the security of data interaction, makes the data circulation safer.
4. The method and the device can rapidly carry out cross-domain interaction on the latest data in real time, and solve the problem of data island.
In addition to the objects, features, and advantages described above, there are other objects, features, and advantages of the present application. The present application will be described in further detail with reference to the accompanying drawings.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application, illustrate and explain the application and are not to be construed as limiting the application. In the drawings:
fig. 1 is a flow chart of a method of cross-network data penetration interactions in accordance with a preferred embodiment of the present application.
Fig. 2 is a data interaction schematic diagram of a method of cross-network data penetration interaction according to a preferred embodiment of the present application.
Fig. 3 is a flow chart of a method of cross-network data penetration interaction in accordance with another preferred embodiment of the present application.
Fig. 4 is a data interaction schematic diagram of a method of cross-network data penetration interaction according to another preferred embodiment of the present application.
Fig. 5 is a schematic diagram of data interaction of a forwarding center in a preferred embodiment of the present application.
Fig. 6 is a schematic diagram of data interaction when no forwarding center is provided.
Fig. 7 is a block diagram of a cross-network data penetration interaction device according to a preferred embodiment of the present application.
Fig. 8 is a schematic block diagram of an electronic device entity according to a preferred embodiment of the present application.
Fig. 9 is an internal structural view of the computer device of the preferred embodiment of the present application.
Detailed Description
It should be noted that, in the case of no conflict, the embodiments and features in the embodiments may be combined with each other. The present application will be described in detail below with reference to the accompanying drawings in conjunction with embodiments.
Cross-network data penetration interactions:
the model belongs to two owners, the related data which is inferred by the model is divided into local and different places, the local data must be penetrated to different place interaction according to the requirements of a first party, and the result is penetrated back to the local, so that two scene conditions are shared:
assuming that there are two domains, domain A and domain B
1. Domain a sends a request to penetrate to domain B, which penetrates the result to domain a
2. Domain B sends the request to penetrate to domain a, which penetrates the result to domain B
Related art term interpretation definition:
rest interface: to specify how an application interacts with an API provider at the HTTP layer.
An adapter: the service for adapting the data source can read the data of the data source through sql and compress the data into a compressed file for data transmission.
And the forwarding center: for interacting data with an adapter, comprising:
1. the local domain data is sent to a forwarding center, and the forwarding center sends the local domain data to a different domain adapter.
2. The data in different domains is sent to a forwarding center, and the forwarding center sends the data in different domains to a local domain adapter.
And a dispatching center: and the method is used for sending scheduling instructions to the adapter, including sending tasks and recording task states.
Referring to fig. 1 and 2, a preferred embodiment of the present application provides a method for cross-network data penetration interaction, where the method is used to achieve that domain a takes an interaction result from domain B, and includes the steps of:
s1, after receiving a cross-network data penetration interaction task sent by a client through a rest interface, a dispatching center respectively sends the cross-network data penetration interaction task to an adapter A and a forwarding center which are deployed in a domain A, wherein the adapter A is used for adapting the service of a big data center A of a data destination city A in the domain A, reading data of the big data center A through sql, and compressing the data into a compressed file for data transmission;
s2, the forwarding center transmits a cross-network data penetration interaction task to an adapter B deployed in a domain B to be interacted, the adapter B is used for adapting the service of a big data center B serving as a data source city B in the domain B, reading the data of the big data center B through sql, compressing the data into a compressed file, and transmitting the data;
s3, the adapter A of the data destination city A acquires corresponding data A from the big data center A of the data destination city A according to the cross-network data penetration interaction task, and sends the acquired corresponding data A and state information to the forwarding center, and meanwhile, the adapter B of the data source city B acquires corresponding data B from the big data center B of the data source city B according to the cross-network data penetration interaction task, and sends the corresponding state information to the forwarding center;
s4, the forwarding center transmits corresponding data A acquired from the data destination city A to an adapter B of the data source city B;
s5, the adapter B of the data source city B carries out memory-based MapReduce calculation on the acquired corresponding data B and the corresponding data A acquired from the data destination city A (MapReduce is a cluster-based high-performance parallel calculation framework), and pushes a calculation result to a forwarding center;
s6, pushing the calculation result to an adapter A of the data destination city A by the forwarding center;
and S7, writing the calculation result into a big data center A of the data destination city A by an adapter A of the data destination city A, so that the data destination city A obtains a desired interaction result.
Specifically, the big data center A and the big data center B refer to data sources for storing data in domains, the data sources adopt big data clusters, and the big data clusters comprise Hua big data, CDH and HDP. Alternatively, the data source employs a legacy database that includes oracle, mysql, mpp.
Specifically, the dispatching center receives a cross-network data penetration interaction task sent by a client through a rest interface, wherein the rest interface is used for standardizing how an application performs data interaction with an API provider in an HTTP layer. Specifically, the content of the cross-network data penetration interaction task includes a data source sql and a calculation result sql, where sql is a standard sql99, the data source sql is used for querying and extracting required data, the calculation result sql is used for calculating the queried and extracted data, such as cross-linking, merging, and the like, and the structured data sql can increase adaptability and reduce user thresholds.
Referring to fig. 3 and 4, another preferred embodiment of the present application provides a method for cross-network data penetration interaction, the method being used for implementing that domain B takes an interaction result from domain a, and comprising the steps of: the method comprises the following steps:
s1, after receiving a cross-network data penetration interaction task sent by a client through a rest interface, a dispatching center respectively sends the cross-network data penetration interaction task to an adapter A and a forwarding center which are deployed in a domain A, wherein the adapter A is used for adapting the service of a big data center A of a city A of a data source place in the domain A, reading data of the big data center A through sql, and compressing the data into a compressed file for data transmission;
s2, the forwarding center transmits a cross-network data penetration interaction task to an adapter B deployed in a domain B to be interacted, the adapter B is used for adapting the service of a big data center B serving as a data destination city B in the domain B, reading the data of the big data center B through sql, compressing the data into a compressed file, and transmitting the data;
s3, the adapter B of the data destination city B acquires corresponding data B from the big data center B of the data destination city B according to the cross-network data penetration interaction task, and sends the data B and the state information to the forwarding center; meanwhile, the adapter A of the data source city A acquires corresponding data A from the big data center A of the data source city A according to the cross-network data penetration interaction task, and sends corresponding state information to the forwarding center;
s4, the forwarding center transmits corresponding data B acquired from the data destination city B to an adapter A of the data source city A;
s5, the adapter A of the data source city A carries out memory-based MapReduce calculation on the acquired corresponding data A and the corresponding data B acquired from the data destination city B, and pushes a calculation result to the forwarding center;
s6, pushing the calculation result to an adapter B of the data destination city B by the forwarding center;
and S7, writing the calculation result into a big data center B of the data destination city B by an adapter B of the data destination city B, so that the data destination city B obtains a desired interaction result.
The method for cross-network data penetration interaction in the embodiment comprises the following advantages:
1. from the structure of the model, the complex task is interfaced in the process, and the needed party transmits parameters to the interface, so that the needed data result set can be quickly obtained.
2. The complex scene is processed, and the adapter is adapted to various complex data sources, so that the parallel development can be effectively realized.
3. The function of the forwarding center can simplify the data penetration strategy, only the effective ports are fixedly opened, and the safety is improved.
As shown in fig. 5, when a forwarding center is provided, the forwarding center and the city B port include:
20667: a data transmission port, IP verification;
20668: message communication port, IP check;
the forwarding center only needs to open a plurality of fixed ports with the city B in the domain B, so that the forwarding center can interact data with a plurality of cities in the same domain of the domain A.
As shown in fig. 6, if there is no forwarding center, each time domain a adds a city, ports are added or IP verification is added, and after there is a forwarding center, the penetration policy only needs to be done once.
The above embodiments of the present application propose an adapter to adapt different types of data sources having commonalities, such as:
data source commonality and adapter development:
oracle: oracle adapter
Mysql: mysql adapter
…
Hua is big data: huacheng big data adapter
CDH: cdh adapter
…
The data sources of each city are diversified, but the data sources are generally stored by a specific database or a large data platform, and the data sources universal for all markets are adapted, so that the data of each data island can be obtained in real time, and the first step of data interaction calculation is solved.
In addition, the forwarding center is arranged, so that the penetration strategy is simplified, the problem of increasing the penetration strategy due to the increase of the point-to-point type is avoided, the problem of data circulation is mainly solved, fewer IPs are solidified, the change security strategy is reduced, and the data circulation is safer.
As shown in fig. 7, another embodiment of the present application further provides an apparatus for cross-network data penetration interaction, including:
the system comprises a local domain task sending module, a dispatching center and a forwarding center, wherein the local domain task sending module is used for respectively sending cross-network data penetrating interaction tasks which are sent by a client through a rest interface to the adapter and the forwarding center which are deployed in the local domain after the cross-network data penetrating interaction tasks are received by the dispatching center, the content of the cross-network data penetrating interaction tasks comprises a data source sql and a calculation result sql, the adapter is used for adapting the service of a large data center which is used as a data destination or a data source in the local domain, reading the data of the large data center through the sql, compressing the data of the large data center into a compressed file and carrying out data transmission;
the system comprises a transmission center, a different domain task sending module and an adapter, wherein the transmission center is used for transmitting a cross-network data penetrating interaction task to the adapter deployed in the different domain to be interacted, and the adapter is used for adapting the service of a large data center serving as a data destination or a data source in the different domain, reading the data of the large data center through sql, compressing the data into a compressed file and transmitting the compressed file;
the data acquisition module is used for acquiring corresponding data from the big data center of the data destination according to the cross-network data penetration interaction task, transmitting the acquired corresponding data and state information to the forwarding center, and simultaneously, acquiring the corresponding data from the big data center of the data source according to the cross-network data penetration interaction task and transmitting the corresponding state information to the forwarding center;
the data calculation module is used for transmitting the corresponding data acquired from the data destination to the adapter of the data source, and the adapter of the data source carries out memory-based MapReduce calculation on the acquired corresponding data and the corresponding data acquired from the data destination and pushes a calculation result to the forwarding center;
and the data writing module is used for pushing the calculation result to the adapter of the data destination by the forwarding center, and writing the calculation result into the big data center of the data destination by the adapter of the data destination so as to enable the data destination to obtain the desired interaction result.
Each of the modules in the above-described apparatus may be implemented in whole or in part by software, hardware, and combinations thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.
As shown in fig. 8, the preferred embodiment of the present application further provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor executes the program to implement the method of cross-network data penetration interaction in the foregoing embodiment.
As shown in fig. 9, the preferred embodiment of the present application also provides a computer device, which may be a terminal or a living body detection server, the internal structure of which may be as shown in fig. 9. The computer device includes a processor, a memory, and a network interface connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface of the computer device is used for communicating with other external computer devices through network connection. The computer program is executed by a processor to implement the method of cross-network data penetration interactions described above.
It will be appreciated by those skilled in the art that the structure shown in fig. 9 is merely a block diagram of a portion of the structure associated with the present application and is not limiting of the computer device to which the present application applies, and that a particular computer device may include more or fewer components than shown, or may combine some of the components, or have a different arrangement of components.
The preferred embodiment of the present application further provides a storage medium, where the storage medium includes a stored program, and when the program runs, controls a device where the storage medium is located to execute the method for cross-network data penetration interaction in the foregoing embodiment.
It should be noted that the steps illustrated in the flowcharts of the figures may be performed in a computer system such as a set of computer executable instructions, and that although a logical order is illustrated in the flowcharts, in some cases the steps illustrated or described may be performed in an order other than that illustrated herein.
The functions described in the method of this embodiment, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in one or more computing device readable storage media. Based on such understanding, a portion of the embodiments of the present application that contributes to the prior art or a portion of the technical solution may be embodied in the form of a software product stored in a storage medium, comprising several instructions for causing a computing device (which may be a personal computer, a server, a mobile computing device or a network device, etc.) to perform all or part of the steps of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: a usb disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk or an optical disk, or other various media capable of storing program codes.
The foregoing description of the preferred embodiment of the present application is not intended to limit the invention to the particular form disclosed, but on the contrary, the intention is to cover all modifications, equivalents, alternatives, and alternatives falling within the spirit and scope of the invention.
Claims (9)
1. A method for cross-network data penetration interactions, comprising the steps of:
after receiving the cross-network data penetration interaction task sent by the client, the dispatching center respectively sends the cross-network data penetration interaction task to an adapter and a forwarding center which are deployed in the domain, wherein the adapter is used for adapting the service of a large data center serving as a data destination or a data source in the domain, reading the data of the large data center, compressing the data into a compressed file, and transmitting the data;
the forwarding center transmits the cross-network data penetrating interaction task to an adapter deployed in a different domain to be interacted, and the adapter is used for adapting the service of a large data center serving as a data destination or a data source in the different domain, reading the data of the large data center, compressing the data into a compressed file and transmitting the data;
the adapter of the data destination acquires corresponding data from the big data center of the data destination according to the cross-network data penetration interaction task, and transmits the acquired corresponding data and state information to the forwarding center;
the forwarding center transmits the corresponding data acquired from the data destination to an adapter of the data source, the adapter of the data source performs memory-based MapReduce calculation on the acquired corresponding data and the corresponding data acquired from the data destination, and the calculation result is pushed to the forwarding center;
the forwarding center pushes the calculation result to an adapter of the data destination, and the adapter of the data destination writes the calculation result into a big data center of the data destination, so that the data destination obtains a desired interaction result.
2. The method of cross-network data penetration interactions of claim 1,
the large data center refers to a data source for storing data in a domain, the data source adopts a large data cluster, and the large data cluster comprises large data, CDH and HDP.
3. The method of cross-network data penetration interactions of claim 1,
the large data center refers to a data source storing data in a domain, and the data source adopts a traditional database, wherein the traditional database comprises oracle, mysql, mpp.
4. The method of cross-network data penetration interactions of claim 1,
and the dispatching center receives a cross-network data penetration interaction task sent by the client through a rest interface, and the rest interface is used for standardizing how the application performs data interaction with the API provider at an HTTP layer.
5. The method of cross-network data penetration interactions of claim 1,
the cross-network data penetration interaction task comprises a data source sql and a calculation result sql.
6. The method for cross-network data transmission interaction according to claim 1, wherein after the dispatching center receives the cross-network data transmission interaction task sent by the client, when the cross-network data transmission interaction task is respectively sent to the adapter and the forwarding center deployed in the local domain, the cross-network data transmission interaction task is sent to the message queue, and the forwarding center takes the task through the message queue.
7. An apparatus for cross-network data penetration interactions, comprising:
the system comprises a local domain task sending module, a dispatching center and a forwarding center, wherein the local domain task sending module is used for respectively sending cross-network data penetrating interaction tasks which are sent by a client through a rest interface to the adapter and the forwarding center which are deployed in the local domain after the cross-network data penetrating interaction tasks are received by the dispatching center, the content of the cross-network data penetrating interaction tasks comprises a data source sql and a calculation result sql, the adapter is used for adapting the service of a large data center which is used as a data destination or a data source in the local domain, reading the data of the large data center through the sql, compressing the data of the large data center into a compressed file and carrying out data transmission;
the system comprises a transmission center, a different domain task sending module and an adapter, wherein the transmission center is used for transmitting a cross-network data penetrating interaction task to the adapter deployed in the different domain to be interacted, and the adapter is used for adapting the service of a large data center serving as a data destination or a data source in the different domain, reading the data of the large data center through sql, compressing the data into a compressed file and transmitting the compressed file;
the data acquisition module is used for acquiring corresponding data from the big data center of the data destination according to the cross-network data penetration interaction task, transmitting the acquired corresponding data and state information to the forwarding center, and simultaneously, acquiring the corresponding data from the big data center of the data source according to the cross-network data penetration interaction task and transmitting the corresponding state information to the forwarding center;
the data calculation module is used for transmitting the corresponding data acquired from the data destination to the adapter of the data source, and the adapter of the data source carries out memory-based MapReduce calculation on the acquired corresponding data and the corresponding data acquired from the data destination and pushes a calculation result to the forwarding center;
and the data writing module is used for pushing the calculation result to the adapter of the data destination by the forwarding center, and writing the calculation result into the big data center of the data destination by the adapter of the data destination so as to enable the data destination to obtain the desired interaction result.
8. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that,
steps of a method of cross-network data penetration interaction according to any of claims 1 to 5 when said program is executed by said processor.
9. A storage medium including a stored program, characterized in that,
the method of cross-network data penetration interaction of any one of claims 1 to 5, when said program is run, controlling a device in which said storage medium is located.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111201791.0A CN114006883B (en) | 2021-10-15 | 2021-10-15 | Cross-network data penetration interaction method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111201791.0A CN114006883B (en) | 2021-10-15 | 2021-10-15 | Cross-network data penetration interaction method, device, equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114006883A CN114006883A (en) | 2022-02-01 |
| CN114006883B true CN114006883B (en) | 2023-06-27 |
Family
ID=79922928
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111201791.0A Active CN114006883B (en) | 2021-10-15 | 2021-10-15 | Cross-network data penetration interaction method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114006883B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10326830B1 (en) * | 2016-09-02 | 2019-06-18 | Amazon Technologies, Inc. | Multipath tunneling to a service offered at several datacenters |
| CN110502559A (en) * | 2019-07-25 | 2019-11-26 | 浙江公共安全技术研究院有限公司 | A kind of data/address bus and transmission method of credible and secure cross-domain data exchange |
| WO2021057198A1 (en) * | 2019-09-27 | 2021-04-01 | 北京市天元网络技术股份有限公司 | Big data-based cross-domain service whole-process routing and penetration method and apparatus |
| US11121989B1 (en) * | 2020-05-29 | 2021-09-14 | Bank Of America Corporation | Centralized repository and communication system for cross-network interactions |
| CN113422837A (en) * | 2021-07-02 | 2021-09-21 | 广东电网有限责任公司 | Cross-network data transmission device, method, terminal and readable storage medium |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8407366B2 (en) * | 2010-05-14 | 2013-03-26 | Microsoft Corporation | Interconnecting members of a virtual network |
| US9582319B2 (en) * | 2010-08-13 | 2017-02-28 | International Business Machines Corporation | Migrating virtual machines across network separated data centers |
| US11126481B2 (en) * | 2013-12-09 | 2021-09-21 | Micro Focus Llc | Fulfilling a request based on catalog aggregation and orchestrated execution of an end-to-end process |
| US20180191856A1 (en) * | 2016-12-29 | 2018-07-05 | Synology Inc. | Cross-domain communication methods and proxy servers using the same |
-
2021
- 2021-10-15 CN CN202111201791.0A patent/CN114006883B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10326830B1 (en) * | 2016-09-02 | 2019-06-18 | Amazon Technologies, Inc. | Multipath tunneling to a service offered at several datacenters |
| CN110502559A (en) * | 2019-07-25 | 2019-11-26 | 浙江公共安全技术研究院有限公司 | A kind of data/address bus and transmission method of credible and secure cross-domain data exchange |
| WO2021057198A1 (en) * | 2019-09-27 | 2021-04-01 | 北京市天元网络技术股份有限公司 | Big data-based cross-domain service whole-process routing and penetration method and apparatus |
| US11121989B1 (en) * | 2020-05-29 | 2021-09-14 | Bank Of America Corporation | Centralized repository and communication system for cross-network interactions |
| CN113422837A (en) * | 2021-07-02 | 2021-09-21 | 广东电网有限责任公司 | Cross-network data transmission device, method, terminal and readable storage medium |
Non-Patent Citations (3)
| Title |
|---|
| Cross-area Node Selection Algorithm for Data Replica Storing in Cloud Data Centers;Yang Lu 等;IEEE;全文 * |
| 东数西算：我国数据跨域流通的总体框架和实施路径研究;王建冬 等;电子政务(第03期);全文 * |
| 面向服务的数据中心安全框架;胡腾 等;电信科学(第1期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114006883A (en) | 2022-02-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20190140982A1 (en) | Communication Methods and Systems, Electronic Devices, and Computer Clusters | |
| WO2021088641A1 (en) | Data transmission method, data processing method, data reception method and device, and storage medium | |
| WO2010087829A1 (en) | Selectively communicating data of a peripheral device to plural sending computers | |
| CN111679911B (en) | Management method, device, equipment and medium of GPU card in cloud environment | |
| US20170085653A1 (en) | Method, device and system for message distribution | |
| CN110046026A (en) | Cloud host specifies virtual disk method for limiting speed, calculates equipment and cloud platform | |
| CN110287146A (en) | Using the method, equipment and computer storage medium of downloading | |
| CN114385091A (en) | Method and device for realizing network disk drive character, network disk and storage medium | |
| CN102043645A (en) | Plug-in loading method and device | |
| CN112162925A (en) | Test method and system based on service virtualization | |
| CN103428260A (en) | System and method for allocating server to terminal and efficiently delivering messages to the terminal | |
| CN109218393A (en) | A kind of implementation method of push, device, equipment and computer storage medium | |
| CN109150956A (en) | A kind of implementation method, device, equipment and computer storage medium pushing SDK | |
| CN104281488A (en) | Implementation method and system of server engine | |
| CN111371809A (en) | Server based on reverse proxy architecture and GIS service access control method | |
| CN114006883B (en) | Cross-network data penetration interaction method, device, equipment and storage medium | |
| CN109347899A (en) | The method of daily record data is written in distributed memory system | |
| CN111049913B (en) | Data file transmission method and device, storage medium and electronic equipment | |
| CN109040491A (en) | On-hook behavior processing method, device, computer equipment and storage medium | |
| CN108494700A (en) | Across link data transmission method, device, computer equipment and storage medium | |
| CN114218000A (en) | Log management method and device, electronic equipment and computer readable storage medium | |
| CN204425400U (en) | Application server system | |
| CN113760318A (en) | Information processing method, information processing apparatus, server, and storage medium | |
| CN109189746A (en) | Implementation method, device, equipment and the storage medium of general streaming Shuffle engine | |
| CN113707346B (en) | Novel online inquiry system and method based on Internet hospital |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |